In this work we have used a large scale dataset, with small label noise, while keeping down the number of manual annotation. The main idea was to train a classifier to rank the data presented to the annotators. This procedure has been developed for faces, but is evidently suitable for other object classes as well as fine grained tasks. Then as another contribution we have showed that a well trained deep convolutional neural network can achieve very good results comparing to the state of the art.