Hello everybody. I am making a catalog of all my photos. I am looking for any software or script that can efficiently find duplicates on my hard drive. Is there anything that can search my hard drive for duplicates using EXIF data? Thank you very much.
Windows Explorer will do the job. Start>search file and folders search for *.jpg or *.cr2 what ever extension you like click desktop and explorer will open showing all the jpg's on your drive. Click on a empty space on the columns and put a check box by any exif you like "date created" click on that column header and the files are sorted by that criteria.
There are programs to do the job but this may be all you need.
What do you consider a duplicate? Files with the same or different name with identical exif data may all have some different modifications to the image while other image files may have beem made from the original camera file and may have had the exif data stripped in the process. What do you consider a duplicate?"
Check this page out. This guy tested the free options and there are some very good ones. I have tested a few and have yet to find anything fast enough for my NAS. It is loaded with dupes and I am not patient enough to wait 3 or 4 days to find them all...
lottel, you might do well to get Lightroom so that you can manage your library of photos. It will let you search its catalog for files based on all sorts of stuff including EXIF data and things that you assign for yourself such as ratings, labels and keywords. You can create collections too.
In my opinion a catalog of photos is of little use if you can't find what you want quickly, and that's where Lightroom and similar programs are very good. However, like any database work - whether automated or manual - it can't help you much if you don't take the time to feed it info. It gets the EXIF data automatically but you need to assign ratings and so on yourself.
lottel wrote:
Hello everybody. I am making a catalog of all my photos. I am looking for any software or script that can efficiently find duplicates on my hard drive. Is there anything that can search my hard drive for duplicates using EXIF data? Thank you very much.
As Mr. Mouse asked, it all depends on what you consider a duplicate. Is a derivative post-processed PSD or JPEG file a duplicate, or do you consider it a separate original file from the one first shot by your camera? Also, it depends on how rigorous you want the search to be. Ideally you want to use software that can at minimum generate and compare CRC checksums to do the duplicate detection. In terms of detection rigor; CRC checksums are the fastest, followed by software that use MD5 hash functions, and the slowest is an actual byte-by-byte comparison of files.
I've done this type of file sorting and consolidation in the past, so the first thing I would suggest is to use a file ingestion tool like Downloader Pro by Breeze Systems or ImageIngester by Marc Rochkind to scan your drive(s) where your photos are currently located, and move and soft them into file folders based on the embedded EXIF metadata to a central location. I'm partial to Downloader Pro since I've used it for about 7 years to do all my downloading, renaming, and sorting from my cameras but Image Ingester is also excellent software. Both have trial periods so you should not have to buy them unless you find them useful to keep beyond their trial periods.
If you need help configuring Downloader Pro to perform the task just let me know and I'd be happy to help. But the basic idea is to use either software to scan your current photo location(s), then physically move all the photos to a central location (ideally a new hard drive but anything can work), while sorting the files into date-based folders based on the EXIF metadata. You might also find it useful to also use the software to rename your files into a consistent format.
This may be all you need/want to do and once your files are all sorted and consolidated, you can do an "eyeball" comparison of the files in each folder to see if there are duplicates (unlikely since DLPro does basic duplicate checking). Or you can use other software that do comparisons based on checksums or hashes. I know that Downloader Pro supports the Authentegrity plug-in for duplicate detections but I've personally never used it: http://www.authentegrity.com/Downloads/downloads1.html
Thanks for all of your responses. What happened was that, I had this bad characteristic of having paranoia. Several times, I have downloaded images to my hard drive from the CF card multiple times unknowingly. And so for a period of time, I think I have doubled some of them to my hard drive. I usually just drag and drop them without using any software to import the images. Once I had the time, I used Media Pro to arrange them by month and year of capture time and then renamed them chronologically (i.e. 2012 09 001, 2012 09 002... for images captured in 9/2012). I then created a folder for each year and then subfolder for each month. So know, I want to go back to each folder (monthly) and see if there are duplicates and erase them. So now, I am trying to find a software that could find duplicates so I don't spend a lot of time searching per images on each of the folders and subfolders individually.
