HELP- 1 TB of multiple copies of Photos -Consolidation plan

Comfy Old Boots

Well-known member
UKGSer Subscriber
Joined
Apr 9, 2007
Messages
11,054
Reaction score
379
Location
Dublin
Hi there,

I purchased a reasonably large external drive to begin the above process. I currently have a number of different backups of my photographs taken on my nikon, fuji and iphones. I've put them all onto one big drive, sourced from two other external drives, two pc's and my current mac. It's a little over 1 TB in size.

What I'd like to do, is to consolidate the multiple versions of files down to just one version, with a view to then going through all of the photos over the next couple of weeks and discarding all the stuff I don't need and keeping and archiving the stuff that I consider is good or worth keeping. Trouble is, there is a lot of files and some are in triplicate.

I've been looking online and there seems to programs that will look at all the files and remove duplicates but none are filling me with confidence. When I worked in post production sound for cinema, we had programs that would do this. That was twenty years ago now and I'm sure things have moved on.

I'm willing to throw money at this problem but would rather not with out a bit of advice. I also accept, that the only way to do this might be to go through each and every shot individually.

Like I said, all the photos are on one big drive. The individual backups / copies are in discrete folders that are called pc 1 or pc2 or hd 500gb etc.

Any suggestions on how to get myself out of this hole without losing stuff or driving myself mad?!!!!! I'm hoping to do this just once and be a good organised boy in the future:aidan
 
If you want to quickly view them as a galelry to aid sorting maybe something like ACDSee Photo Studio [there a 30 day trial] would assist you. As for sorting duplicates Id be averse to doing it automatically as its likey any such software would go by name and size, which given you have different sources could hit different image files of a similar ilk.
 
I'll take a look

Hi Andy, That is a very good suggestion. It seems to be doing a lot of the heavy lifting for me. Shows up what it thinks are duplicates into little groups and allows me to delete them as i see appropriate. I'll spend a little time with it and report back. It's not doing exactly what I want but it is accomplishing my needs in a different way. Mind you that is to be confirmed. It is taking a VERY long time to go through the whole drive but the little test I did showed promise. Thanks.
 
I've been using a tool called "Duplicate Photos Fixer Pro" recently. It's Mac - and I don't know if it's available for PC or not.

But it does an excellent job. It's not just comparing names and sizes, but also dimensions, dpi, metadata, gps location capture and modification dates. So it does a really good job of avoiding false duplicates.

You can set it run through your entire library, then it will mark what it thinks are duplicates and pull up thumbnails of them so you can see for sure.

I just eliminated 780 duplicates from library in about 10 minutes.
 
I've been using a tool called "Duplicate Photos Fixer Pro" recently. It's Mac - and I don't know if it's available for PC or not.

But it does an excellent job. It's not just comparing names and sizes, but also dimensions, dpi, metadata, gps location capture and modification dates. So it does a really good job of avoiding false duplicates.

You can set it run through your entire library, then it will mark what it thinks are duplicates and pull up thumbnails of them so you can see for sure.

I just eliminated 780 duplicates from library in about 10 minutes.

Hey, that sounds just the ticket. Andys suggestion is pulling up duplicates that are not in fact duplicates. I'll give the program you suggested a go. I'm on Mac as my main OS so fingers crossed it's what I'm looking for.

I'm prepararing myself to start a new lightroom catalogue, import the entire drive (not importing duplicates) and go through every single folder. I expect it will sort them by date. I've a broken ankle and I'm confined to bed so time is not an issue!
 
Now that is the business!
f9ebe8ab94c8c724478af4732937cae9.jpg


My first concern is that it only searched for .jpg. I'm interested in sorting all my RAW images too> . It's a good start though. I'll delve deeper. Thanks.

Another thing i've noticed is that it considers bracketed shots duplicates despite them having different file names.
 
I'll take a look

Hi Andy, That is a very good suggestion. It seems to be doing a lot of the heavy lifting for me. Shows up what it thinks are duplicates into little groups and allows me to delete them as i see appropriate. I'll spend a little time with it and report back. It's not doing exactly what I want but it is accomplishing my needs in a different way. Mind you that is to be confirmed. It is taking a VERY long time to go through the whole drive but the little test I did showed promise. Thanks.
https://www.lightroomqueen.com/clean-duplicate-photos/
 
The other thing worth mentioning is that my Lightroom catalogue is incomplete so I think I’m going to start again. I’m in the process of deleting the duplicate JPEGs. That program works well. I’ll report back later on the raw side of things. The amount of data is resulting in a lot of time and number crunching by the computer. I’ll check the Lightroom plugin later. Thanks.
 
Duplicates you say?

I’m watching this thread with interest as I could do with doing exactly the same clean up with my images.

I’m also watching this thread with interest as I could do with doing exactly the same clean up with my images. :augie

:thumb2
 
The other thing worth mentioning is that my Lightroom catalogue is incomplete so I think I’m going to start again. I’m in the process of deleting the duplicate JPEGs. That program works well. I’ll report back later on the raw side of things. The amount of data is resulting in a lot of time and number crunching by the computer. I’ll check the Lightroom plugin later. Thanks.

You can keep LR catalogues !

I do it each year and start a new one which is my way of keeping the photos organized a bit
I also create a single new folder in the Mac for that catalogue to store its images I call it fuji 2020 ( thats this years) and i then create sub folders within that through lightroom on the file handling section

As for rooting out duplicates i would suggest that with 1TB to go through a programme that does it for you would be the right answer

I have clean my mac ( it might be Mac X ) and i do believe that has the ability to scan the Mac and drives and root out duplicates ( i think it cost about 30 quid )
 
Sage advice and info I shall apply going forward, In the mean time I'm borked! I have lost a few lightroom catalogues. I've smoke coming out of the computer at the moment it's working so hard :) I've 30,000 RAW images. I knew I needed to do this two years ago. I've been looking through them as they are importing into Lightroom. I am hoping when it finishes importing, it'll sort everything by date and I'm going to start going through every single shoot and I'm going to rate the good ones. I've set lightroom to not import duplicates, so I'll see how well it does but I expect it may be tomorrow before it's completed all it's tasks.

As for JPEG, the program suggested earlier is working really well. It's just not doing what I wanted it to do. I'll touch base tomorrow.
 
... I'll touch base tomorrow.
In full and in simple idiotproof English please.*

*for me - coz i'm hard of thinking when it comes to this stuff. :blast

Ta muchly. :thumb2
 
Morning Marc,

i do not wish to belittle your photography storage/organisational issues but what I'm dealing with here is possibly slightly different or more specifically, more wide ranging and extensive than what you possibly face! So, I'm going to do a little write up for you and then try and document my slightly bigger rats nest. Ok?

I take you have loads of photos in lots of different places? You want to have all the photos in one place but do not want to delete important shots. Like Linda doing wheelies in her leathers?!?

Firstly, decide where you want to move all the photos to. This is a central place, where EVERYTHING will be copied. Let'ss say you have photos on two pcs and two memory sticks. In the central place, create a folder called ALL PHOTOS or whatever. Inside there, create a folder for EACH DIFFERENT photo source. Then copy from that source to the relevant folder.

When all your photos are in the one place, in different folders, within a folder, you can go onto the next step, which is rooting out the duplicates. The program I used to do this is on mac but I am sure there are many for PC.

So, dude, get all your media together as I described and we can go from there, when ya get there!

Also, I'm not saying this is how it should be done but I am suggesting starting off this way so that nothing gets deleted or overwritten.

Also, when copying stuff to your central folder, do NOT delete the originals.
 
I'll update my own story in due course but it's a bit complicated and I'm still working through it so I'll leave it for the moment. However, it is a story of multiple computer operating systems, hard drive formats, compatible hard drive formats, different picture formats, multiple copies of those pictures in different places. Basically a bucket of wee. I am getting somewhere though. I now have all the jpegs isolated from my various formats of RAW files. I'll go through the JPEGS at a later date. I have deleted 40GB of jpegs from the pile. JPEG pile now down to 80GB. What I am trying to accomplish now, is getting all my RAW files into a catalogue in Lightroom. I then want to sort through them, delete the ones I don't need and be a good boy in future. Thanks for the help earlier, some good tips that have got me started. I'll try and do a proper write up again when I have finished the job.

Oh and when I've finished the job, I need to move all the files from the external HD FAT 32 ex formated drive to a mac formated drive 'cos the fat 32 is giving me issues on mac.
 
Good luck with the sorting Im sure it brings back many memories sorting through your collection covering several years.There used to be an assortment of "free" storage services online for backing up such stuff i.e. flickr,dropbox etc but as with many such places you now only get a gigabyte or so with each before charges ensue so local backups are getting used more and more,I have just upgraded my NAS to a Synology DS218+ for such stuff.

d4f6220b89d6d8a93e9a46d27658af70.png
 
Yes indeed. Memories. Amazing what it threw up. I've discarded 99.99% of them. Kept a couple of photos but mixed emotions looking back on a past relationship. Especially the early years when we were very happy. But then looking the later ones and remembering. Anyway, I wish her well but bar two or three photos to remember her face, all purged.

I've gone through all the JPEGS now. Anything of interest is in iPhoto.

I've also imported all the RAW files into lightroom from about 40 different folders. I'm currently exporting the entire catalogue with photos to a new location. It is from there I hope to start weeding out the dross.

I've been at this since ten AM!
 
Mr Boots, you sound like you are an expert on LR, I use LR but don’t claim to know what goes on behind the scenes. I import my RAW images into LR from the camera, edit them etc and then export as JPEGS. The original RAW files are then as I understand it, saved on the MAC’s hard drive in catalogues associated with LR. I’m considering a new Mac as mine is getting slow being a late 2015 model, apart from using the Cloud which I do not use, is there a way I can copy my existing catalogues to an external drive to then reinstate them with a fresh install of Adobe CC on a new Mac. Sounds simple and I’m sure it is, I may be reasonably competent with a camera but not so much with the computer! Thanks in advance.
 


Back
Top Bottom