p.1 #1storing, organizing and backing up a growing photo/video library
I’m curious what everyone is using for long-term photo/video storage, organization, backup, and retrieval.
My amount of data has started growing quickly enough that my current “leave it on the cards until I deal with it” approach obviously isn’t going to work much longer.
My main stills camera is a Nikon Z8, so the RAW files are fairly large. I’m also shooting video with an Insta360 Ace Pro 2, Insta360 X5, drones, and other cameras. Recently added a fujifilm E5. Some of the material is family photography, some is photography challenges/projects, and I’m also starting to produce quite a bit of video for work, including internal training/SOP videos.
What I’d really like is a workflow where I can come home, dump the contents of my memory cards into one central location relatively quickly, clear/reuse the cards, and then come back later when I actually have time to sort, rate, edit, process, or delete things.
I also don’t necessarily want to immediately delete everything that didn’t make the final edit. I’d like to retain RAWs, JPEGs, finished edits, video footage, project files, and at least some of the material that didn’t make the initial cut.
The other important part is that I need the system to be expandable. I’m not looking to spend thousands of dollars immediately on a giant NAS or enterprise-level setup, but I also don’t want to buy something today that has to be completely replaced a year from now.
So I’m curious what your actual workflow looks like:
* What do you dump your cards onto first?
* DAS, NAS, individual external drives, internal storage, something else?
* How much storage did you start with?
* How do you organize folders/projects?
* Do you keep every RAW or eventually cull aggressively?
* How do you separate originals from edited/final files?
* What do you use for your second and/or off-site backup?
* Do you use Lightroom or another cataloging system to find older images?
* If you shoot a lot of video too, do you handle that differently from stills?
* If you were building your system again from scratch, what would you do differently?
I’m less interested in the absolute cheapest solution and more interested in building a sensible system that I can start relatively small and expand over time. A system that may be overkill now but allow the system to grow with me is the plan. (Much like my z8. It’s overkill but damn I love using it)
I’d really appreciate hearing what people are actually using rather than just theoretical ideal setups.
p.1 #2storing, organizing and backing up a growing photo/video library
I do stills, but the setup is probably the same for video other than using bigger drives.
1) I have all my current working files on a fast TB4 NVME drive. This is the most expensive part of my array.
2) The above drive is backed up hourly to a large capacity, single USB 3 spinners. This second drive also houses all my historical files.
3) I have drive 2 backed up to an identical spinning drive every other hour for complete redundancy.
4) Monthly or after any large shoot, I further backup drive 2 to a third identical spinning drive which I store offsite for fire or other catastrophic loss.
The beauty of the above system is firstly it's inexpensive. Next, if any drive fails, I can pretty easily recreate it. The TB4 NVME drive is essentially as fast on I/O as my main system SS drive, so as a working drive it's awesome. It's also portable so I can take it with me and access those working files on my laptop as needed. I offload dormant files as needed for NVME space since they're already sitting on the other backup drives anyway. The only slowdown I experience is the rare occasion I need to go back and move an older shoot on the spinner back to the NVME for new edits; and here the spinning drive I/O speed is a bottleneck. Oh, I use Carbon Copy Cloner for my backups: https://bombich.com . You can schedule it however and even enact failsafe anti-delete protocols if desired.
My file organization is simple for me simply because I started this way before catalogs became a thing and I'm too lazy to recategorize everything. It might be different if I were starting off today... That caveat said, I have a main folder for each year. Inside that is a folder for each dedicated shoot, and here I tag the front of the folder-shoot name with a reverse numerical date, for example: "26.09.30.Shootname" so they all order themselves chronologically inside the year folder. (And in fact given my system they would order themselves chronologically without any year folder, but the year folders keep things cleaner.) This way, I know I did the so-and-so shoot sometime in April, so all I have to do is scroll back to the list of "26.04.xx.whatever" and find the specific shoot if I need to access it
p.1 #3storing, organizing and backing up a growing photo/video library
I have 2 home offices with identical laptops, so I store very little on either. I use a 4 TB SSD as my working drive, with 8 TB SSDs at each location for backups of all documents and photos. I've backed up all my older rotating drives to SSDs, and I keep them in a dry cabinet.
p.1 #4storing, organizing and backing up a growing photo/video library
I'll just state some minimums here from a while ago.
5+ sets of data in multiple locations, totaling X PB of HDDs. Mostly they are in RAID-Z2, some RAID 5 or RAID 6 conventional ext4, up to 180TB per array. WIP (work in progress) on 48+ TB internal SSDs in main system, about the same in portable SSDs. I really prefer to work from SSDs on a PCIe bus, using NAS HDDs for bulk storage. If I'm going to work on a project extensively it is replicated to the internal SSDs and then is pushed back when finished. I sometimes use single loose drives for extra copies of archives.
The cost of storage is so high now, that for the first time in 25 years, I'm not buying any new drives unless absolutely necessary. I'm even looking at repurposing some old He drives from years ago.
The most important thing is to have a good redundancy strategy, such as traditional 3-2-1 or the newer 3-2-1-1-0 model.
p.1 #5storing, organizing and backing up a growing photo/video library
Couple more quick comments. Over the years I have owned a handful of NAS/RAID boxes.
1) I have had exactly 2 spinning drives fail, and both times it was faster to re format and rebuild the entire array (since I had full redundancy I could copy back from) than let the array rebuild itself.
2) 3 of the NAS/RAID boxes themselves eventually failed.
3) While R0 did improve performance, mirroring was mandatory; and in the end not really much faster than R5.
4) NVME speed blows any RAID (with spinning drives) away.
5) It was after experiencing all of that I developed the above system as a relatively cheap and yet fast I/O solution. But I accept it’s not ideal for a workgroup and here some type of NAS is going to be preferable.
p.1 #6storing, organizing and backing up a growing photo/video library
If you buy hundreds of hard drives and dozens of NVMe SSDs, M.2, SATA, U.2/U.3 then there will be failures. It's like getting a flat tire or dead battery. It happens, you replace, and you move on. I much prefer ZFS over ext. If hard drives are over 3 years and one fails, I would possibly trash the array and replace them all, but gosh they are so expensive now!
Of course a NAS can fail, but the better ones are pretty reliable or I have been quite lucky. The ones
with XENON CPUs and ECC RAM sold as business/low enterprise grade are usually solid. The ~$1000 specials with an external power supply brick and ARM SoC seem like they are more likely to fail.
Running RAID 5 with NVMe SSDs requires a lot of CPU or an accelerator for good performance. That's beyond most consumers/enthusiasts mean or needs. You can stripe M.2 SSDs due to small size and backup to NAS, though that's officially advised.
There are so many possibilities depending on the amount of files. People with 25TB/50TB/100TB/more all have different options.
p.1 #7storing, organizing and backing up a growing photo/video library
My working drives are 4 TB NVMe SSDs.
These are automatically backed up to a TrueNAS server that I built using mostly spare parts I had lying around, then bought "new/used" HDDs. The NAS is configured as a RAIDZ2, 8 wide, ZFS. Most people recommend a RAIDZ2 configuration (2 parity drives). This has been running for almost 2 years and I recently had a drive fail. While this server was built with the intent as an archive, I've discovered the magic of self-hosting, so I am running a number of apps for media-streaming, photo library (Immich), then other "home lab" experiments for different purposes.
Then another redundancy is that I have an older NAS that I received from a neighbor for free. It houses 4 drives, but is too slow to use for anything other than an archive. It is running Unraid (which allows for different sized drives). My photo library is automatically sync'd from the TrueNAS to the Unraid using Syncthing. Ideally, I would like to move this Unraid offsite.
I used to have a tertiary redundancy on my local Windows rig, but it was too much. Right now, the main objective will be to move the Unraid offsite and I think I will be good.
p.1 #8storing, organizing and backing up a growing photo/video library
I ran TrueNAS CE for a while, but decided to revert to commercial NAS for space, portability, and efficiency. I'm not familiar with UnRAID. It seemed strange to me with the licensing and not convinced it's better than the TrueNAS, but I'm not really an IT person.
p.1 #9storing, organizing and backing up a growing photo/video library
EB-1 wrote:
I ran TrueNAS CE for a while, but decided to revert to commercial NAS for space, portability, and efficiency. I'm not familiar with UnRAID. It seemed strange to me with the licensing and not convinced it's better than the TrueNAS, but I'm not really an IT person.
EBH
It really depends on your use case. I find TrueNAS to be pretty much enterprise grade, so it is a bit overkill for my use. UnRAID is much more flexible in that you can bring in HDDs of any size, and you can swap in larger drives so long as your parity drive is your largest. The drawback is that I think UnRAID is supposed to be slower, so if you are in a setting where people are working off of the server files, the UnRAID system isn't really ideal. That being said, I think they now have an option for ZFS format. The other option is that it looks like there are more hobbyists with home labs using UnRAID and writing apps for it.
I think if I were going to set up everything again, I would probably go with UnRAID because I like the flexibility of system, and how it can grow with your needs - if you need more space, you just swap out the smaller drive with a larger one or keep adding drives. With the RAIDZ2, I essentially have to have all the drives matched in size and specify the size of your array at the beginning. So I'm essentially locked into this pool.
But regardless, I love my TrueNAS system. I keep finding more and more uses for the server...
p.1 #10storing, organizing and backing up a growing photo/video library
Over the years I have developed some principles and procedures that match them.
First my shooting is selective. I don't shoot and then supposedly cull later. Before pressing the shutter I seriously decide if the shot is likely to be something I will value. That could be simply capturing a memory or capturing an image I would likely want to print.
Next I cull aggressively. I use the same criteria: a great memory, or something that falls into the fine art category and I would be proud to print and show others. I try to cull soon and the same day after a shooting session.
In spite of selective and deliberate shooting and aggressive culling, I still end up with way more files that fail to meet expectations. My next step is to do a quick post processing for images that are clearly keepers. That is typically no more than 2-12 images for a days shooting.
Organizing and storage follows some simple steps. Each day's shooting goes into a file with the date and event, location, and/or subject description.
Backup storage is simple and follows a couple of principles. First I never use any sort of automated backup. That principle was decided years ago when backup software failed to work as intended. Next all storage is on separate drives which are stored in different locations and are never used at the same time. For daily storage, my files are maintained on my desktop and a separate HD which I only power up when in use. Periodically I further backup on two more HDs in different locations. One is kept in a basement safe and the other is in a detached garage.