Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | General Gear-talk | Join Upload & Sell

1
       2       end
  

Archive 2013 · Photo storage backup choices

  
 
safcraft
Offline
• • •
Upload & Sell: Off
p.1 #1 · p.1 #1 · Photo storage backup choices


So i have now 7 years of RAW data storage, which translate into around 600GB.
These are personal photos, not professional, so i would like to keep them safe "forever".

Currently they are in 2 separate hard drives , both 3.5" external.
I have beed daunted with harddrive failures in the past, lost some data, but fortunately not this particular data.
I would like to know what do you guys use to "keep them safe"....

RAID storage ? Cloud ?
With up to 5GB per week ratio...i can't keep uploading all data to a cloud, not enough bandwidth....would take forever.

I am thinking of buying an AirPort Time Capsule with 3TB....but is it smart to have it 24/7 turned ON, and hope that it will not "die" in say 5 years ...or sooner ?




Jul 19, 2013 at 01:58 PM
OccAeon
Offline
• • •
Upload & Sell: Off
p.1 #2 · p.1 #2 · Photo storage backup choices


If you want them to be safe, you have to have at least two copies, with one copy kept outside of your home.

There are lots of things that you need to protect your data from, including:
(1) Device failure
(2) Fire/flood/natural disaster
(3) Theft
(4) Viruses/malware (uncommon)
(5) User error (i.e., dragging one folder over another on a Mac)

I have about 600GB of photos myself. My solution is three storage locations: Two 2TB USB HDDs, and a 4TB mirrored ioSafe N2 (fire/flood resistant HDD array). The first 2TB HDD is my day-to-day drive, where I load all of my photos from my flash cards. The ioSafe N2 sits on my network, and my computer backs up the 2TB HDD to it nightly. The other 2TB HDD sits at my office, and I take it home and update it to the current version periodically.

That way, if my house burns down, I'll still hopefully have my nightly backup. If my house is robbed and they find my drives, I'll still at least have my periodic backup.

It's not perfect, with the worst case scenario being theft, but I'm reasonably OK with it.

Edit: As far as a Time Capsule, I use Time Machine and yet another 2TB HDD as a failsafe backup of my laptop. I wouldn't worry about the Time Capsule running all of the time, but I wouldn't use it as my only backup. It's not a RAID device as far as I know.



Jul 19, 2013 at 02:12 PM
rscheffler
Offline
• • • • • •
Upload & Sell: Off
p.1 #3 · p.1 #3 · Photo storage backup choices


600GB is a pretty reasonable amount and not difficult to manage because you can fit the entire collection on single drives. Therefore it would make sense to make duplicates and as mentioned, store at least one off-site. If you have multiple copies at home, keep at least some of those completely disconnected from any devices/power, since power surges can also cause failures.

Something like a Time Machine drive is a good idea to have running constantly as an immediate backup in case you accidentally delete files from the computer, the computer crashes and you need to restore, etc. But I wouldn't use one for a photo archive where most of the data remains unchanged for long periods of time.

I'm personally wary of RAID devices as backups because the RAID system adds another layer of complexity. For example, if it's a hardware RAID, and years down the road the box dies, can you replace it with one from the same manufacturer that will recognize the old array? With simple HDDs, you don't have to worry about this. But, as I've also discovered, it appears at least some external hard drive solutions behave similarly. For example, I bought some Seagate Expansion USB drive enclosures because they were cheaper than buying the bare drives. My goal was to pop them out of their housings to use with my existing drive bays. But before doing so, I formatted a drive and copied over some files, as a test. Everything was fine. After removing the drive from the enclosure and placing it in my usual HDD dock, the contents of the drive, as well as the formatting of the drive, wasn't recognized by OSX, which asked me to initialize the drive. This tells me that the enclosure is not just a simple pass-through housing, but is also adding a layer to the equation. From my understanding, a fairly common problem with external drives is the enclosure going bad. You could recover the physical drive from it, but it may not necessarily be readable by the computer unless it's back in another identical enclosure. Therefore I prefer to remain with bare drives used in drive docks, allowing them to be easily used and recognized by a variety of HDD interface devices.

For cloud storage I would recommend you convert all of the RAW files to higher quality Jpegs with reasonable compression, and upload those. At least this way you will have a version of every photo in the cloud, even though they're not the RAW originals. While you're doing this you might flag your most favorite images and upload the RAW originals to the cloud as well.

For cloud storage, you should deal with an established player who will maintain a longterm presences. Something like Google Drive is probably good. Flickr might also be an option, though recovering those files will probably mean having to download each one individually. I guess Apple now offers something too. PhotoShelter is also good, though probably not the cheapest.

I use Amazon's S3 (Simple Storage Service), which is the actual storage space used by a lot of cloud storage vendors. http://aws.amazon.com/ They're likely to be around for a long time. You pay for how much you use, and it's not too bad. If you opt for the reduced redundancy option, it's 7.6¢/GB/month (though this can vary based on the geographical region of the server group you use - I see you're in Portugal). With about 150GB uploaded, I'm at around $12/month currently. And they're constantly reducing their storage pricing. When I first started a few years ago it was closer to 10¢/GB/month. S3 can be accessed through Amazon's online portal or via an ftp application that supports it. Amazon also has Glacier, which is about 1¢/GB/month, but it's a near-offline long term solution for data you don't anticipate requiring access to, even infrequently.

Hope this helps somewhat.



Jul 19, 2013 at 03:20 PM
15Bit
Offline
• • • • •
Upload & Sell: Off
p.1 #4 · p.1 #4 · Photo storage backup choices


There is no "safe forever" i'm afraid.

- You can write out to archive quality DVD's or Blu-Rays, but they might not last well and who knows how long the hardware will be available to read them.
- Web services are vulnerable to going out of business
- Hard disks aren't "archive" grade either when live or stored on a shelf.
- Tape backups are probably best, but are expensive and again there is the potential problem of equipment being available to read the data in years to come.

There is also the problem of how you do backups. No matter which backup solution used, most of us regard backing up as copying our primary live data to whatever backup medium we have chosen. This involves the assumption that live data is "uncorrupted", but this is not necessarily the case. With big photo collections it is easy to accidentally delete files without noticing, or mistakenly change keywords or edits across lots of files in lightroom etc. Even worse, with increasingly large hard disks and massive quantities of data, the problem of "bit rot" (random data corruption on the disk) becomes more and more prevalent. Over time, if such errors are unnoticed they can propagate all the way through a backup system and reduce it's usefulness.

The only way around such problems is to use online hardware that can detect and correct bit-rot, and to keep permanent "snapshots" of data that are never overwritten. For the former you need to use ZFS (the only file system that is bit-rot tolerant as far as i know) and for the latter you need a *lot* of hard disks / Blu-rays or tapes, especially if you are keeping duplicate off-site backups. Unfortunately using ZFS implies higher than average geek capabilities and lots of storage media costs a lot of money.



Jul 19, 2013 at 03:24 PM
OccAeon
Offline
• • •
Upload & Sell: Off
p.1 #5 · p.1 #5 · Photo storage backup choices


rscheffler wrote:
I'm personally wary of RAID devices as backups because the RAID system adds another layer of complexity. For example, if it's a hardware RAID, and years down the road the box dies, can you replace it with one from the same manufacturer that will recognize the old array? With simple HDDs, you don't have to worry about this. But, as I've also discovered, it appears at least some external hard drive solutions behave similarly. For example, I bought some Seagate Expansion USB drive enclosures because they were cheaper than buying the bare drives. My goal was to pop them out
...Show more

I disagree about RAID devices generally, I've trusted my data to various RAID devices for over a decade. So have innumerable corporations, banks, governments, etc. RAID 1 (mirroring) is probably easiest, because at worst, you can just pull the data from one of the drives.

I also disagree about external drives. I used to be an IT guy, and I've pulled apart dozens of external drives, and I've never run across one that couldn't be pulled and connected directly over IDE or SATA if need be. It's certainly possible, but it seems unlikely to me.

I thought that your cloud storage recommendations make a lot of sense, though. I haven't yet checked out Amazon S3 -- it sounds pretty good.



Jul 19, 2013 at 04:35 PM
safcraft
Offline
• • •
Upload & Sell: Off
p.1 #6 · p.1 #6 · Photo storage backup choices


I'll take the external USB drives route. Just purchased 2x 2Tb WD Elements.
Will keep 1 at home and 1 at brother's home to be safe.
Cheers all!



Jul 21, 2013 at 05:52 PM
SSISteve
Offline
• • • •
Upload & Sell: On
p.1 #7 · p.1 #7 · Photo storage backup choices


I bought 2 WD drives and put them in an external case with a cooling fan. They have an on/off switch so they are only on when I need to access the pictures. I copy new images to one of the drives and then make a backup copy to the other drive. I also have a WD portable drive that I copy all of the images to it once every so often. This gives me 3 copies and I feel pretty safe doing it this way.



Jul 21, 2013 at 10:07 PM
15Bit
Offline
• • • • •
Upload & Sell: Off
p.1 #8 · p.1 #8 · Photo storage backup choices


OccAeon wrote:
I disagree about RAID devices generally, I've trusted my data to various RAID devices for over a decade. So have innumerable corporations, banks, governments, etc. RAID 1 (mirroring) is probably easiest, because at worst, you can just pull the data from one of the drives.


I doubt any of those have trusted their data security to RAID. RAID is an uptime and performance solution, not a backup solution. I can pretty much guarantee that all of those quoted industries actually "trusted" their data to a very expensive magnetic tape system.



Jul 22, 2013 at 01:11 AM
justruss
Offline
• • • • •
Upload & Sell: Off
p.1 #9 · p.1 #9 · Photo storage backup choices


At your current storage size, assumed shooting patterns, and use-- I'd go with multiple HDDs. The more, the better.

My own basic plan is this:

1. Internal drive with full library. (Upgrade drives as required.*)
2. Periodically backed up external drive with full library. (Upgrade drives as required.*)
3. *When upgrading drives, take old library/backup drives out. Label. Package for reasonable storage duration (10÷5 years), and bring offsite (actually off-continent, but previously out-of-state) for retirement/total loss of other systems backup.

Rinse, repeat.

So at any given time I have my internal library and my on-site external backup. Then I have incremental (with big increments), off-site backups for catastrophic losses. Reverting to these incremental backups would still involve losing a big chunk of recent images. Let's say I have new yearly incremental backups. By 10 years out, I expect a significant proportion (over half) of those incremental backups to no longer spin up and/or have been made inoperable by magnetic degradation. I also do very rare DVD backups. These shoooould last 10 to 30 years depending on quality.

An additional layer of cloud-based backup would be great, but it does open its own can of worms, from costs + convenience to the security risks of making it easier for snooping governments to get access to one's images. This last concern probably isn't much of an issue for most people; depending on what your photograph where, and for whom (like journalistic publication), it can be a real issue that needs to be considered.



Jul 22, 2013 at 01:40 AM
charld
Offline
• • •
Upload & Sell: On
p.1 #10 · p.1 #10 · Photo storage backup choices


Try the following from B&H, simple, effective and good price.

Western Digital 4TB My Book Studio Edition II Quad Interface External Hard Drive
http://www.bhphotovideo.com/c/product/625843-REG/Western_Digital_WDH2Q40000N_4TB_My_Book_Studio.html



Jul 22, 2013 at 04:10 AM
OccAeon
Offline
• • •
Upload & Sell: Off
p.1 #11 · p.1 #11 · Photo storage backup choices


15Bit wrote:
I doubt any of those have trusted their data security to RAID. RAID is an uptime and performance solution, not a backup solution. I can pretty much guarantee that all of those quoted industries actually "trusted" their data to a very expensive magnetic tape system.


No kidding. My point is that RAID is still a worthwhile investment, and I'd take a RAID 1 array + backup over a single HDD + backup any day.



Jul 23, 2013 at 08:55 AM
15Bit
Offline
• • • • •
Upload & Sell: Off
p.1 #12 · p.1 #12 · Photo storage backup choices


OccAeon wrote:
No kidding. My point is that RAID is still a worthwhile investment, and I'd take a RAID 1 array + backup over a single HDD + backup any day.


From a redundancy perspective there is actually little difference between those - you have 2 copies of your data in both cases. What you gain with the RAID is 2-3 days of uptime in the event that a hard disk dies and a new one needs to be posted out to you, and you don't have to go through the hassle of restoring from a backup. The cost of that uptime and convenience is the price of a hard disk (for a RAID 1). It's up to you whether that is worth the money.

I'm not knocking RAID, i'm just pointing out that it isn't the solution that many people think it is. I run a RAID 5 at home, because it allows me to make up an array that is larger than a single disk, and because i am willing to pay the extra for the uptime and avoiding the hassle that comes with restoring from a backup. It also allows me to run the ZFS filesystem, which does bring with it a little extra data security in the form of bit-rot protection. That is via the file system though, it is not the RAID that does that.



Jul 23, 2013 at 09:12 AM
nswelton
Offline
• •
Upload & Sell: Off
p.1 #13 · p.1 #13 · Photo storage backup choices


here is my backup strategy. i abandoned raids and now use a bunch of massive drives, CC cloner and copy.com.

http://www.dreamtimeimages.com/blog/best-free-cloud-storage-for-photographers-photography-backup-strategies/



Aug 04, 2013 at 04:05 PM
binary visions
Offline
• • • •
Upload & Sell: Off
p.1 #14 · p.1 #14 · Photo storage backup choices


600gb, while a lot of data, is not so unwieldy as to be totally unmanageable for an off-site backup service, if you have reasonable broadband.

My general opinion is if you have a manual process, somewhere, somehow, it's going to break at the worst possible time. For instance, I had a manual backup process that was: one copy on my computer, one copy in my house but not on my computer, one copy in a safe deposit box.

I got robbed, so both my in-house copies disappeared, and it was at the absolute tail end (actually, I was a little late) of when I was due to make another copy for the safe deposit box. Obviously, some backups were better than none but I still lost a bunch of data.

Consider a service like Crashplan. Online services can go out of business, yes, but you should still have your local data so this shouldn't be a problem. Another advantage of Crashplan is you can back up to a friend's house, so you can "seed" your backup onto an external drive (you won't have to do the whole 600gb across the internet), and then send it to them.



Aug 05, 2013 at 06:45 AM
aubsxc
Offline
• • •
Upload & Sell: Off
p.1 #15 · p.1 #15 · Photo storage backup choices


I use multiple NAS (Network Attached Storage) boxes for redundant storage. My main server (always on with high quality UPS power) currently has about 46TB unformatted storage, running FreeNAS (based on BSD Unix) on bare metal server hardware (16GB ECC memory, Supermicro board with IPMI, Xeon 1230 HT quad, which is overkill frankly). Probably 80% of the space used on this server is media (misic, blu-ray rips etc) that lives on raidz2 (Solaris ZFS Raid 6) pools. The rest is stuff that I cannot afford to lose (about 4TB currently) that also lives on a raidz2 pool.

Everything on Server1 is backed up with snapshots to an almost identical Server2 that also stays home but is only turned on every few days to sync with Server1.

The critical files (about 4TB) get backed up to a small NAS box also running FreeNAS that lives in a friend's home nearby. This gets synced every 3 to 4 weeks depending on how fast changes accumulate.

I built all the NAS boxes and set up the software myself. There are significant advantages to using an enterprise grade ZFS based storage solution and FreeNAS provides an easy to use interface that can be used by pretty much anyone, with access to CLI if you need it. To buy prebuilt servers with similar capabilities would have cost me a ton more money. With 1 or 2 TB of data you don't need a NAS and can work with multiple eSATA or USB3 external drives. This is the easiest option if you don't want to spend time looking into NAS storage.



Aug 05, 2013 at 01:00 PM
15Bit
Offline
• • • • •
Upload & Sell: Off
p.1 #16 · p.1 #16 · Photo storage backup choices


46TB in Raidz2 on enterprise level hardware (with an identical mirror for backups) to primarily store music and movies? I think we have to give you the "excessively geeky hardware overkill award" for this week...

Repeat after me - "I spend too much money on my toys".

PS - I doubt even 5 people here know anything about ZFS...



Aug 05, 2013 at 02:18 PM
binary visions
Offline
• • • •
Upload & Sell: Off
p.1 #17 · p.1 #17 · Photo storage backup choices


46TB is more than I have in the production SAN at work. Sheesh. I bet you don't have 8x 10Gbps connectors for distributing it, though

I had two (development) FreeNAS ZFS arrays eat themselves with no discernible cause, so I now stay away from FreeNAS. Not that I would necessarily say everyone should do so, given my very small sample size (5-6 different FreeNAS boxes built, 2 failures), just my experience and YMMV...

Typically, I don't recommend that casual users build their own storage systems like this, even with setups like FreeNAS that can be put together without deep knowledge, primarily because there are many moving pieces to fail and troubleshooting becomes a chore if you aren't familiar with computers. The all-in-one systems are a little simpler, with a single vendor to go to if there's a problem.



Aug 05, 2013 at 06:30 PM
15Bit
Offline
• • • • •
Upload & Sell: Off
p.1 #18 · p.1 #18 · Photo storage backup choices


Hmm, that's not such great news - I just moved over to ZFS for my home (ubuntu) server. I'd rather it didn't die...


Aug 05, 2013 at 11:50 PM
binary visions
Offline
• • • •
Upload & Sell: Off
p.1 #19 · p.1 #19 · Photo storage backup choices


Well... like I said, there are tons of ZFS zealots out there, some who I'm sure are smarter than I am, and lots more FreeNAS distributions than my own. I have no doubt that my experience was an anomaly - I just don't know how MUCH of an anomaly. Just one of those once bitten, twice shy things for me.

In both cases, the servers appeared to be healthy, the drives were not in a failed state (and I went on to use them afterwards), and the ZFS status reported thousands of files with permanent errors.

I have switched my couple remaining server-based NAS setups over to OpenFiler, which does not support ZFS but the servers have good quality RAID cards and caches in them.



Aug 06, 2013 at 06:47 AM
Sami Ruusunen
Offline
• • •
Upload & Sell: Off
p.1 #20 · p.1 #20 · Photo storage backup choices


I've been happy with my current backup system:

- hourly time machine backup to time capsule which is located in a fire safe box
- carbon copy clone once a week from entire hardisk to external HD, which I keep at work

(I've had 2 complete hard disc failures since I started digital photography, which required entire system re-installed, so luckily I had a backup like this)



Aug 06, 2013 at 07:15 AM
1
       2       end




FM Forums | General Gear-talk | Join Upload & Sell

1
       2       end
    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.