Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | General Gear-talk | Join Upload & Sell

1
       2       end
  

Archive 2015 · Storage, backup and archiving

  
 
StarNut
Offline
• • • •
Upload & Sell: Off
p.1 #1 · p.1 #1 · Storage, backup and archiving


Hi,

I'm not a photography pro, but I take a lot of digital images.

These quickly fill up my hard drive.

So I'm looking for a reasonably efficient (meaning fairly easy and fairly automated) method of storing my images of of my hard drive, so that (i) they are readily available; (ii) I am properly backed up, and (iii) I ultimately can archive them safely when I'm sure I'm done with them.

At present, I keep them on my hard drive until I need space, then move them to a couple of external hard drives. I back up every night onto an external hard drive.

But this is a bit clunky, since I now have a large collection of external hard drives.

Is there anything better that doesn't cost a fortune?

Thanks.

Mark



Feb 10, 2015 at 11:56 AM
jcolwell
Online
• • • • • • •
Upload & Sell: On
p.1 #2 · p.1 #2 · Storage, backup and archiving


I use multiple internal and external hard drives. All of my digital photos and scanned MF/LF negs and transparencies going back to 2004 are on both internal drives and external 'mirror' drives. Every time I download an image card, I put on the current internal archive disk, and then I copy the files from the internal drive to the 'matching' external drive before formatting the image card. All of my image folders use a yyyy-mm-dd name format (e.g. 2015-02-10), and I have a spreadsheet where I record the dates of all activties and jobs. It's really easy to find stuff.

I started doing this with 100 GB and 200 GB drives in 2004. For a while, every time I filled a disk I'd get a pair of new ones with each having at least twice the capacity of the previous ones, and start the new archive drives by copying the old, full drives into the replacements. I now have a 7 TB archive across three internal drives in my desktop, plus two more internal drives for OS and work space, and another copy of the 7 TB archive across three external drives that are stored in a fireproof box. The archive disks are Western Digital 'green' SATA 3.0 drives.



Feb 10, 2015 at 01:17 PM
Daniel Smith
Offline
• • •
Upload & Sell: Off
p.1 #3 · p.1 #3 · Storage, backup and archiving


Portable hard drives work but give no protection against major magnetic events. CD's, DVD's and BluRay discs give protection against that but store a lot less and do get scratched if not taken care of.

A combination makes sense for many.

Not sure how the newer Solid State drives fare in magnetic events like major solar storms that hit us from time to time.



Feb 11, 2015 at 10:48 AM
StarNut
Offline
• • • •
Upload & Sell: Off
p.1 #4 · p.1 #4 · Storage, backup and archiving


Thanks, guys!

I'm pondering putting together a very large RAID array as a solution.

This is why I don't want more pixels!

Mark



Feb 11, 2015 at 11:49 AM
atwl77
Offline
• •
Upload & Sell: Off
p.1 #5 · p.1 #5 · Storage, backup and archiving


Yeah, RAID is the practical way to go. If you're putting them in a NAS, four or more Western Digital reds in RAID 0+1 would be my suggestion.

StarNut wrote:
This is why I don't want more pixels!

Just a little side rant, as this reminds me of something: it often irks me when some people give a legitimate reason against more resolution, and the standard/canned response is "storage is cheap, buy more". That's easy to say but doesn't really consider the full logistics of it, including transferring and backups like what we're discussing now.

Edited on Feb 12, 2015 at 06:04 AM · View previous versions



Feb 11, 2015 at 10:40 PM
ReyGay
Offline
• • • •
Upload & Sell: Off
p.1 #6 · p.1 #6 · Storage, backup and archiving


Have you ever guys thought of what to do if your house get burglarized - all of that back ups gets stolen... Or, during a major catastrophe... All of these back ups of backs up are left at home or on some form of online cloud storage. Do you have a back up that you carry around with you? In a major worldwide catastrophe, I don't think it would be easy to access online cloud storage. I now am planning to carry everyday a lightweight hdd back up such as the WD My Passport Ultra where ever I go, just in case some crazy event happens.


My Flickr Gallery
My interesting photos on Flickriver
my Deviantart gallery



Feb 12, 2015 at 01:48 AM
atwl77
Offline
• •
Upload & Sell: Off
p.1 #7 · p.1 #7 · Storage, backup and archiving


I think in the event of a worldwide catastrophe, access to cloud storage would be the least of your worries.

In any case, I do carry a 2TB WD My Passport as my "off-site" backup.



Feb 12, 2015 at 05:58 AM
ben egbert
Offline
• • • • • •
Upload & Sell: Off
p.1 #8 · p.1 #8 · Storage, backup and archiving


I have 302 cd's going back to 1999 (yes they started out on floppies and got transferred later. This is insane because it include all images, I recently reduced this to just the keepers.

I have 3 HD's attached to my computer and three internal. I upload my cards using Breeze uploader pro which renames them by date and creates day date folders.

I use 2nd copy (an automated backup program) to backup my Photoshop working files and all other important data to an external HD. Because of my work flow, I will have several folders of images. Web, print, a master PSD and the raw keeper folder. I now have other folders for HDR work, so I might have as many as 8 copies of any image that is still in 16 bit full size.

I don't use raid, its over my head. I just use WD hard drives and I have only had one fail in the last 20 years.




Feb 12, 2015 at 11:57 AM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.1 #9 · p.1 #9 · Storage, backup and archiving


I would not trust many CDs from 1999. Are the files stored with some crc and redundancy? Do you test them? I'd put all 303 of them on other storage media. It's less than 240GB.

EBH



Feb 12, 2015 at 11:51 PM
Chris Anthony
Offline
• •
Upload & Sell: On
p.1 #10 · p.1 #10 · Storage, backup and archiving


I agree, I have many CDs from 03, 04 and 05 that are basically useless now. I can't access most of the images on them. Its sucks and really makes you wonder about the long term viability of any digital storage system.


Feb 13, 2015 at 12:55 AM
ben egbert
Offline
• • • • • •
Upload & Sell: Off
p.1 #11 · p.1 #11 · Storage, backup and archiving


EB-1 wrote:
I would not trust many CDs from 1999. Are the files stored with some crc and redundancy? Do you test them? I'd put all 303 of them on other storage media. It's less than 240GB.

EBH


They are also on HD's. I have two HD's that are full and disconnected. I test the CD's frequently, (but not all of them) the CD's are often easier to find a specific date than is hooking up an old HD. None of my images prior to about 2008 are of much interest except family photos.

I should also point out, the archive that will survive in 50 years will be the print. None of my heirs are going to keep transferring this stuff to modern technology as the old becomes obsolete like floppy's for example. All of my important family images are printed.

I have never had a problem with a CD.



Feb 13, 2015 at 10:00 AM
Ghostinz
Offline
• • •
Upload & Sell: On
p.1 #12 · p.1 #12 · Storage, backup and archiving


Just to revisit.... I catalog all my photos my month and year. Then backup them up to too two external drives. One that stays local, and one that I give to family off site. I only backup the offsite drive about once a quarter.

Then I burn every month to DVDs as well. It might be a bit much, but several years ago, while I was married, I had my main computer crash. I was able to recover my images, but the scare was too much!!

My next move is to update some of my older drives with some newer 1 TB and 2 TB portable drives. They are about $60 now.



Feb 15, 2015 at 11:54 AM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.1 #13 · p.1 #13 · Storage, backup and archiving


ReyGay wrote:
Have you ever guys thought of what to do if your house get burglarized - all of that back ups gets stolen... Or, during a major catastrophe... All of these back ups of backs up are left at home or on some form of online cloud storage. Do you have a back up that you carry around with you? In a major worldwide catastrophe, I don't think it would be easy to access online cloud storage. I now am planning to carry everyday a lightweight hdd back up such as the WD My Passport Ultra where ever I go, just
...Show more

Sometimes I maintain a portable HDD or SSD with differential backups in the car or at work until I have a chance to update my offsite archives. Online does not work for me due to the dozens of TB of files.

EBH



Feb 15, 2015 at 12:53 PM
paulkris
Offline
• •
Upload & Sell: On
p.1 #14 · p.1 #14 · Storage, backup and archiving


At home I use multiple hard drives for storage and back up. All are SATA type drives and most are 2TB. I have a working drive and two separate back up drives on all files.

When I travel, and most of the trips are 30 days+, I store on my laptops 1tb drive, on an outboard 500GB LaCie drive and I also keep the original memory cards. Cards are so cheap today that I prefer not to reuse or edit them until I'm safely home.
Cloud storage isn't an option for me as my internet access is usually very slow when I'm traveling and only slightly better when I'm at home.



Feb 15, 2015 at 05:48 PM
amlsml
Offline
• • • •
Upload & Sell: On
p.1 #15 · p.1 #15 · Storage, backup and archiving


Has anyone used a Qnap Nas? Any issues? thaks


Feb 16, 2015 at 06:54 PM
rscheffler
Offline
• • • • • •
Upload & Sell: Off
p.1 #16 · p.1 #16 · Storage, backup and archiving


I think RAID would be a reasonable near-term solution for quick access to an archive, where everything appears in a single volume, or logical set of volumes, and which depending on the system chosen, may be continuously expandable in capacity. But in itself, a RAID array is complex and is not considered to be a good archive solution. Basically, you will still need to have a back-up of some sort for the RAID. It's worth doing some research on this. Things that scared me off RAID were problems with controller cards or enclosures and their controller cards going bad and no longer being supported by the manufacturer to where it would no longer be a simple matter of putting the drives in a new box and just turning it on. This is in addition to the usual drive failure concerns.

My feeling is the more automated and simpler the solution becomes, the less control the user has over what's happening behind the scenes, especially when a problem arises. For example, Apple's Time Machine is very useful, slick, and simple. But I've run into problems with it where the archive was not longer accessible or reliable, and I couldn't really figure out why, at least not easily. I've since switched to using Carbon Copy Cloner (CCC) for regular OS backups, which it keeps in a structure identical to the original, also allowing it to be bootable, if needed.

I've stuck to my own variation of a 'JBOD' system that many others also seem to have adopted. I transfer off the memory card to a ~500GB SSD that contains currently active projects. SSD for speed of access, it's easy to bring along to offsite events/jobs and theoretically is more durable in such applications. I'll immediately use CCC to back up the new project to two external drives. As I work on the project, I use CCC to copy over just the updated files. It's set up to move the older files on the back-up drives into 'archive' folders that can be revisited in case a recent update breaks something, or I need to recover a file deleted from the working drive. When the project is finished, I'll do a final clone/back-up to the two drives, where one of the drives is an archive specific to the type of project, while the other fills with projects of all kinds on a chronological basis. After being certain everything has copied over correctly, I'll delete the project from the SSD. I also keep a set of low-rez captioned and keyworded images on the OS drive in a folder with the same name as the original, which become searchable with Spotlight. The low-rez files, like the originals, are named in a YYYYMMDD_XXXX format, with the project folder also beginning with YYYYMMDD. It's easy to do a Spotlight image search based on a keyword, determine when the image was shot by the filename or EXIF/caption, and trace it back to the appropriate hard drive(s). Or, sometimes if it's just a file or two, it's easier to pull it from the cloud, especially if I'm not home. I also use the same naming/filing system for backing up final processed files to the cloud. In my case, to Amazon's AWS S3 service, with some duplicates also residing in dedicated Google Drive accounts, which I will use for certain client deliveries, where a web gallery is desirable.

Over the longterm, a key seems to be to continuously migrate data forward, to new drives and storage mediums, as they become established and reliable. I agree that diversification of backup media is a good idea and I'm no longer sold on CDs/DVDs for some of the reasons stated above. They just don't hold much and take up a lot of space. A collection of cheap micro SD or simple SDHC cards, written once, might be a better solution to complement the usual HDDs. Here the biggest concern I have over migration is the introduction of bit rot due to physical media decay, which then carries over with future migrations. There are solutions to this, such as cloning solutions that check against the original, or the ZFS file system, but right now I'm playing the odds that it won't become a critical problem.

As already hinted, I'm relying on the cloud as a last resort, offsite solution, to which I backup only the final processed Jpeg selects from every shoot. Obviously this overlooks the original RAWs, but IME, I am unlikely to revisit 99% of my RAWs after the first time through the RAW converter. This might change, but at present the storage and bandwidth requirements to backup even just the RAW selects is not feasible for me.



Feb 16, 2015 at 07:46 PM
Michael White
Offline
• • • •
Upload & Sell: Off
p.1 #17 · p.1 #17 · Storage, backup and archiving


My best advice is get the dam book. Data asset management book. It is the best resource for designing and implementing a secure redundant asset library.


Feb 22, 2015 at 03:25 AM
Sarsfield
Offline
• • •
Upload & Sell: Off
p.1 #18 · p.1 #18 · Storage, backup and archiving


Has anyone used a Qnap Nas? Any issues? thaks

My first NAS that I purchased is a Qnap TS-419P+ that is still going strong 24/7 since the day I bought it in 2012. I use it in Raid 5 configuration and did have one disk go bad after about a year that was simple to change out and the Raid rebuilt itself in less than 24 hours. They do a very good job with keeping their firmware and applications current and they have a very active and helpful forum. I have never had to use their customer support but I have read that they are responsive and helpful in times of need. I also have a Synology DS412+ that also runs 24/7 and I have had no major issues with it either. As far as I'm concerned, both companies provide very nice home NAS's that are fairly easy to set-up and use. I would not hesitate to buy another one from either company. By the way, there's a very nice Qnap TS-469 Pro being offered on the tech b+s board for a great price. I almost bought it twice to have another backup system. Let me know if you have any other questions about my home NAS's. I'd be glad to offer my long winded advice!



Feb 22, 2015 at 05:22 AM
ReyGay
Offline
• • • •
Upload & Sell: Off
p.1 #19 · p.1 #19 · Storage, backup and archiving


We just hope our files still even are recognised in 50yrs time. I just print what I can, on Blurb books and on A2 prints and archive them. I want to enjoy my flies in many years to come....but how long will our civilisation last?


My Flickr Gallery
My interesting photos on Flickriver
my Deviantart gallery



Feb 23, 2015 at 02:51 AM
photo1a
Offline
• • •
Upload & Sell: Off
p.1 #20 · p.1 #20 · Storage, backup and archiving


I use duplicate external backup drives. That is, I have the same photos on two different drives. I use a program called SyncBackSE for backing up from my desktop. I keep one backup hard drive in a fireproof (resistant) safe at home, the other in a safe at a friend's house.


Mar 28, 2015 at 04:29 PM
1
       2       end




FM Forums | General Gear-talk | Join Upload & Sell

1
       2       end
    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.