usb flash as long term storage/archive

Moderated by: guardian

Visit the FM Store · Image Upload · Buy & Sell
FM Forum Rules

FM Forums | PRO digital corner | Join Image Upload

2 end

alexhibbert
Offline
Image Upload: Off

p.1 #1 · usb flash as long term storage/archive

What are people's thoughts on using usb flash drives as a long term archival solution in preference to an extra HDD, online, DVD or Blu-ray?

This is for archive (i.e. offsite and permanent data record), NOT for daily backup (for which a local external hard drive is ideal.)
They seem to be
1 safe for 10years+ (so say the reviews) and resilient
2 large (16-32Gb common) without putting too many eggs in one basket.
3 quick compared to burning dvds etc.

4 fine for write cycles as for archival you may only write to them a few times and then activate the write protection.
5 compact

I'm looking for a quality offsite archival system to compliment my onsite HDD backup. I'm experimenting with Taiyo Yuden DVD+R but it's SUCH a long process with no certainty of long term reliability. Blu-rays are large but just as potentially unreliable and even slower to burn.
Any thoughts?

Thanks.

Aug 17, 2008 at 12:07 PM

haijak
Offline
Image Upload: Off

p.1 #2 · usb flash as long term storage/archive

Crazy expensive. A couple of hundred bucks for 32GB. For that kind of money you should look into online services. They offer access guarantees and insurance in some cases. They you don't have to worry about managing it at all.

Edited on Aug 17, 2008 at 01:37 PM

Aug 17, 2008 at 01:29 PM

Scott Sewell
Offline
Image Upload: On

p.1 #3 · usb flash as long term storage/archive

haijak wrote:
Crazy expensive. A couple of hundred bucks for 32GB. For that kind of money you should look into online services. They offer access guarantees and insurance in some cases. They you don't have to worry about managing it at all.

Even a stack of hard drives would be a much more cost effective solution. Hard drives are inexpensive to the point you could double your back up power and really not be taking up much physical space.

I guess if you've only got a few GB of photos to back up--and that's all you ever expect to have--a flash drive might be an option. But I'm going on 2TB (and growing) and cringe to think of trying to get all that on those little flash drives! What a nightmare!

Aug 17, 2008 at 01:56 PM

alexhibbert
Offline
Image Upload: Off

p.1 #4 · usb flash as long term storage/archive

hmm. i see the point on expense - although a 32Gb drive from OCZ or Corsair (top makes) are £70 (~$140?). It's a lot but maybe necessary for images to survive long term. They are a lot more practical and faster than optical discs - which I previously resolved to use but am now not sure about.

I guess the key is that I'm trying to use 2 different technologies for backup and archiving in order to spread the risk. Using a 3rd HDD is clearly the cheapest and fastest option, but it doesn't stop corruptions or human errors (deletions etc.) being copied across primary, backup and archival drives.

I'm after a certain level of permanence (write-once DVD+Rs?) with my archival so that it can recover data should something awful happen in my workflow. A third HDD just doesn't inspire enough peace of mind.

Online archival sounds great, especially with these $5/month unlimited deals, but the bandwidth speed is the problem. Only a few Gb would take days.

Any more thoughts?

Aug 17, 2008 at 03:28 PM

Hammy
Offline
Image Upload: Off

p.1 #5 · usb flash as long term storage/archive

Other notes:
http://www.fredmiranda.com/forum/topic/652853/0

$140 will buy you a Terabyte in a few weeks. That's 30x more space for the money.

As others noted, if you have time and money to keep up with all those little peices - storing them so they are not lost/damages, filing them so you don't have to go through all of them - the time to xfer to all of them, etc...

Bottom line is that any media you choose, the format will die out before the data does. You'll have to migrate to the next thing in 10-15 years anyway. Make it as cheap and easy for yourself now - with proven media.

Hammy.

Aug 17, 2008 at 08:00 PM

dwayne anderso
Offline
Image Upload: Off

p.1 #6 · usb flash as long term storage/archive

I have a seagate 160gb exsternal drive with many years of important photos that no longer works, the drive is not recanized by my computer at all and seagate wants $2,500.00 to retrieve my data, so i have no trust in just storing photos on exsternal drives only, even though they cost a lot of money, i now make several copys on dvd, but who knows how long they will last, Its very time comsuming but its a bad fealing when you no longer have your family photos.

Aug 17, 2008 at 08:10 PM

Nathan Whitchu
Offline
Image Upload: Off

p.1 #7 · usb flash as long term storage/archive

dwayne anderso wrote:
I have a seagate 160gb exsternal drive with many years of important photos that no longer works, the drive is not recanized by my computer at all and seagate wants $2,500.00 to retrieve my data, so i have no trust in just storing photos on exsternal drives only, even though they cost a lot of money, i now make several copys on dvd, but who knows how long they will last, Its very time comsuming but its a bad fealing when you no longer have your family photos.

Which is precisely why most of us make multiple copies of data on at least a couple types of media. The only thing worse then losing your family photos is losing business and getting a bad reputation for not delivering the product you promised your client. I have two copies of files on the computer I do PS work on, another copy on the computer at the office I wind up ordering photos on and an external HD that I use to transport the photos. On top of that when I'm done editing I'll burn a copy to DVD and file it. Hard drives are cheap, buy a new set every couple of years and count it as a business expense. For a couple hundred bucks you could probably double your storage space every year.

Aug 18, 2008 at 01:53 AM

Kyle Yates
Offline
Image Upload: Off

p.1 #8 · usb flash as long term storage/archive

Note here we are talking about ARCHIVE not short term data backup so you shouldn't need to retrieve the data very often.
With Hard disks being really cheap now there really isn't any reason NOT to use these for short term backup -- you don't have to have these online all the time either.

If you need to access the data regularly then you aren't really talking about ARCHIVE but about BACKUP which are two totally different animals. In EITHER case always make duplicate copies as the chance of BOTH copies going bad at the same time is fairly negligible.

For the ultimate PROVEN longevity you could use DVD-RAM. Quite fast now (used to be slow but can now do up to 8- 12X) but the drawback is capacity is limited to only 4.7GB per side or 9.4GB per double sided disks. Media is easily sourced and is only slightly more expensive than standard DVD+/-RW discs -- and if it's your permanent archive the extra expense is well worth the money.

Guaranteed to last over 200 years (enough tests all over the web on this) and re-writable -- more than 200,000 times unlike DVD+/- RW.

The re-writable idea is also OK as you can re-use the media when updating the archive.

These however DO make excellent ROBUST prefessional quality SAFE RELIABLE archive storage. They come in 2 flavours - Cartridge type and Non-cartridge type. For Archive you might consider the Cartridge type but burners with cartrige caddies in them are increasingly hard to find. The non cartridge types are easier to get hold of and with decent handling should be just as reliable.

http://en.wikipedia.org/wiki/DVD-RAM

I've been looking at BLU-RAY -- the problems with this type of device are :

a) the technology is fairly new so the suitability for long term archive hasn't been really tested yet
b) both media and writers seem to be currently hard to find.
c) the current size is 25GB but 50 GB is in the offing (coming shortly) so I'd hold on before going the Blu Ray route.
d) if you are at a different computer and need to retrieve data from your archive you will probably not be able to until BLU RAY devices become more widespread. In the current economic situation rushing out and buying totally new hardware is probably the last thing on people's minds so I'd suspect you would have to wait at least 1 -2 years before Blu Ray becomes as ubiquitious as the current DVD devices.
e) speed is fairly slow but of course it's early days yet. I'm sure generation 2 or 3 will have this speeded up by a number of times.

The media comes in two types - BD-R (like DVD+/-R only write once) or BD-RE which is re-writable.
50GB archive media on the size of a standard DVD sounds attractive for permanent archive so this might make a decent archive system in the near future.
(note for daily backups / very short term archive external hard drives are ideal as they come in small footprints with around 500 GB capacity and are "self powered" via the computers's USB port. These are really portable too -- but as discussed these really aren't really suitable for PERMANENT / LIBRARY / VAULT type archives).

Large commercial organisations such as big banks etc still use TAPE but for home / small business users I wouldn't go down this route - expensive, fiddly to use and no random access so it's often a problem just to restore a particular file - particularly from a multi-volume tape backup -- you often need to restore the entire data .

Incidentally don't go the Online route -- the problem here is not the amount of data you are allowed to store -- the online vendors are quite cheap in this regard -- but most people and businesses that have to rely on an ISP or cable company for their Internet access will find that these ISP's / Cable companies are moving away from the unlimited up / download models to a "capped" amount such as 10 GB per month.

This is actually rather a nasty way of charging since you'll eat up this limit really quickly with ANY type of decent quality Video streaming --especially HD (High Definition) so you'll be paying many 100's of USD's. GBP's ,EUR's, Roubles etc etc. per month and even with a 16MB download connection the upload will be much much slower and just isn't worth even considering .

The "Capped" model is one of the nastiest techniques these ISP's have of getting a HUGE price increase through the back door. -- Audio streaming doesn't consume much bandwiidth so users often think a 10GB per month allowance is plenty but with more and more people wanting instant access to high quality and High Definition Video -- watch the howls start as people's bills go through the roof.

I'm lucky as I still have an original "Uncapped" account for my home ISP --whilst the tariff is slightly higher I'm pleased with it.
New customers aren't even offered this option any more. I'm sure I'm not alone in experiencing this.
Cheers

-K

Edited on Aug 18, 2008 at 08:23 AM

Aug 18, 2008 at 07:42 AM

Hammy
Offline
Image Upload: Off

p.1 #9 · usb flash as long term storage/archive

Kyle Yates wrote:For the ultimate PROVEN longevity you could use DVD-RAM. Guaranteed to last over 200 years (enough tests all over the web on this) and re-writable -- more than 200,000 times unlike DVD+/- RW.

Kyle,

Not to be argumentative about this, but how can anyone (on the web) back up a media to last 200 years when it is only 10 years old?

http://en.wikipedia.org/wiki/DVD-RAM

I know, they have tests and such like Wilhelm for prints. But those test were under specified conditions - which not everybody necessarily adheres to.

More importantly, what drives will be available to read it in 50 years? Remember 8" floppy discs? Maybe 5.25" - certainly everybody still has 3.5" discs in their computer

Like you said, hard drives are too cheap and easy to use for now. And depending on how much media one has to burn/catalog and store, another media may suite one's fancy. We all just have to make sure that we migrate that data to the next best thing every 15 years or so before there are no drives or connections to read the format our data is currently on.

Hammy.

Aug 18, 2008 at 03:32 PM

alexhibbert
Offline
Image Upload: Off

p.1 #10 · usb flash as long term storage/archive

thanks. but this is not really tackling the issue of a tertiary system for archiving. no-one is disputing the fact that HDDs are the best for backup onsite every day. what i was trying to discover is the best thing for offsite, infrequent archiving.

alex.

Aug 18, 2008 at 06:05 PM

mdude85
Offline
Image Upload: On

p.1 #11 · usb flash as long term storage/archive

What is meant by "long-term"?

Aug 18, 2008 at 06:15 PM

TJ Asher
Offline
Image Upload: Off

p.1 #12 · usb flash as long term storage/archive

This is an interesting thread.

I have heard of people who buy cards for a job and they are never used again for any other job. The cost is, of course, passed along to the client.

I don't do it this way but it is an intriguing idea.

Aug 18, 2008 at 06:23 PM

jay tieger
Offline
Image Upload: Off

p.1 #13 · usb flash as long term storage/archive

Aug 18, 2008 at 10:14 PM

ajkessler
Online
Buy and Sell: On

p.1 #14 · usb flash as long term storage/archive

alexhibbert wrote:
thanks. but this is not really tackling the issue of a tertiary system for archiving. no-one is disputing the fact that HDDs are the best for backup onsite every day. what i was trying to discover is the best thing for offsite, infrequent archiving.

alex.

...Hard drives are also best for tertiary off-site, infrequent archiving. Unless you want something that remains offsite (ie you connect to over the web, which usb drives won't provide but HDD's could provide) HDD's are still your best option. Archive your stuff, and leave it at your office, home, or wherever 'offsite' happens to be. By far the easiest and cheapest way.

HDD's don't magically melt down. They'll last a plenty good long time if they're only run infrequently. Of course you'll need to retrieve and re-backup eventually, but as media changes you'll end up doing this anyways. Seems like you're making this much harder than it is...

Edited on Aug 19, 2008 at 01:04 AM

Aug 19, 2008 at 01:02 AM

markle
Offline
Image Upload: Off

p.1 #15 · usb flash as long term storage/archive

I used to rely on CDs then DVDs then when I had the need to get something back they gave me a miserable rate of success.

I started backing up some 5 years ago: after two years the media is not reliable.

by far the hard disk give a better (much better) performance.

now you can get a cable that will connect any HD via USB (IDE,SATA,3.5",2.5" ..anything)

the cost of several hard disks of medium capacity (dunno why but I trust the smaller size more then the big ones) is irrelevant and way more reliable.

that's how I back up now.

flash cards are too small.

Aug 19, 2008 at 05:36 AM

Kyle Yates
Offline
Image Upload: Off

p.1 #16 · usb flash as long term storage/archive

Hi there

I still stand by my statement that at the present time DVD-RAM is the best media for LONG TERM archiving for typical home and small business users.

External HD's are fine for short term backup but not for permanent archive usually stored in a vault or off site.

Longevity tests can be done in a Lab. It's very easy to simulate conditions of aging etc etc.

Of course nobody has actually seen the DVD RAM devices last 200 years but the Lab tests are valid enough.

Unfortunately not many people do Engineering or Science these days so I'm afraid a typical response would be --nobody's been around for 200 years with these devices so how do they know they will survive that long.

Whilst the above question is valid and the only 100% certain method is to actally wait 200 years, decent Lab testing conditions wil yield a 99.9999999% good enough answer -- more than OK for me.

I still say at the current time for off site permanent Archive use Double sided DVD-RAM discs -- 9.4 GB per double sided disc.

When Blu Ray matures a bit more then have a look at that.

Incidentally always duplicate your archive and store one copy in a different place.

Before filing away also check the archive is good (try restoring to a temporary disc.

For archiving use some decent backup software such as arconis true image

http://www.acronis.com

You'll need also to keep some record of what's on what media. For smallish archives you can use EXCEL or make your own catalog via something like MySQL database (Free).

Otherwise invest in some cataloging software -- this will read the DIsc volume name and the files and directories on it.

I find even for a largish archive a home gron MySQL application is fine.
I label the archive Discs before archiving the data then cut and paste the directory names and files into my database.

(Incidentally if you do this backup your data base as well otherwise you'll have a load of DVD's around without a clue as to what's on them or when you made the archive.

Cheers

-K

Edited on Aug 19, 2008 at 06:08 AM

Aug 19, 2008 at 06:03 AM

panos.v
Offline
Image Upload: Off

p.1 #17 · usb flash as long term storage/archive

Easiest thing is a couple of hard drives that act as the backup to your computer. You could have a disk always on doing a backup and another one doing a monthly/weekly/whatever which you plug-in once, take it to your favourite nuclear bunker and that's that. This way you have instant access. DVDs might be good for "200 years" but I know that they last a lot less as I've had discs that failed. And trying to find anything on a DVD is just a nightmare.

In fact, my day-to-day backup is handled by TimeMachine in OS X. Plug in the disk, turn it on every time I do some major work and there you go. No DVDs, no faffing about. Do a backup to another disk every now and take it off-site and that's that. 3 copies, two locations, instant access to everything. If anything goes wrong, and they all die at the same time, it's probably a nuke going off so I'm not sure I'll care that much about the photos.

Aug 19, 2008 at 11:21 AM

alexhibbert
Offline
Image Upload: Off

p.1 #18 · usb flash as long term storage/archive

that's certainly a cheap and simple solution, but it doesn't provide a solution that protects against human error/corruption. For example, accidental deletions on the two onsite HDDs will be replicated on the offsite copy without you knowing - then it's too late.

Aug 19, 2008 at 11:28 AM

panos.v
Offline
Image Upload: Off

p.1 #19 · usb flash as long term storage/archive

With TimeMachine you can restore to a particular time, overcoming deletion problems. I'm sure other types of backup exist that can do that.

And there is always only that much you can do before you get to user error. There's nothing stopping you not writing the right files to the DVD in the first place.

Edited on Aug 19, 2008 at 12:39 PM

Aug 19, 2008 at 12:39 PM

mdude85
Offline
Image Upload: On

p.1 #20 · usb flash as long term storage/archive

Kyle Yates wrote:
Unfortunately not many people do Engineering or Science these days

Huh?

Anyway, the companies producing these DVD RAM discs, such as Panasonic, typically rate their archival at around 30-60 years. I've not seen a company release a white paper or anything similar quoting something as high as 200 years. I think that is just hearsay. Many libraries and archival agencies, such as the National Archives of the USA and Australia still use CD-R and DVD-R to record as these discs have life expectancies of 25-75 years at least. It's easy to imagine that an archival system having nearly infinite life expectancy will come along in the next several decades and our current archiving forms will become obsolete.

In order to test the life expectancy of discs, the companies perform short term tests and then extrapolate those results using variations of the Arrhenius equations, which are principles relating the effect of such variables as temperature on chemical reaction rates. The breakdown of these chemicals as they react is one cause of data loss over time.

Edited on Aug 19, 2008 at 08:16 PM

Aug 19, 2008 at 08:14 PM

ajkessler
Online
Buy and Sell: On

p.1 #21 · usb flash as long term storage/archive

Kyle Yates wrote:
Hi there

I still stand by my statement that at the present time DVD-RAM is the best media for LONG TERM archiving for typical home and small business users.

External HD's are fine for short term backup but not for permanent archive usually stored in a vault or off site.

Longevity tests can be done in a Lab. It's very easy to simulate conditions of aging etc etc.

Of course nobody has actually seen the DVD RAM devices last 200 years but the Lab tests are valid enough.

Unfortunately not many people do Engineering or Science these days so I'm afraid a typical response would be --nobody's been around for 200 years with these devices so how do they know they will survive that long.

Whilst the above question is valid and the only 100% certain method is to actally wait 200 years, decent Lab testing conditions wil yield a 99.9999999% good enough answer -- more than OK for me.

I still say at the current time for off site permanent Archive use Double sided DVD-RAM discs -- 9.4 GB per double sided disc.

When Blu Ray matures a bit more then have a look at that.

Incidentally always duplicate your archive and store one copy in a different place.

Before filing away also check the archive is good (try restoring to a temporary disc.

For archiving use some decent backup software such as arconis true image

http://www.acronis.com

You'll need also to keep some record of what's on what media. For smallish archives you can use EXCEL or make your own catalog via something like MySQL database (Free).

Otherwise invest in some cataloging software -- this will read the DIsc volume name and the files and directories on it.

I find even for a largish archive a home gron MySQL application is fine.
I label the archive Discs before archiving the data then cut and paste the directory names and files into my database.

(Incidentally if you do this backup your data base as well otherwise you'll have a load of DVD's around without a clue as to what's on them or when you made the archive.

Cheers

-K

Even assuming everything you claim is true (which I don't believe anybody is really making 200 year disks anyway) the 'archival' super long lasting disk that are gold plated are several bucks a piece! Plus it takes for freaking ever to write 10gb of data to a cd. With $/gb getting under 15 cents, it's simply easier and cheaper to buy a 500gb HDD and treat it like CD. Fill it up, stick it in a safe and replace it every 5-10 years.

Aug 19, 2008 at 09:31 PM

Kyle Yates
Offline
Image Upload: Off

p.1 #22 · usb flash as long term storage/archive

External HDD's certainly have their place -- no probs at all provided you have enough of these and can store them off site individually. Most people don't use HDD's this way.

DVD-RAM at 8X write doesn't take that long.

You also need to consider the Physical space needed for your archive and how do you retrieve the files.

HDD's as archive need to be write-protected - this is automatic with DVD-/+R. DVD-RAM can set a Write protect mechanism as well.

You can lose data quite easily by accidentual erasure -- easily done.

The main problem of people using external HDD's as archives is that these aren't archives at all but backups. Nothing wrong in this but once you start keeping these disks online you are going to lose files or certainly you'll have files on the external hard disk and on your "real working Disks" out of sync with each other.

An archive should only be accessed VERY RARELY.

Cheers

-K

Edited on Aug 19, 2008 at 09:54 PM

Aug 19, 2008 at 09:51 PM

The Image
Online
Image Upload: Off

p.1 #23 · usb flash as long term storage/archive

You might also want to ask that question on these forums too Alex www.thedambook.com

Aug 19, 2008 at 10:08 PM

Hammy
Offline
Image Upload: Off

p.1 #24 · usb flash as long term storage/archive

removeable discs...

- open
- insert
- find 4.7Gb data - tweaking to fit
- write
- eject
- flip over
- insert
- find 4.7Gb data - tweaking to fit
- write
- eject
- catalog 9.4Gb data
- label disc
- store in jewel case
- don't scratch it along the way
- REPEAT FOR EVERY 4.7GB

HDD:

- open
- stick in $40 USB dock
- find 1TB of data
- copy overnight
- pull drive
- catalog
- label the drive

For smaller sets of data, removable discs can work for most. Although, all of us know that our archive set is growing exponentially every year with more cameras and more resolution.

To make the archive set work, simply rotate through a few drives every month.
One onsite for a week, getting new files, pull, swap with another drive at home. When full, store one copy at the bank, another at home.

Keeping track of 1 drive per 100+ discs...hmmm

And my point for the lists above was time. I know my time is more valuable that burning 4.7Gb at a time. The cost benefit of the media has already been illustrated, but lets say it takes 10 minutes to burn a DVD-RAM at 8x. You can use that time to find the new data set. But I still have to sit and swap discs 212 times compared to starting one copy. That comes out to 35 hours of manually swapping discs and manning the copies.... hmmm

Again, if you don't shoot much, that may be the way to go, but I accumulate 50-300Gb per weekend 6 months a year (2-3TB per season = ~500 discs to keep burn and keep track of - but I'm different.

)

Hammy.

Aug 19, 2008 at 11:45 PM

nathanlake
Offline
Image Upload: On

p.1 #25 · usb flash as long term storage/archive

What makes you think that flash drives ensure long term survivability? While the devices may last a long time, most reports suggest that you can only expect about 10 years for the data stored on them.

http://www.chuckstr89134.com/newsltrs/015.htm
http://arstechnica.com/reviews/hardware/flash.ars
http://www.cdw.com/shop/products/default.aspx?edc=959056

Aug 20, 2008 at 02:54 AM


FM Forums \| PRO digital corner \| Join Image Upload 1 2 end