RAID systems

Home · Register · Search · Software · Hosting · Software · Join Upload & Sell

Moderated by: Steve Tinetti, Fred Miranda

FM Forum Rules
Wedding Resource List

FM Forums \| Wedding Photographer \| Join Upload & Sell
1 2 end

RAID systems

awad
Offline
• •
Upload & Sell: Off

p.1 #1 · RAID systems

I need a better solution for working on files. I currently have a mac pro with 4 3TB HDs. one for 2013 season, one as a backup to the 2013 drive, one for the 2012 season, and my system drive.

The 2013 and 2012 drives are both backed up to two different externals every night with SuperDuper and the whole lot is backed up to crashplan.

I'd like to add in a working RAID 5 box into the mix to build in some redundancy and save me from keeping track of 6 separate drives.

I was originally looking at a Drobo, but the proprietary system made me rethink it.

I'm currently looking at the Synology DS413

http://www.synology.com/products/product.php?product_name=DS413&lang=us

I've read Spencer's link about data management.
http://www.spencerboerupblog.com/2009/09/data-management-backup-solutions-for-photographers.html

but I'm hoping to try something different now with a little more redundancy built in. Or is my current solution the best way?

Chime in if you've got any ideas or solutions.

Jan 16, 2013 at 11:06 PM

SloPhoto
Offline
• • •
Upload & Sell: On

p.1 #2 · RAID systems

----to repeat the neccesary---
Raid in itself is not a backup plan and would not change the rest of your process. Using it to replace your working and backup drives would not be valid.

----on to real comments----

If the goal is to improve how robust your system is, plan for failure and examine how well things recover from hardware failure.

-What is the process to restore the array when a HD fails
-What is the process to restore if the controller fails (some raid setups require a near identical controller to be tracked down)

Another thing you should be concerned about is error propagation. Lets just say your hd controller corrupts one set of photos (this has happened to me), how long does it take for that error to propagate to your backups and will you loose data.

I use a software raid based ZFS system because of how it addresses the above issues, but it takes quite a bit more time/knowledge to setup.

Jan 17, 2013 at 12:30 AM

whtrbt7
Offline
• •
Upload & Sell: On

p.1 #3 · RAID systems

The Synology product you listed is an NAS. It's not really suitable for what I think you're using it for. Your entire "system" of data management may need some evaluation. RAID can be used either for a singular backup or it can be used as a working volume. Backup should not only consist of a singular back up plan. As SloPhoto said, you need to examine how you can recover from hardware and software failures. You might want to ask yourself what you really need out of a system and then go from there. That said, the Synology could work as a single working volume so you don't have to work from 3 separate volumes from your Mac Pro. If you get another unit, you can use it as local backup via time machine or SuperDuper. If you get a 3rd one, you can have redundant local backup. I'm not sure about data restoration from the Synology units but if you're going for RAID 0, you can have high speed so you can keep everything on one big volume that can get backed up by another large RAID 0 for great speed plus single backup. If you're afraid of losing a session's worth of data go with something like RAID5 or RAID6 so you get some redundancy with striping so if one drive fails, you chuck it and rebuild the array by slapping in a new drive. Overall, for an office situation, I prefer the speed of RAID0 and then adding in multiple backups. It's really a stylistic thing.

Jan 17, 2013 at 03:10 AM

Rogue416
Offline
•
Upload & Sell: On

p.1 #4 · RAID systems

whtrbt7 wrote:
The Synology product you listed is an NAS. It's not really suitable for what I think you're using it for. Your entire "system" of data management may need some evaluation. RAID can be used either for a singular backup or it can be used as a working volume. Backup should not only consist of a singular back up plan. As SloPhoto said, you need to examine how you can recover from hardware and software failures. You might want to ask yourself what you really need out of a system and then go from there. That said, the Synology could work as a single working volume so you don't have to work from 3 separate volumes from your Mac Pro. If you get another unit, you can use it as local backup via time machine or SuperDuper. If you get a 3rd one, you can have redundant local backup. I'm not sure about data restoration from the Synology units but if you're going for RAID 0, you can have high speed so you can keep everything on one big volume that can get backed up by another large RAID 0 for great speed plus single backup. If you're afraid of losing a session's worth of data go with something like RAID5 or RAID6 so you get some redundancy with striping so if one drive fails, you chuck it and rebuild the array by slapping in a new drive. Overall, for an office situation, I prefer the speed of RAID0 and then adding in multiple backups. It's really a stylistic thing.

+1 I totally agree with you on this whtrbt7.

Jan 17, 2013 at 03:33 AM

Rogue416
Offline
•
Upload & Sell: On

p.1 #5 · RAID systems

awad wrote:
I need a better solution for working on files. I currently have a mac pro with 4 3TB HDs. one for 2013 season, one as a backup to the 2013 drive, one for the 2012 season, and my system drive.

The 2013 and 2012 drives are both backed up to two different externals every night with SuperDuper and the whole lot is backed up to crashplan.

I'd like to add in a working RAID 5 box into the mix to build in some redundancy and save me from keeping track of 6 separate drives.

I was originally looking at a Drobo, but the proprietary system made me rethink it.

I'm currently looking at the Synology DS413

http://www.synology.com/products/product.php?product_name=DS413&lang=us

I've read Spencer's link about data management.
http://www.spencerboerupblog.com/2009/09/data-management-backup-solutions-for-photographers.html

but I'm hoping to try something different now with a little more redundancy built in. Or is my current solution the best way?

Chime in if you've got any ideas or solutions.

Seeing that you are using 4 3TB drives, I think RAID 6 would be best for you. RAID 6 uses two independent parity schemes that are distributed across the drives. It requires at least four drives versus a minimum of three drives for RAID 5. Since two parity schemes must be computed, the controller overhead is greater than that of RAID 5. RAID 5 can suffer one drive failure and RAID 6 can suffer multiple drive failures depending on the number of drives in the array. RAID 6 is implemented through hardware (drive controller), whereas RAID 5 can be either software or hardware.

A couple of links to help you decide:

http://www.raidrecoveryonline.com/raid_5_recovery/

http://www.raidrecoveryonline.com/raid_6_recovery/

Have you considered investing in a Blu-ray burner as additional option? With the new BD-XL discs you can burn 100GB or 128GB worth of data compared to 25GB or 50GB with the BD-R/RW discs.

HTH.

Trent

Jan 17, 2013 at 03:50 AM

SloPhoto
Offline
• • •
Upload & Sell: On

p.1 #6 · RAID systems

Rogue416 wrote:
... RAID 5 can suffer one drive failure and RAID 6 can suffer multiple drive failures depending on the number of drives in the array. RAID 6 is implemented through hardware (drive controller), whereas RAID 5 can be either software or hardware.

Raid6 can only withstand 2 drive failures, and can be done through software.

Raid5 with 4 3tb drives = 9tb usable space, 3tb dedicated to parity.
Raid6 with 4 3tb drives = 6tb usable space, 6tb dedicated to parity.

Raid5 with 5 3tb drives = 12tb usable space, 3tb dedicated to parity.
Raid6 with 5 3tb drives = 9tb usable space, 6tb dedicated to parity.

I use ZFS2 to be specific, which is ZFS with two sets of parity bits (like raid6). I am paranoid though. No matter how much redundancy you build into your working drive, it is still not a backup.

Jan 17, 2013 at 06:59 AM

ricardovaste
Online
• • • •
Upload & Sell: Off

p.1 #7 · RAID systems

I went through all this last year whilst updating my hard drive setup. I found that controller fail is extremely rare, hard drive fail does happen & auto-backups for everything can often result in lost data.

I run two 4 bay HDD enclosures on RAID5, manually backing the one to the other. I know, the controllers COULD go, but both at the same time, not likely. Then, this is what my offsite backup is for.

As long as you have something you're happy with in yourself, and you know covers everything well, I think you're good to go. You can't protect everything indefinitely, just try and fight the odds in a realistic fashion.

Jan 17, 2013 at 11:56 AM

Rogue416
Offline
•
Upload & Sell: On

p.1 #8 · RAID systems

SloPhoto wrote:
Rogue416 wrote:
... RAID 5 can suffer one drive failure and RAID 6 can suffer multiple drive failures depending on the number of drives in the array. RAID 6 is implemented through hardware (drive controller), whereas RAID 5 can be either software or hardware.

Raid6 can only withstand 2 drive failures, and can be done through software.

Raid5 with 4 3tb drives = 9tb usable space, 3tb dedicated to parity.
Raid6 with 4 3tb drives = 6tb usable space, 6tb dedicated to parity.

Raid5 with 5 3tb drives = 12tb usable space, 3tb dedicated to parity.
Raid6 with 5 3tb drives = 9tb usable space, 6tb dedicated to parity.

I use ZFS2 to be specific, which is ZFS with two sets of parity bits (like raid6). I am paranoid though. No matter how much redundancy you build into your working drive, it is still not a backup.

Correct. It has been a few years since I have done anything with RAID, so my knowledge is a little rusty. Thank you.

Jan 17, 2013 at 02:43 PM

Brit-007
Offline
• • • •
Upload & Sell: Off

p.1 #9 · RAID systems

You were originally looking at DROBO and I have been using one for numerous years. Have not had any issues with it. I added a second unit recently and that is running sweet. I have numerous friends in the Pro Photography field and they are all using the DROBO without issue. The beauty of the uni9t is that if you are running low on space, you can upgrade a drive without losing any time by continuing to work. Yes it is propriety but with a RAID system you have to let it rebuild before accessing data and if you lose a couple of drives you can lose everything. I know Scott Kelby had issues but without knowing exactly how he handled the units while on location one can never know. I can only go by experience.

Jan 18, 2013 at 12:02 AM

Beni
Offline
• • • • •
Upload & Sell: On

p.1 #10 · RAID systems

In the studio we are shooting to an internal 2TB in a software RAID 0. After working on it we are moving it all to a NAS 21TB drive (8 disk) in Raid 6 (gives us 16TB working drive). This gives us a 2 disk redundancy and is unbelievably fast the read write speeds have to be seen to be believed! It is then backed up on LTO 5 tapes to two external locations.

And I still worry, I want an additional external location in another country.

Not weddings though, photographing ancient manuscripts for a museum with a 40 megapixel MFDB at a throughput of about one of those 21TB drives per year.

Jan 18, 2013 at 06:17 AM

AlephOne
Offline
•
Upload & Sell: Off

p.1 #11 · RAID systems

RAID is good for reliability, but isn't a substitute for backup.

Rule of thumb - you need three copies of every file.

I have a Synology DS412 (predecessor to the DS413) operating in RAID 5 as one of my copies. I have a Thecus N5200 Pro also in RAID 5 as one of the others. The NAS boxes sit on separate UPS. I backup to external hard drives (USB 3 makes backup rather faster).

I download cards to an SSD, and back them up to both NAS boxes before erasing the cards. I process the images on SSD - if you think hard drives in RAID 0 is fast, I suggest you try SSD instead - bring up thumbnails from SSD is noticeably faster. Using RAID 0 also means you are exposed - if either drive fails, you lose everything on the RAID 0 volume.

So far, I have had two hard drive failures in NAS boxes (over the course of five or six years). In both cases, I've replaced the failed hard drive and rebuilt the volume. Even if the rebuild hadn't worked, I wouldn't have lost anything, because I had backups and copies on the other NAS boxes.

Jan 18, 2013 at 07:35 AM

sboerup
Offline
• • • • •
Upload & Sell: On

p.1 #12 · RAID systems

AlephOne wrote:
RAID is good for reliability, but isn't a substitute for backup.

Rule of thumb - you need three copies of every file.

Gospel here folks

Jan 20, 2013 at 08:11 PM

RDKirk
Offline
• • • • •
Upload & Sell: Off

p.1 #13 · RAID systems

sboerup wrote:
AlephOne wrote:
RAID is good for reliability, but isn't a substitute for backup.

Rule of thumb - you need three copies of every file.

Gospel here folks

There are so many ways to skin a cat. I find RAID cumbersome and harder to manage and requires more effort in the long run. Just make 3 copies of the file and you should be safe, especially if the copies reside in different physical locations.

I agree. A RAID mirror is for high-availability: Functions that can't afford a single minute of downtime, like an online bank. Otherwise, if you have time to mount a backup (which would likely include anyone on this forum), you're better off using that additional drive for an additional copy.

Further, a RAID mirror is only protection against drive failure, not against any of the many and much more likely forms of data corruption. Failure of a data backup drive is a mere inconvenience of minutes when you have multiple copies, but the immediate propagation of data corruption by a RAID mirror is a real problem.

This has nothing to do with using some flavor of RAID striping to speed IO on a working drive, this is about using a RAID mirror as a backup.

Jan 20, 2013 at 10:28 PM

Mr645
Offline
• • •
Upload & Sell: Off

p.1 #14 · RAID systems

Here was I use, and I have been shooting digitally only since Jan 2000. No lost files.

I use a Power Mac G5 and a HIghpoint PCI-X RAID card. I have 4 2TB drives but the card can handle 8 drives.

I have 4 drives plus the boot drive in my G5 and they are set up at a 6TB RAID 5 array. I have a second identical system as a back up.

1) cards come into the studio and RAW files are copied to the main server, edited, color corrected renamed and processed for proofing. At that point the job is copied to the back up server and also a set of DVD's are burned to file in the clients folder. The CF cards are then returned into service.

2) Back ups are updated as needed, not daily because often something is damaged due to human error, but it is nice to go back a few days or so.

3) about 6 months after the album is delivered and the job is done, I will burn another set of DVD's and remove the event from both servers.

There are times that I will be a week or two behind editing so I will store a back up set of files unedited just until I get to it.

In 13 years plenty of drives have failed, but loosing two drives at the same time to kill and array has never happened. If something really bad does happen, I can go back to the DVD's and start over.

The RAID was set up made years ago, first with 8 400GB drives and later updated with 4 2TB drives, and I may move to 3TB drives this year. I can fit 5 drives inside my Power Macs so I can add more drives to thecurrent array but that means external cables which I would like to avoid.

Jan 21, 2013 at 06:18 PM

Occulai
Offline
•
Upload & Sell: Off

p.1 #15 · RAID systems

I too am currently looking to make a move to something with a little more space and redundancy. As a bit of a computer nerd, I don't mind the extra hassle of building my own, and appreciate the added flexibility. As such, I will put in another vote for building a ZFS based system. The ability to do snapshots to be able to go back in time with your files, and the built in block-level checksum to combat corruption make it the choice for me. If you set up a separate box for it, you can add a nice web-interface called napp-it that I find very helpful. Depending on how you go about it, you can also pull this off cheaper than some of the alternatives.

Then for an offsite backup, something like crashplan+ (crashplan.com) looks like it might be pretty handy, though I have no experience with it. Regardless, if you're in the market for some sort of NAS solution, there have been some good deals on the Western Digital Red drives recently. Best of luck!

Jan 22, 2013 at 05:55 AM

mineymole
Offline
• • • •
Upload & Sell: On

p.1 #16 · RAID systems

Okay I have a question...

Buying new thunderbolt drives....

Should I get 2 Lacie 4tb thunderbolt drives (non raid), or should I get one 8tb Lacie Thunderbolt Raid system and let the Raid duplicate the files or should I get two 4tb Lacie Thunderbolt Raid Systems each running raid (one system is the backup of the other system).... or lastly one 4tb Lacie Thunderbolt Raid System (running raid 0 for speed) backed up to a 4tb Lacie non-raid thunderbolt drive.

Jan 22, 2013 at 03:07 PM

Mr645
Offline
• • •
Upload & Sell: Off

p.1 #17 · RAID systems

I would do two seperate RAID systems. If the controller goes out on a single array system your done. I am not a fan for RAID 0 for storage, but I do use a lot of striped arrays for boot drives on the workstations. I like RAID 5 for storage and a back up to another RAID 5 set up.

Jan 22, 2013 at 05:09 PM

sozypozy
Offline
• • •
Upload & Sell: Off

p.1 #18 · RAID systems

I agree, RAID is great but not exactly for backup.

Edited on Jan 24, 2013 at 08:31 AM · View previous versions

Jan 22, 2013 at 08:27 PM

HappyCamp
Offline
• • •
Upload & Sell: Off

p.1 #19 · RAID systems

Mr645 wrote:
Here was I use, and I have been shooting digitally only since Jan 2000. No lost files.

<snip>

The RAID was set up made years ago, first with 8 400GB drives and later updated with 4 2TB drives, and I may move to 3TB drives this year. I can fit 5 drives inside my Power Macs so I can add more drives to thecurrent array but that means external cables which I would like to avoid.

Do you also do off-site storage? Just wondering if an asteroid comes down and flattens your home/office while you are on vacation in Hawaii, will you still have your data? Or whatever disaster you can think of that destroys your home/office and all data residing there.

Sorry if I blindly missed where you mentioned that

Jan 22, 2013 at 08:31 PM

Jeff Simpson
Offline
• •
Upload & Sell: Off

p.1 #20 · RAID systems

nobody talks about RAID 1, this is definitely the best way to have your files backed up. RAID 0 is stupid when you need to protect important files, and the increase is read/write speed is negligible.

As said before you always need a 'proper' backup of files.. off-site or online backups are most common.

Jan 24, 2013 at 04:17 PM

1 2 end
FM Forums \| Wedding Photographer \| Join Upload & Sell 1 2 end