Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | Post-processing & Printing | Join Upload & Sell

1
       2       3       4       5       end
  

Archive 2022 · Raid recommendtions

  
 
Luvwine
Offline
• • • •
Upload & Sell: On
p.1 #1 · p.1 #1 · Raid recommendtions


I am currently using external hard drives. I have one 16 TB drive and an assortment of smaller hard drives and use Backblaze for my offsite backups. My thought is that my cheap 16TB external is not very fast but will likely serve as a backup for a while as I have about 8TB of photos I want to backup. Therefore, I was thinking of getting a raid 5 or raid 6 setup to use with my new Mac Studio (getting in May). I have shied away from raid for a long time due to an incident I had many years ago where I had some data loss. I think I am ready to try it again and am divided between getting a four drive enclosure for a raid 5 setup or trying for even more redundancy and getting raid 6 (assume a 6 or 8 drive enclosure is called for there. I don't think enterprise drives are worth the money, but am looking for recommendations as to what makes sense form others who have likely plowed this ground before me. Any specific good deals now are appreciated. I think SSD drives are the future but suspect the price will make me not choose that medium for the time being. Thanks for any thoughts/recommendations.


Mar 27, 2022 at 08:16 PM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.1 #2 · p.1 #2 · Raid recommendtions


I have multiple NAS units, 3x QNAP, 3x Synology and home-built with FreeNAS.
QNAP offers QTS and QTS Hero, the latter uses ZFS as their more resilient file system. Synology supports btrfs as their better file system. Either ZFS or btrfs should allay data integrity concerns, especially if you also use a NAS that supports ECC RAM. Both QNAP and Synology also support the traditional EXT4 file system. If you don't know about Linux file systems, read this for example.

I typically use the Seagate Exos helium drives. Enterprise drives are often less expensive or at least similar to the NAS branded drives. I look for drives with the highest MTBF (2.5M hours), highest annual TBW (usually 550TB), and low UBER (10^-15). You don't want any drive that uses SMR rather than CMR.
I tend to use RAID 5 up to 5 drives and RAID 6 for 8+ drives, but I have fully redundant NAS units so RAID 5 doesn't bother me too much.

The CPU and ethernet connection can be performance bottlenecks. Avoid the wimpy ARM processors and look for an Intel or AMD x86-64 CPU if you want to reach anywhere near saturating 10GbE with a parity RAID mode. If you only use 2.5GbE, then a weaker CPU might be OK, though RAID 6 (RAID-Z2) requires nearly double the processing of RAID 5 (RAID-Z1). 1GbE is slow, being more suited to backups than a primary drive for large amounts of photos or videos.

Take a look at the Synology DS1621+/DS1821+ or QNAP TS-473A/673A/873A as relatively inexpensive NAS with good performance, slots for 10GbE cards (IIRC MAC uses 10GBASE-T), and support for ECC SO-DIMMS. You can get better NAS with Xeon CPUs, but that might be beyond your needs. Unfortunately Synology is practically enforcing use of their ridiculously overpriced Toshiba drives with firmware in their higher grade NAS (newer xs and 22 series).

EBH




Mar 27, 2022 at 10:03 PM
Luvwine
Offline
• • • •
Upload & Sell: On
p.1 #3 · p.1 #3 · Raid recommendtions


EB-1 wrote:
I have multiple NAS units, 3x QNAP, 3x Synology and home-built with FreeNAS.
QNAP offers QTS and QTS Hero, the latter uses ZFS as their more resilient file system. Synology supports btrfs as their better file system. Either ZFS or btrfs should allay data integrity concerns, especially if you also use a NAS that supports ECC RAM. Both QNAP and Synology also support the traditional EXT4 file system. If you don't know about Linux file systems, read this for example.

I typically use the Seagate Exos helium drives. Enterprise drives are often less expensive or at least similar to the NAS branded
...Show more

Wow. Perhaps I should rephrase my query. I understood only part of your kind reply. I appreciate you taking the time but I am beginning to think the most important thing will be to get a raid setup that is made for dummies. The Synology enclosure (Server?) looks great but wonder how user friendly for a novice on a Mac. Any thoughts about OWC? Their 4 bay raid 5 Thunderbay seems more novice friendly tho I could be totally wrong. Thx again!



Mar 27, 2022 at 10:45 PM
Rajan Parrikar
Offline
• • • • •
Upload & Sell: On
p.1 #4 · p.1 #4 · Raid recommendtions


SSD will be very expensive for RAID 5/6 especially if your Terabyte requirements are high.

Later this year I am upgrading to the 16TB SSD Thunderblade array from OWC (link below) to be used in RAID 0 configuration that will offer the full 16TB capacity. Not cheap, but fast and highly portable, and it is bye-bye forever to hard disks. I keep a lot of redundant backups via cheaper portable SanDisk SSDs.

https://eshop.macsales.com/shop/owc-thunderblade



Mar 28, 2022 at 05:46 AM
Camperjim
Offline
• • • • •
Upload & Sell: On
p.1 #5 · p.1 #5 · Raid recommendtions


It seems you lost some data the last time you tried a RAID. That should have told you something. No matter how many disks are included, a RAID should be considered as a single backup. The risks might be low but it can be affected by hardware or operator error.

You really need to consider storage on at least 2 completely isolated systems. One should be unconnected and in a remote protected area. Individual hard drives serve that purpose and are relatively low cost. I went with 3 backups. I have one next to the computer for frequent backups and ease of use. Another set of drives is in my basement in a fire safe and a third I keep in my detached garage. In addition to having multiple backups, I add a new drive every few years and retire one of the older drives. Currently 8 Tb drives serve my purpose, but if I needed more I would go to 16 Tb or pairs of 8 Tb drives.

Your situation might be different but I find that I rarely need to look at my very old files. If I needed more speed and convenience I would consider adding an SSD just for my most recent work. For me a 1 Tb drive would be more than sufficient for storing the last few years of work.

Another consideration is the size of your files. I do not agree with the idea that storage is cheap. In addition to the hardware, it takes time and planning to maintain storage archives. With time I have become aggressive in culling after each shooting session. I am also careful about how much shooting I do.




Mar 28, 2022 at 06:26 AM
jhapeman
Offline
• • • •
Upload & Sell: On
p.1 #6 · p.1 #6 · Raid recommendtions


Rajan Parrikar wrote:
SSD will be very expensive for RAID 5/6 especially if your Terabyte requirements are high.

Later this year I am upgrading to the 16TB SSD Thunderblade array from OWC (link below) to be used in RAID 0 configuration that will offer the full 16TB capacity. Not cheap, but fast and highly portable, and it is bye-bye forever to hard disks. I keep a lot of redundant backups via cheaper portable SanDisk SSDs.

https://eshop.macsales.com/shop/owc-thunderblade


This is what I use. Not cheap...but crazy fast, and no moving parts to fail. I have two, one for backup and on for backup offsite. Considering I have 22 years of images on there, it's worth the money IMO. The failure rate on NVMe is very low if you're not constantly writing/erasing--for long-term archive its a great solution.



Mar 28, 2022 at 08:29 AM
Camperjim
Offline
• • • • •
Upload & Sell: On
p.1 #7 · p.1 #7 · Raid recommendtions


jhapeman wrote:
The failure rate on NVMe is very low if you're not constantly writing/erasing--for long-term archive its a great solution.


I have no information on the failure rates for the specific device you bought, but overall failures for SSDs is still a concern. The data I have seen indicates the failure rate is only about 25% better than for a HD. Life expectancy for the current technology is probably well less than 10 years and decreases dramatically with use; i.e., read/write cycles.

If speed is highly important then SSDs are much, much better. Life expectancy not so much especially considering the very high cost.



Mar 28, 2022 at 08:48 AM
jhapeman
Offline
• • • •
Upload & Sell: On
p.1 #8 · p.1 #8 · Raid recommendtions


Camperjim wrote:
I have no information on the failure rates for the specific device you bought, but overall failures for SSDs is still a concern. The data I have seen indicates the failure rate is only about 25% better than for a HD. Life expectancy for the current technology is probably well less than 10 years and decreases dramatically with use; i.e., read/write cycles.

If speed is highly important then SSDs are much, much better. Life expectancy not so much especially considering the very high cost.


Even 25% better is very significant and I don't buy them for the MTBF, I buy them for the speed. That said, failure in SSD's is almost 100% tied to the number of read/writes, as they have a limit essentially baked into them. Spinning drives on the other hand have a host of potential mechanical failure points. I also transport my drives around, something that shortens the life of spinning disks, but has zero impact on SSDs. Cost and life expectancy are two entirely different things, unless the intent from day one is to reduce costs only.



Mar 28, 2022 at 08:56 AM
Rajan Parrikar
Offline
• • • • •
Upload & Sell: On
p.1 #9 · p.1 #9 · Raid recommendtions


jhapeman wrote:
This is what I use. Not cheap...but crazy fast, and no moving parts to fail. I have two, one for backup and on for backup offsite. Considering I have 22 years of images on there, it's worth the money IMO. The failure rate on NVMe is very low if you're not constantly writing/erasing--for long-term archive its a great solution.


I have every expectation at the Thunderblade SSD unit will last me 6 years at least (if it does, I consider it money well spent). I think OWC also offers other options with enclosures and user replaceable SSD drives.

My Pegasus Thunderbolt hard disk array is now 4+ years old. Terrific unit, but I am now in that regime where hard disk failure is a high probability event.



Mar 28, 2022 at 09:15 AM
jhapeman
Offline
• • • •
Upload & Sell: On
p.1 #10 · p.1 #10 · Raid recommendtions


Rajan Parrikar wrote:
I have every expectation at the Thunderblade SSD unit will last me 6 years at least (if it does, I consider it money well spent). I think OWC also offers other options with enclosures and user replaceable SSD drives.

My Pegasus Thunderbolt hard disk array is now 4+ years old. Terrific unit, but I am now in that regime where hard disk failure is a high probability event.


You can also swap the drives out in the Thunderblade. I got an open-box 1TB unit from them for really cheap and put 4 Sabrent 4TB drives in it. The newer Sabrent drives didn't work quite right, so I used a set of the older PCIe 3.0 drives and they work like a champ. A year ago that was much cheaper to do, but OWC has dropped their prices and the 16GB is really a great deal IMO.




Mar 28, 2022 at 10:00 AM
bobby350z
Offline
• • • •
Upload & Sell: On
p.1 #11 · p.1 #11 · Raid recommendtions


OP, how is this raid box going to get used? Is it just for backups or as a working drive? Will you use the main drive on your MAC Studio as the working drive? And do you need to access files from the raid box from another machines?

My current setup is simple. I used to have drives inside my PC but now I have a USB3.2 enclosure which can take 2 drives in raid 0. I have 2 10TB drives in it. And I work from that enclosure. CaptureOne runs fine. I will test it when I get my MAC Studio and then I would need to upgrade this setup to something with thunderbolt 3 or 4. I am the only user working on these files so I don't have a need from another machine. Typical NAS boxes using ethernet and even fiber would be slower but then they do allow connection over a network.



Mar 28, 2022 at 10:46 AM
Luvwine
Offline
• • • •
Upload & Sell: On
p.1 #12 · p.1 #12 · Raid recommendtions


Camperjim wrote:
It seems you lost some data the last time you tried a RAID. That should have told you something. No matter how many disks are included, a RAID should be considered as a single backup. The risks might be low but it can be affected by hardware or operator error.

You really need to consider storage on at least 2 completely isolated systems. One should be unconnected and in a remote protected area. Individual hard drives serve that purpose and are relatively low cost. I went with 3 backups. I have one next to the computer for frequent backups and ease
...Show more

Appreciate the comments, but yes, I have already two copies of everything locally on hard disks and remote backup via Backblaze.

My intention is to get a fair amount of capacity in a Raid array--perhaps Raid 5 as cheaper and more easily available--and treat it as a single drive unit.

My new Mac Studio I have configured with a 4 TB internal SSD drive, which should allow me to keep recent files there and copy them to external drives for backups. I usually shoot with two SSD cards in the camera with each image duplicated when shot. I then make sure I have copied these images to multiple backups before formatting the cards.




Mar 28, 2022 at 11:23 AM
Luvwine
Offline
• • • •
Upload & Sell: On
p.1 #13 · p.1 #13 · Raid recommendtions


bobby350z wrote:
OP, how is this raid box going to get used? Is it just for backups or as a working drive? Will you use the main drive on your MAC Studio as the working drive? And do you need to access files from the raid box from another machines?

My current setup is simple. I used to have drives inside my PC but now I have a USB3.2 enclosure which can take 2 drives in raid 0. I have 2 10TB drives in it. And I work from that enclosure. CaptureOne runs fine. I will test it when I get my MAC
...Show more

Hi, no need for network as I am a single user of the machine and drives.

My thought right now (in advance of actually getting the Mac Studio) is to use the 4 TB internal SSD drive as the initial location for newly shot images (unless there is some advantage not do so). Then, say monthly, as I decide to delete/format the SSD cards in the cameras, to move the files to the Raid array and external backup. I would then use the Raid Array as a working disk for all but the most recent files and the external slow HD's for local backup and Backblaze as the offsite backup.



Mar 28, 2022 at 11:27 AM
bobby350z
Offline
• • • •
Upload & Sell: On
p.1 #14 · p.1 #14 · Raid recommendtions


Luvwine wrote:
Hi, no need for network as I am a single user of the machine and drives.

My thought right now (in advance of actually getting the Mac Studio) is to use the 4 TB internal SSD drive as the initial location for newly shot images (unless there is some advantage not do so). Then, say monthly, as I decide to delete/format the SSD cards in the cameras, to move the files to the Raid array and external backup. I would then use the Raid Array as a working disk for all but the most recent files and the external slow HD's
...Show more

Same plan as me. I only went for the 1TB internal SSD as biggest of my sessions will be < 256GB. Let me know what external raid box you decide on. I would say look for direct attached one (over thunderbolt) and not the ethernet, if you plan to work from it directly in the future. I would also like a hardware raid and not a sw raid. I also use separate external USBs for another copy and upload to backblaze.



Mar 28, 2022 at 11:48 AM
jhapeman
Offline
• • • •
Upload & Sell: On
p.1 #15 · p.1 #15 · Raid recommendtions


bobby350z wrote:
Same plan as me. I only went for the 1TB internal SSD as biggest of my sessions will be < 256GB. Let me know what external raid box you decide on. I would say look for direct attached one (over thunderbolt) and not the ethernet, if you plan to work from it directly in the future. I would also like a hardware raid and not a sw raid. I also use separate external USBs for another copy and upload to backblaze.


If you want hardware RAID the best choice is the Promise. The OWC boxes are software RAID, and in my experience their SoftRAID software is kind of touchy. I use MacOS software RAID on my OWC Thunderblades and its worked perfectly. The overhead for the RAID calculations is very low with the power of today's processors. The Promise Pegasus32 R4 16TB RAID System is pretty cost-effective and quite fast.




Mar 28, 2022 at 12:06 PM
Rajan Parrikar
Offline
• • • • •
Upload & Sell: On
p.1 #16 · p.1 #16 · Raid recommendtions


jhapeman wrote:
If you want hardware RAID the best choice is the Promise. The OWC boxes are software RAID, and in my experience their SoftRAID software is kind of touchy. I use MacOS software RAID on my OWC Thunderblades and its worked perfectly. The overhead for the RAID calculations is very low with the power of today's processors. The Promise Pegasus32 R4 16TB RAID System is pretty cost-effective and quite fast.



Another vote for Promise Pegasus RAID arrays if you want to go the HDD route.

Re. SoftRAID - I, too, want to avoid it and use macOS to configure RAID 0 since all I want is one volume.




Mar 28, 2022 at 12:20 PM
bobby350z
Offline
• • • •
Upload & Sell: On
p.1 #17 · p.1 #17 · Raid recommendtions


Regarding this SoftRAID in MAC, you recommend but an enclosure with 2 drives and then setup RAID on my MAC side? I will also read up on this.


Mar 28, 2022 at 01:24 PM
jhapeman
Offline
• • • •
Upload & Sell: On
p.1 #18 · p.1 #18 · Raid recommendtions


bobby350z wrote:
Regarding this SoftRAID in MAC, you recommend but an enclosure with 2 drives and then setup RAID on my MAC side? I will also read up on this.


I recommend NOT using SoftRAID--that's a product sold by OWC. Just use the software RAID functionality built into MacOS--you can create a RAID set using Disk Utility.



Mar 28, 2022 at 01:38 PM
Luvwine
Offline
• • • •
Upload & Sell: On
p.1 #19 · p.1 #19 · Raid recommendtions


Rajan Parrikar wrote:
Another vote for Promise Pegasus RAID arrays if you want to go the HDD route.

Re. SoftRAID - I, too, want to avoid it and use macOS to configure RAID 0 since all I want is one volume.



Thanks. In looking at reviews, looks like a good system albeit several comments about it being noisy.....

SSD would doubtless be quieter albeit at a considerable price premium.



Mar 28, 2022 at 01:40 PM
Luvwine
Offline
• • • •
Upload & Sell: On
p.1 #20 · p.1 #20 · Raid recommendtions


jhapeman wrote:
If you want hardware RAID the best choice is the Promise. The OWC boxes are software RAID, and in my experience their SoftRAID software is kind of touchy. I use MacOS software RAID on my OWC Thunderblades and its worked perfectly. The overhead for the RAID calculations is very low with the power of today's processors. The Promise Pegasus32 R4 16TB RAID System is pretty cost-effective and quite fast.



Thanks. 16TB may be a bit small to me as I already have about 8TB of data and with Raid 5, one loses some capacity so would be more like 12TB I think. Guess I will need to get more like 24 or 32 TB. Wish it was less noisy (per the reviews) but I guess one cannot have everything. Thx much!



Mar 28, 2022 at 01:42 PM
1
       2       3       4       5       end




FM Forums | Post-processing & Printing | Join Upload & Sell

1
       2       3       4       5       end
    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.