Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | Post-processing & Printing | Join Upload & Sell

1       2       3      
4
       5       end
  

Archive 2022 · Raid recommendtions

  
 
Ryukyu
Offline
• •
Upload & Sell: On
p.4 #1 · p.4 #1 · Raid recommendtions


Seriously, I’d really like to know what you all are doing that requires that much speed.


May 06, 2022 at 12:50 PM
Luvwine
Offline
• • • •
Upload & Sell: On
p.4 #2 · p.4 #2 · Raid recommendtions


"Reason not the need" quoth the Bard. Seriously, "requires" is not really the test. Sure, I could do 99.9% of what I want to do with a slower computer, but recently I had a lot of frustration with my iMac with 42 Gig of ram just choking on files where I was doing a large stitch. It took over an hour to combine and when I came back, I would find the computer had restarted and I would have to start again--really annoying.

The more common usage for me is to do 3-image HDR images and then say do 3-4 image focus stacks of the scene. Sometimes, I would wait 1.5 minutes or longer for one three-image stack to be combined. Multiplying such things by dozens of images meant long post processing sessions that can be greatly shortened via a speedier machine. Your mileage may vary and likely I could have gotten by with a cheaper machine, but hopefully this will last me a long time. Time will tell. Bottom line is that we all have different amounts we are willing to pay for incremental increases in performance (law of diminishing return).



May 06, 2022 at 02:05 PM
Ryukyu
Offline
• •
Upload & Sell: On
p.4 #3 · p.4 #3 · Raid recommendtions


OK I get that, but that's computer speed not i/o external storage speed.


May 06, 2022 at 02:17 PM
Luvwine
Offline
• • • •
Upload & Sell: On
p.4 #4 · p.4 #4 · Raid recommendtions


I see, you are addressing why use SSD instead of HDD? I did not get a RAID array to get increased speed over a single SSD. If I could have found an SSD that was larger than 8 TB and of a quality that could be used reliably for a long time, I would have bought it. Unfortunately, SSD drives are almost all pretty small so to get large capacity one has to get a Raid setup. SSD have advantages, of course, including speed, noise, and reliability. The speed is helpful when doing exports, imports, and retrieving multiple files at once. The time saved there is much less significant, but it does help. I figured since I was spending $$ on a speedy computer and needed another large external drive anyway, I should just go ahead and and get a fast external drive and remove bottlenecks where I can. I just copied nearly 8 TB of files from an HDD to my new RAID array. Only took 14 hours due to the slow speeds of the HDD. Hopefully, won't need to wait that long next time.... Edit: I did consider getting an HDD RAID setup to increase transfer speeds (over a single slow HDD) and capacity, but I figured that would not be cheap, be significantly noisier, take up more desk real estate, and be more likely to fail than an SSD RAID.



May 06, 2022 at 02:30 PM
Ryukyu
Offline
• •
Upload & Sell: On
p.4 #5 · p.4 #5 · Raid recommendtions


Thank you for your explanation. It’s not even SSD vs. HDD so much as is there a need for that kind of performance, or just a desire to have it? I guess I’m thinking more practically. I edit video and don’t need the speed of some of the RAID speeds I’ve seen mentioned, so was just wondering if I’m missing something.
Enjoy your new system!



May 06, 2022 at 02:52 PM
jhapeman
Offline
• • • •
Upload & Sell: On
p.4 #6 · p.4 #6 · Raid recommendtions


Ryukyu wrote:
Thank you for your explanation. It’s not even SSD vs. HDD so much as is there a need for that kind of performance, or just a desire to have it? I guess I’m thinking more practically. I edit video and don’t need the speed of some of the RAID speeds I’ve seen mentioned, so was just wondering if I’m missing something.
Enjoy your new system!


In my case it's not really a need or want for the performance. It's just that a I vastly prefer SSD for other reasons. It's dead silent, for one, and incredibly compact. I can have 32TB of storage in a device that's not much larger than two slices of bread stacked. The speed is very nice, too, although it could be half as fast and still be good. Even the quietest HDD devices are much noisier, quite bulky and much slower.



May 06, 2022 at 03:19 PM
Luvwine
Offline
• • • •
Upload & Sell: On
p.4 #7 · p.4 #7 · Raid recommendtions


Ryukyu wrote:
Thank you for your explanation. It’s not even SSD vs. HDD so much as is there a need for that kind of performance, or just a desire to have it? I guess I’m thinking more practically. I edit video and don’t need the speed of some of the RAID speeds I’ve seen mentioned, so was just wondering if I’m missing something.
Enjoy your new system!


Absolutely true. I don't need 2800+ mb/sec for anything. Something faster than 150, however, is nice to have, especially if it is also silent, reliable, and small.



May 06, 2022 at 03:54 PM
Tony Ross
Offline
• • •
Upload & Sell: Off
p.4 #8 · p.4 #8 · Raid recommendtions


Luvwine wrote:
Wow. Perhaps I should rephrase my query. I understood only part of your kind reply. I appreciate you taking the time but I am beginning to think the most important thing will be to get a raid setup that is made for dummies. The Synology enclosure (Server?) looks great but wonder how user friendly for a novice on a Mac. Any thoughts about OWC? Their 4 bay raid 5 Thunderbay seems more novice friendly tho I could be totally wrong. Thx again!


The Mac Studio has a 10GbE (10 gigabit Ethernet) port. To get good data transfer speeds you will want a NAS with a 10GbE connection AND a switch which support 10 gigabit Ethernet. Oh, and preferably Cat 6a ethernet cables (you can use Cat 6 or Cat 5e, but if you are buying new cables, get Cat 6a - they are not expensive). You need all that - if you plug the Mac Studio and the NAS into a regular router/switch you'll get 1GbE and your transfer speeds top out at 125MB/s.

Make sure you get a switch with multi-speed (1 / 2.5 / 5 / 10 gigabits/second) - if nothing else, you'll want your uplink to the rest of your LAN to run at 1 gigabit/second. Fortunately we can finally get desktop 10GbE switches - you don't have to get one that's meant for a rack mounted closet and fitted with tiny screaming fans!

I'm using two Synology DS18xx boxes (the new one is a DS1821, and it's very good). Synology software is rather friendly to beginners, and most of the defaults are fine. Still, you'll want to learn a little about your options, and there are lots of options. I'd strongly recommend not opening your NAS up to the outside world - if you are not familiar with the security issues, keep the NAS limited to inside your own LAN.




May 06, 2022 at 05:16 PM
sbay
Offline
• •
Upload & Sell: Off
p.4 #9 · p.4 #9 · Raid recommendtions


Ryukyu wrote:
Seriously, I’d really like to know what you all are doing that requires that much speed.


It can be much faster opening / saving files (very noticeable). But if you use compression (depending on the type), sometimes that is the bottleneck and then the disk speed doesn't help.



May 10, 2022 at 11:09 AM
amv8
Offline
• •
Upload & Sell: Off
p.4 #10 · p.4 #10 · Raid recommendtions


sbay wrote:
What type of problems has it caused?


Although I've generally been quite happy with SoftRaid, there's a bug in MacOS Monterey such that a RAID 5 volume setup through SoftRaid on an M1 Mac will not work unless you do some intrusive work arounds; you'll likely get system crashes. Hopefully, Apple is fixing this for the next major OS release. But for now, it makes using SoftRaid with RAID 5 a challenge on an M1 Mac.



May 13, 2022 at 11:48 PM
amv8
Offline
• •
Upload & Sell: Off
p.4 #11 · p.4 #11 · Raid recommendtions


Luvwine wrote:
I just set up Carbon Copy Cloner to copy everything daily to my external 16TB hard disk and everything is backed up continually to Backblaze. I also don't delete the information off my camera SD cards until after the data is in all three locations (two hard drives and backblaze).


Hi Luvwine - just fyi, Backblaze doesn't really backup continuously, especially for external drives. When Backblaze says "You're backed up as of Today, time" that's not necessarily true. If you've just added some files (e.g. off your SD card) and you want to know if Backblaze really has them, you need to check as if you were going to do a restore. In my experience, you sometimes have to wait as long as 24+ hours for new files to get picked up. Sounds like you're generally covered with your strategy, but just wanted to make sure you're not mislead by Backblaze's status message.



May 13, 2022 at 11:57 PM
Rajan Parrikar
Offline
• • • • •
Upload & Sell: On
p.4 #12 · p.4 #12 · Raid recommendtions


I just set up my Thunderblade 16TB SSD array from OWC as RAID 0. The silence is eerie. Glad to have noiseless operation without fans. It would be hard to go back to hard drives now.



Jun 14, 2022 at 01:43 PM
Rajan Parrikar
Offline
• • • • •
Upload & Sell: On
p.4 #13 · p.4 #13 · Raid recommendtions


I see that the Thunderblade light turns from blue to white when I switch the iMac Pro off. However, it remains blue when the iMac Pro is put to sleep. Shouldn't it turn white in the latter case as well?



Jun 14, 2022 at 08:46 PM
rji2goleez
Offline
• • • • • •
Upload & Sell: On
p.4 #14 · p.4 #14 · Raid recommendtions




Wow. Perhaps I should rephrase my query. I understood only part of your kind reply. I appreciate you taking the time but I am beginning to think the most important thing will be to get a raid setup that is made for dummies. The Synology enclosure (Server?) looks great but wonder how user friendly for a novice on a Mac. Any thoughts about OWC? Their 4 bay raid 5 Thunderbay seems more novice friendly tho I could be totally wrong. Thx again!


I've been using OWC Thunderbays for several years. I have two, each filled with 4TB HD configured in a RAID 0. One RAID 0 volume is my data volume (16TB), the second RAID 0 volume (16TB) backs up the first. I use carbon copy cloner to automate the backup in the background.



Jun 16, 2022 at 02:25 PM
elkhornsun
Offline
• • • • •
Upload & Sell: Off
p.4 #15 · p.4 #15 · Raid recommendtions


With a NAS the throughput depends primarily on the CPU, the operating system, and the ports provided. My old NAS has 1GB ports but my new one has a 10GB Ethernet port that I connect to my workstation and bypassing the router.

A two drive NAS from Synology or QNAP can provide protection as one drive is backed up to the second drive in real time. When one drive fails the second drive is still available for use. Add in a new drive to replace the failed one and the NAS will rebuild the new drive automatically.

With a NAS the available space depends on how many drives are used. With a 2-drive unit only half the combined data capacity is available (a bit less as formating takes space on the drives). With a 4 drive NAS 25% is consumed by RAID and so for example a box with four 6 TB drives will provide about 90% of 3x6 or roughly 16 TB for your use.

Recommend buying NAS specific drives and the lower RPM ones. Higher RPMs is important with a single drive in a computer or server but no gain in a NAS and the higher rpm drives run a lot hotter and cost more.

With a Mac planned I would look for a NAS that has Thunderbird as well as Ethernet ports. A NAS can be connected via a wired Ethernet connection or connected to a wireless router for access by any computer in your house.



Jun 21, 2022 at 05:47 PM
purduephotog
Offline
• •
Upload & Sell: Off
p.4 #16 · p.4 #16 · Raid recommendtions


I've run all sorts of raids for many years, both professionally and DIYd.

I also have an array of images that has several of my earliest shots corrupted because RAID-5 was inadequate to protect against multiple disks having errors during a rebuild.

So for the sake of your image archives, please learn that RAID is NOT BACKUP.

I have backups, too- and they were nicely updated with the corrupted image.

Personally I now run RAID-6 at home, and RAID-7 (ZFS-3) for mission critical information. That is then 'snapped' to another server (professionally).

I would get a cheap 8tb drive or two and do a full set of backups, then put them on the shelf, and do it yearly or whatever is appropriate for you. Offsite them to your inlaws or a trusted friend.

Anyway... Good luck.



Jun 26, 2022 at 02:42 PM
bobby350z
Offline
• • • •
Upload & Sell: On
p.4 #17 · p.4 #17 · Raid recommendtions


^^ How do you protect against disk errors? Thanks.


Jun 26, 2022 at 03:08 PM
purduephotog
Offline
• •
Upload & Sell: Off
p.4 #18 · p.4 #18 · Raid recommendtions


That's the question that costs $$$ Seriously though... and now with SSDs messing around with how things are written (aka, a 1 doesn't mean it's a 1 written to disk, it could be a 0 depending on data around it)...

I've chosen and been happy with the research I did with ZFS. ZFS writes checksums over the block data and for each part- by adding 3 disk redundancy (yeah, I know, expensive) however I can operate with multiple failures- but more importantly when I 'scrub' the data the entire drive is checked (all drives in the array).

This prevents the 'punctured raid' that just burned a program at work- there were flaws on the drive, a drive failed, someone slapped in a new drive without checking the logs... and now all the data is corrupt and not recoverable.

Best way to protect though is regular backups. And Archives. Don't be like me and overwrite your archives...



Jun 26, 2022 at 03:43 PM
molson
Offline
• • • • • • •
Upload & Sell: On
p.4 #19 · p.4 #19 · Raid recommendtions


rattlebonez wrote:
I would just delete all the out of focus, bad shots, duplicates (due to high fps), etc.
Then there would not be the need for multiple Terabytes of storage

I did exactly that and can now easily find the good in focus keeper shots


I opted for the 1TB SSD in my Studio, which forced me to do the same thing...



Jul 12, 2022 at 09:00 AM
Alan321
Offline
• • • • • •
Upload & Sell: Off
p.4 #20 · p.4 #20 · Raid recommendtions


Speedy storage is not just about letting you do your present work quicker. It also greatly speeds up maintenance tasks such as creating and subsequently testing file checksums. This is very significant when dealing with many GB per volume / drive because it needs to be done on all data rather than just a few single files. However, it is best done with multi-threaded software to get the job done quicker, and data-intensive multi-threading works best with data on speedy SSDs partly because there is none of the physical head thrashing that typically slows HDDs right down.

A very advantageous benefit of being able to do such testing quickly is that you're more likely to do it more often too.

If you can't afford an entire system of speedy SSDs then consider getting at least one so that you can copy HDD contents onto it and test the files on the SSD. The copying will be relatively fast and the testing will be super fast.

I should say that typical HDD RAIDs are generally faster than single HDDs, but they still slow down way more than SSDs will when the files are fragmented or when you're accessing files out of their stored sequence.

And finally, a warning about using SSDs - they can lose data integrity over time if they are not powered up every several months or so. An HDD can sit in a cupboard for years and then still work ok when it is connected to a computer. An SSD will not be so reliable in that situation. However, this does not impact normal computer usage patterns; you can still go on vacation without your SSDs losing their data. Just don't use them for long-term archival storage.



Aug 02, 2022 at 12:47 PM
1       2       3      
4
       5       end




FM Forums | Post-processing & Printing | Join Upload & Sell

1       2       3      
4
       5       end
    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.