Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | General Gear-talk | Join Upload & Sell

1              3       4       end
  

SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBo...

  
 
schlotz
Offline
• • • • •
Upload & Sell: On
p.2 #1 · p.2 #1 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


Ron above spelled it out and your subsequent reply confirms it. The slowness is due to LR having to access the files on the RAID in order to create a preview. While putting all the photos on a very fast NVMe solution will definitely improve photo access time, but until the previews have been created LR will still have to process each of them regardless of where they are located.

@EB-1's suggestion seems to be the best approach.

Until you actually reconfigure, I would point LR to where all the photos are on the RAID and tell it to build the previews. Library>previews. Then off to bed an let it churn. JMTC



Jun 21, 2023 at 03:02 PM
eyal
Offline
• • •
Upload & Sell: Off
p.2 #2 · p.2 #2 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


Previews worked out perfectly - thank you!

I changed my previews to 1024 size since I really only need the thumbnails until I dive further into the folder. Let it run overnight and the 25k+ images only took up an extra 3-4 gigs of space on the internal drive.

Now running through is super fast.

Thanks!



Jun 23, 2023 at 06:12 AM
eyal
Offline
• • •
Upload & Sell: Off
p.2 #3 · p.2 #3 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


Also, I have the Sandisk Extreme portable SSD which I normally use for backing up photos on vacation and decided to clean it up by doing a proper delete (FileShredder app on Mac) rather than just deleting to the trash can or formatting.

It was taking FAR longer than expected so pulled up iStats and found the drive was only reading/writing at 65 MB/s after using up the cache.

When done, I checked speeds on BlackMagic and they were 800 MB/s r/w

I can definitely appreciate the importance of cache but am honestly shocked the SSD would slow to less than spinning drive speed.



Jun 24, 2023 at 06:08 PM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.2 #4 · p.2 #4 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


eyal wrote:
Also, I have the Sandisk Extreme portable SSD which I normally use for backing up photos on vacation and decided to clean it up by doing a proper delete (FileShredder app on Mac) rather than just deleting to the trash can or formatting.

It was taking FAR longer than expected so pulled up iStats and found the drive was only reading/writing at 65 MB/s after using up the cache.

When done, I checked speeds on BlackMagic and they were 800 MB/s r/w

I can definitely appreciate the importance of cache but am honestly shocked the SSD would slow to less than spinning
...Show more

The Extreme uses QLC; the Extreme Pro uses TLC (mine has a 4TB WD SN850XE).
I don't know the MAC software, but if it overwrites more than once per cell that is really putting a lot of write cycles. I find it practical to simply use encrypted drives for travel in case they are lost or stolen. A simple format then suffices.

One of the issues especially with external SSDs is that it takes time for both garbage collection (reallocation of partial blocks) and pseudo-SLC cache recovery. Unfortunately people tend to turn off/unplug the drives in a dirty state and not give them enough time to recover. I'm not much in favor of the larger dynamic pseudo-SLC cache some makers implement, because it takes quite a while to recover, sometimes about an hour.

EBH



Jun 24, 2023 at 07:13 PM
eyal
Offline
• • •
Upload & Sell: Off
p.2 #5 · p.2 #5 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


I'm getting too old for this...

I used to know this stuff backwards and forwards but have lost touch with the intricacies of computers such as QLC, TLC, cache, garbage collection, etc.

Appreciate the info



Jun 24, 2023 at 09:17 PM
wordfool
Offline
• • • •
Upload & Sell: Off
p.2 #6 · p.2 #6 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


I might also be getting too old, but I do remember this stuff doing my head in even 10 years ago when spinning drives were still dominant for large-capacity storage. I find settling on a system of working drives and backup/archives drives that is both secure and somewhat futureproof a bit challenging.

What I'm curious about is longevity of backups/archives on SSDs vs spinning drive. Spinners need to be spun up every now and then, but IIRC that's purely a mechanical thing and the data is pretty much set for long-term storage. SSD memory cells, from what I've read, do not necessarily hang on to data as well, long term, without regular use so I'm curious if there's any solution that that or is is still the case that archives and long-term backups are best stored on spinners?

Now my 3TB of internal drives are filling up fast I'm looking for an external photo library solution for use with LR and am considering 2x4TB NVMe working drives set up in either a RAID 1 configuration or simply with some sort of daily software backup of one to the other and then offloading files to spinners for longer-term backups.



Jun 25, 2023 at 12:41 PM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.2 #7 · p.2 #7 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


There is much FUD about SSDs, but generally they are designed for 5 years just like hard drives. I have data on HDDs for 20 years and on SSDs over 10 years that was just fine. Neither HDDs not SSDs are considered archival media and no single copy of data should be considered 100% reliable. Moving data to newer media at intervals should be part of your data management plan. I have many hundreds of TB of HDDs and very few failures under 5 years; most of those in the first year. I would operate under the assumption that there is about a 10% chance of individual device failure over 5 years from DOM.

EBH



Jun 25, 2023 at 12:52 PM
rscheffler
Offline
• • • • • •
Upload & Sell: Off
p.2 #8 · p.2 #8 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


wordfool wrote:
What I'm curious about is longevity of backups/archives on SSDs vs spinning drive. Spinners need to be spun up every now and then, but IIRC that's purely a mechanical thing and the data is pretty much set for long-term storage. SSD memory cells, from what I've read, do not necessarily hang on to data as well, long term, without regular use so I'm curious if there's any solution that that or is is still the case that archives and long-term backups are best stored on spinners?


IMO the best solution is frequent migration to new storage, whether new, larger HDDs, SSDs, etc. This is also an area where the cloud can be beneficial because you're transferring hardware maintenance to the storage provider. You might pay more for the cloud than your own local storage, but this is one example of a feature you're paying for to let someone else worry about the nitty-gritty details. I'm not saying move your only archive to the cloud because other things can go wrong, like bankruptcy, massive natural disasters, etc. Rather, consider it as an additional back-up option.

From what I've read, with SSDs, you just want to plug them in every once in a while to let them run their maintenance routines. I'm sure many of us have decades-old memory cards from which we can still pull images, so the reality is the data is probably fairly durable, but also no harm in being cautious and conservative.

Now my 3TB of internal drives are filling up fast I'm looking for an external photo library solution for use with LR and am considering 2x4TB NVMe working drives set up in either a RAID 1 configuration or simply with some sort of daily software backup of one to the other and then offloading files to spinners for longer-term backups.

This sounds reasonable to me. How large is your overall photo archive?




Jun 25, 2023 at 03:37 PM
elkhornsun
Offline
• • • • •
Upload & Sell: Off
p.2 #9 · p.2 #9 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


If you had used NAS specific 5600 rpm drives you would not hear them when the spin up. Or you can lengthen the amount of time before they spin down in the NAS. With RAID there is no performance gain with faster drives that make more noise and run a lot hotter than the "green" drives engineered for use in a NAS.

The only place a SSD array can be helpful is with multiple users accessing the same data throughout the day as with a video studio.My QNAP can house SSD as well as hard drives for this purpose but as mine is a single workstation environment there is no reason to do this and cache data.

I use NVME memory modules inside the workstation to have a RAID1 2TB array to eliminate I/O performance by accessing data inside the computer when doing intensive batch processing of thousands of files. I am using Samsung 970 Plus modules and two have failed after less than 2 years of operation so not a good alternative to SSD or hard drives for storage if you have the space needed.

My NAS has a single 10GB Ethernet port and a few 1GB ports. I connect the 10GB port to my workstation and use the 1GB ports to connect to a 1GB managed switch. With the switch as a bottleneck any gains in NAS output would be negated by the switch. It is a Cisco switch that does provide 1GB throughput with all its ports which is seldom the case with low end switches that can only provide their rated data I/O through a single port and as more ports are used by more devices the throughput drops dramatically.



Jun 25, 2023 at 06:13 PM
Vivek
Offline
• • • •
Upload & Sell: On
p.2 #10 · p.2 #10 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


@EB-1, what external drive enclosure are you using? I chatted with a technical support specialist at OWC today and he confirmed something that I was suspecting. In the smaller TB3 m.2 x4 enclosure (OWC Express 4M2), each m.2 only has one PCIe Gen3 lane limiting the total throughput to about 750 MB/s even though theoretically TB3 should support 2800 MB/s. I am not sure but Thunderbay is probably similar and I need to check this.

Do you know of any enclosure that allows me to get 8TB of external space that I can access at full Thunderbolt speeds? I don't really care about the m.2 form factor as I will have this at home and it will be connected to my desktop.

Thanks in advance for any info you can share.

EB-1 wrote:
The Extreme uses QLC; the Extreme Pro uses TLC (mine has a 4TB WD SN850XE).
I don't know the MAC software, but if it overwrites more than once per cell that is really putting a lot of write cycles. I find it practical to simply use encrypted drives for travel in case they are lost or stolen. A simple format then suffices.

One of the issues especially with external SSDs is that it takes time for both garbage collection (reallocation of partial blocks) and pseudo-SLC cache recovery. Unfortunately people tend to turn off/unplug the drives in a dirty state and
...Show more




Jun 27, 2023 at 06:48 PM
 


Search in Used Dept. 

Vivek
Offline
• • • •
Upload & Sell: On
p.2 #11 · p.2 #11 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


In other words, if cost was not an issue, what would be the fastest enclosure for SSDs (either m.2 or U.2) for a Mac Studio connected to a TB4 port.


Jun 27, 2023 at 07:04 PM
eyal
Offline
• • •
Upload & Sell: Off
p.2 #12 · p.2 #12 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


Vivek wrote:
In other words, if cost was not an issue, what would be the fastest enclosure for SSDs (either m.2 or U.2) for a Mac Studio connected to a TB4 port.


Well, if cost is not an issue, this is the answer but, in addition to high cost, it is a BIG device:

https://iodyne.com/

"twelve lightning-fast NVMe SSDs, a 64GB/s PCI Express Fabric, 8 Thunderbolt ports, and incredible technology running on a System-on-a-Chip"

Cost is $4,450 for 12TB, $7,500 for 24 TB and some amount for 48 TB but their shop page won't show it to me at the moment








Jun 28, 2023 at 06:23 AM
elkhornsun
Offline
• • • • •
Upload & Sell: Off
p.2 #13 · p.2 #13 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


For the NVME cards it pays to have a way to deal with the heat buildup to prolong their life. I used a $150 PCIe board for two of these modules and had them fail and with these in a RAID1 setup when one failed I no longer had access to the data on the good module.

I upgraded to a $300 Sonnet 4x4 M.2 PCIe 3.0 board in my Windows workstation and this card has heat sinks that have a thermal connnection to the top and bottom of the M.2 modules and keeps them running cooler. The Sonnett also can move data much faster as each M.2 module has its own data channel with the 16-lane interface. I doubled the throughput of the two NVME SSDs I mounted on this card.

Cooling is critical and less expensive enclosures or adapters that do not provide this to an adequate degree will result in much shorter life for the SSD's.

I have a 10GB connection to a RAID6 NAS to my workstation and also use the NVME RAID1 inside the workstation when higher data I/O is needed.




Jun 28, 2023 at 12:14 PM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.2 #14 · p.2 #14 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


Nowadays mid and high end motherboards have heatsinks on one or both sides for the M.2 modules to keep them cool enough. The temperature specs have increased in recent years. E.g., the WD SN850X is rated at 0-85C operating temperature. With decent airflow the temperatures of the 4-8TB M.2 SSDs are in the 40s with dual-sided or 50s with only single-sided heatsinks. The 2TB and lower capacities are usually single-sided and not much of a problem to heatsink only on the top.

EBH



Jun 28, 2023 at 08:30 PM
rscheffler
Offline
• • • • • •
Upload & Sell: Off
p.2 #15 · p.2 #15 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


FWIW, my 2TB WD SN850X in a Sabrent USB 10Gbs enclosure plateaued at around 60˚C during a sustained ~1TB write. That NVME stick is single-sided and the enclosure has a thermal pad making contact with that side. Left plugged in and idle most of the time, it sits at around 45˚C. Current ambient in the room is about 26˚C.

Vivek wrote:
@EB-1@, what external drive enclosure are you using? I chatted with a technical support specialist at OWC today and he confirmed something that I was suspecting. In the smaller TB3 m.2 x4 enclosure (OWC Express 4M2), each m.2 only has one PCIe Gen3 lane limiting the total throughput to about 750 MB/s even though theoretically TB3 should support 2800 MB/s. I am not sure but Thunderbay is probably similar and I need to check this.

Do you know of any enclosure that allows me to get 8TB of external space that I can access at full Thunderbolt speeds? I don't really
...Show more

I don't have a direct answer to your question and am kind of wondering the same thing.

A reddit thread I came across recently suggested referring to this website for finding all TB3 and TB4 certified products:

https://www.thunderbolttechnology.net

For example, this page returns TB3 storage products. But of course it doesn't tell you how well these devices actually perform...

And that site only lists one TB4 storage product - the OWC miniStack STX. But OWC's page for this product states only up to 770MB/s throughput... which is quite poor.



Jun 28, 2023 at 10:14 PM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.2 #16 · p.2 #16 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


I have several of those Sabrents with the smaller drives (2TB) and also find that they reach about 60C with SN750 and the original 970 EVO+ (not the crummy post 2020 version). They are fine for reusing old SSDs and/or making your own external drives for travel. In general I far prefer internal SSDs with 4x PCIe 4.0 or faster lanes and onboard heatsinks rather than the USB drives.

EBH



Jun 28, 2023 at 10:39 PM
Vivek
Offline
• • • •
Upload & Sell: On
p.2 #17 · p.2 #17 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


Thanks Ron, I appreciate it. As of right now, as far as know, there is nothing "reasonably" priced that will even come close to the Thunderbolt performance even if we add good SSDs. As you correctly noted the Express 4M2 and miniStackSTX are both limited to 770 MB/s because of poor design (single PCIe lane).

There is nothing specific about any storage options on that page unfortunately. Also, as @eyal mentioned, the iodyne is a good start but super expensive (much worse than I thought). I hope that there are products that can be had in the next six months (Christmas??) that can solve this conundrum. Basically an Iodyne like approach but with a single or two TB3/4 ports that can allow use of PCIe Gen4 SSDs at full speed. I have to think that such devices (basically limited by the chips) are around the corner.

Thank you everyone!



I don't have a direct answer to your question and am kind of wondering the same thing.

A reddit thread I came across recently suggested referring to this website for finding all TB3 and TB4 certified products:

https://www.thunderbolttechnology.net

For example, this page returns TB3 storage products. But of course it doesn't tell you how well these devices actually perform...

And that site only lists one TB4 storage product - the OWC miniStack STX. But OWC's page for this product states only up to 770MB/s throughput... which is quite poor.





Jun 30, 2023 at 12:10 PM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.2 #18 · p.2 #18 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


Is TB5 coming out soon to alleviate at least some of the bottleneck? Maybe take a look at the new MAC Pro which has six PCIe slots (total 64 lanes). They are only PCIe 4.0, but you could add quite a lot of SSDs in there.

EBH



Jun 30, 2023 at 12:17 PM
rscheffler
Offline
• • • • • •
Upload & Sell: Off
p.2 #19 · p.2 #19 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?



https://www.thunderbolttechnology.net


Vivek wrote:
Thanks Ron, I appreciate it. As of right now, as far as know, there is nothing "reasonably" priced that will even come close to the Thunderbolt performance even if we add good SSDs. As you correctly noted the Express 4M2 and miniStackSTX are both limited to 770 MB/s because of poor design (single PCIe lane).

There is nothing specific about any storage options on that page unfortunately. Also, as @eyal@ mentioned, the iodyne is a good start but super expensive (much worse than I thought). I hope that there are products that can be had in the next six months
...Show more

Yes, that link only lists certified products. It's up to you to find reviews or purchase and test...

Came across this comment in the MacRumors forum (the first paragraph refers to TB3):

"A Thunderbolt link is served by 4 PCIe lanes. From within that, a total of ~22Gbit/s is available for PCIe data transfer. That's why all of the peak NVMe speeds have a ceiling of ~2750MByte/s. There are threads here on MacRumors wherein members have created software RAID-0 configurations using multiple Thunderbolt links with independent enclosures to create impressive benchmarks.

Reserved bandwidth for DisplayPort traffic is what sets the ceiling of 32Gbit/s for PCIe data in Tbolt-V4 (and 22Gbit/s in V3)."

While browsing for TB external enclosures I did come across reviews for some that apparently hit the 22Gbs limit, but didn't save that info because I'd already decide to 'settle' for the $30 10Gbs Sabrent USB enclosure...

Just had another look:

https://forums.macrumors.com/threads/thunderbolt-4-nvme-m-2-external-storage-read-write-speeds.2340127/post-32073949

M1 Studio Max.

Acacis TB3/4.

WD SN850X, 2 TB



The ACASIS 40Gbps M.2 Nvme SSD Enclosure has come up in some discussions.




Jun 30, 2023 at 01:41 PM
Vivek
Offline
• • • •
Upload & Sell: On
p.2 #20 · p.2 #20 · SSD vs NVMe and RAID4 vs JBOD config for LRc photo storage with M1 MacBook Pro?


Thanks for the link @rscheffler. This has a treasure trove of information. I think I can get an 8TB ultrafast external SSD without breaking the bank. I am traveling now but will draw this up in a diagram for everyone (specifically also for a friend and myself) that combines the speed and data protection option. Really appreciate it.

@EB-1, I think TB5 is at least a year away. The PCIe Gen5 based SSDs should also start arriving by then. I wish Acasis made an enclosure that could accept a U.2 form factor drive. This would solve many issues for me.



Jun 30, 2023 at 03:37 PM
1              3       4       end






FM Forums | General Gear-talk | Join Upload & Sell

1              3       4       end
    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.