Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | Nikon Forum | Join Upload & Sell

1       2       3      
4
       5       end
  

Archive 2017 · HD capacity: Your storage strategy?

  
 
Mike Veltri
Offline
• • •
Upload & Sell: Off
p.4 #1 · p.4 #1 · HD capacity: Your storage strategy?


In my desktop
C: Samsung MVMe SSD 960 Pro m.2 512GB
D: Samsung 850 EVO 512GB
E: WD 4TB Black
F: WD 4TB Black
G: WD 4TB Black
Qnap 2 Bay Nas with 2 x 4TB Red



Sep 23, 2017 at 04:45 AM
Pavel
Offline
• • • • •
Upload & Sell: Off
p.4 #2 · p.4 #2 · HD capacity: Your storage strategy?


EB-1 wrote:
I've deliberately pulled a drive out and repaired an array afterwards.

I know many are gung-ho about the ZFS, but really how did all the data survive before that? I've tested data to nearly two orders of magnitude above the theoretical with the checksums on EXT4 and without ECC with never an issue.

EBH


Smaller drives.

I'm only repeating what NetApp teaches. Netapp supplies just about everyone from appe to google with the hardware to run their data centers.

We run pathetically cheaper hard drives and systems that power them, with far less knowledge and consideration to all the aspects inherent to storage and the drives get bigger, bigger and bigger. Apparently is the this increase in size that strains the old practices.

You touch in a point I'd like to make special mention of. Yes, a lot of people are somewhat smitten by ZFS and the way I wrote about it, one can infer that I think it's wonderful. I don't'.

It's a wonderful solution for industrial applications and companies with deep pockets and experts on staff. With my middling knowledge and experience of Freenas, and sparse knowledge of NetApp files and FreeBSD (which I've played with for over ten years but don't use except to tinker) I came away with the conclusion that ZFS is a bit like running a nuclear reactor. Overkill . Possible complete kill with one dumb easy to make move, as well.

I was particularly impressed that upon re-purposing one 2tb drive out of a decommissioned four drive pool, I could not re-format it and destroy it using windows. After a few re-formats, which appeared to work, using windows 8, but which Windows could never write to I put it back, and was a bit stunned that everything was back intact in the pool. I had to use GParted, that worked fine. Maybe I did something basic and dumb, but again it goes back to the point that I wan't simple. So ZFS IS fantastic in it's purposed scenario, but for a regular home user, I'm not really much of a fan. And 16 gigs of registered memory is just a starting point. It's also really fussy about hardware - esp FreeNas.

No, too much fussing and IT related geekery for me now. I just want to use my computer, not administer anything.

But 8 tb drives and growing fast in capacity, apparently makes old best practices out of date, and the same with old File systems.

I'll repeat it, just in case the fine point was lost. I don't really care about the Nth degree of security for my photos and files. But some here state that they do. They are the ones that should investigate new more appropriate technology to support that desire for the safety of their valued data. Someone anecdotally saying "I've never had a problem" is just another case of internet folly spreading spreading around.

Remember some want maximum security. "it's worked fer me so far" using poor out of date, sub-par methodology and selling it off as "good enough" - is not for them.



Sep 23, 2017 at 12:07 PM
Pavel
Offline
• • • • •
Upload & Sell: Off
p.4 #3 · p.4 #3 · HD capacity: Your storage strategy?


I've taken the trouble to cut and paste a small section from wikepedia on ZFS, just on the scope of it's reason for being, what it addresses that other file systems do not - or more correctly - did not once do. Some are catching up, though notably not NTFS and it's more robust replacement has fragmented life for us as it is only at this time for Server.

I don't need it's benefits. I don't worry too much about things like raid 5's write holes. But these things are there and they are slowly becoming more prominent with monster sized drives and arrays.

Wikepedia: "One major feature that distinguishes ZFS from other file systems is that it is designed with a focus on data integrity by protecting the user's data on disk against silent data corruption caused by data degradation, current spikes, bugs in disk firmware, phantom writes (the previous write did not make it to disk), misdirected reads/writes (the disk accesses the wrong block), DMA parity errors between the array and server memory or from the driver (since the checksum validates data inside the array), driver errors (data winds up in the wrong buffer inside the kernel), accidental overwrites (such as swapping to a live file system), etc.
A 2012 research showed that neither any of the then-major and widespread filesystems (such as UFS, Ext,[14] XFS, JFS, or NTFS) nor hardware RAID (which has some issues with data integrity) provided sufficient protection against data corruption problems."



Sep 23, 2017 at 12:12 PM
15Bit
Offline
• • • • •
Upload & Sell: Off
p.4 #4 · p.4 #4 · HD capacity: Your storage strategy?


That's a good synopsis of the advantages of ZFS.

I am actually a fan of ZFS for home users. Or perhaps more accurately, i am a fan of ZFS over Linux software RAID (which i ran previously). The reason for this is the extremely simple and easy to understand command set: Setting up a ZFS array and getting it online takes only a few (suprisingly intuitive) commands. Setting up a RAID5 with flexibile partitioning required navigating the arcane arts of mdadm and lvm. ZFS is much easier to both set up and manage. I have had a tinker with some of the more "enterprise" features too, specifically the on-the-fly data compression and the snap-shotting. Both are really useful in appropriate situations, and again not difficult to set up.

The main problem i found with ZFS for home users is that it is an enterprise disk management system and thus has a lot of features that home users don't want, but still need to navigate in the documentation. It is confusing.



Sep 23, 2017 at 01:29 PM
elkhornsun
Offline
• • • • •
Upload & Sell: Off
p.4 #5 · p.4 #5 · HD capacity: Your storage strategy?


I use a QNAP NAS that has four 3TB hard drives and provides a storage pool of 8TB. If one drive fails I have immediate access to all my data until the hard drive that failed can be replaced. The QNAP is fast and has a 1GB Ethernet connection so I can work with data files on it as though they were on an internal hard drive and with no noticeable degradation in performance.

With a 4-drive NAS the RAID5 setup costs me 25% of the total drive capacity. With a RAID1 setup I lose 50% so that two 2TB drives provides only 2TB of storage capacity - actually a lot less as formatting costs another 6-7% of a drive's capacity.

The NAS is connected to a smart hub and to a WiFi router so computers can connect using the LAN connection or via 802.11 Wifi.

Hard drives for the past 35 years have done a write verification on every bit that is written to the platters. It is only with tape drives and floppy discs and similar slow storage devices that this is omitted as it doubles the time it would take to write the data and these are already very slow in terms of I/O rates.

It would be important to know if SSD drives have this write verification but based on the failures I have encountered I do not believe that they do.Certainly the memory cards do not have this functionality. Users would not want their fps to be cut in half with data write verification.

The performance side of NAS devices depends on whether they implement RAID in software as with the less expensive units or have it implemented in hardware which makes for much faster speeds though more expensive boxes.



Sep 23, 2017 at 01:39 PM
Pavel
Offline
• • • • •
Upload & Sell: Off
p.4 #6 · p.4 #6 · HD capacity: Your storage strategy?


elkhornsun wrote:
Hard drives for the past 35 years have done a write verification on every bit that is written to the platters.


Please keep in mind that that data which is verified depends on it being correct in memory. On top of that fact is that it is not a very reliable technology. It requires ECC for soft error correction. Basically if it gets bad date from RAM it writes out and verifies the bad date to disk. And once on disk, other problems like bit rot can happen.

It's easy to be naive about data integrity, but that does not mean it's a good idea. If this wasn't a problem, it would not be the case that Cloud services would have to depend on superior technology such as ZFS.



Sep 23, 2017 at 07:10 PM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.4 #7 · p.4 #7 · HD capacity: Your storage strategy?


Clouds services may be good with avoiding data corruption, but they are notoriously unreliable.

EBH



Sep 23, 2017 at 07:23 PM
tsangc
Offline
• • •
Upload & Sell: Off
p.4 #8 · p.4 #8 · HD capacity: Your storage strategy?




roofdweller49 wrote:
What software are you using to organize this? Like how do I ensure all my photos are duplicated on two drives without copying the entire directory each time in Windows?


I am using a small utility called Create Synchronicity:

http://synchronicity.sourceforge.net/

But there are many other options. I used a Microsoft add on called SyncToy too. The major reason why I like this is that it's not a mirror, if I accidentally delete a file on primary, it doesn't get deleted on secondary.



Sep 24, 2017 at 08:08 AM
msalvetti
Offline
• • • •
Upload & Sell: On
p.4 #9 · p.4 #9 · HD capacity: Your storage strategy?


I also use SyncToy, but I'm interested in other options. Microsoft doesn't support this anymore, and I figure soon it may no longer work in Windows.

I saw that CanadaMark uses SyncBackPro (Post #7 on this thread). Does that offer some type of verification to check on data corruption in the backup files? Trying to figure this out from their website, but it's not obvious to me.

I have to admit that my eyes glazed over during the ZFS discussion above. Why is this so difficult? It seems like it should be pretty straightforward.

Mark



Sep 24, 2017 at 10:27 AM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.4 #10 · p.4 #10 · HD capacity: Your storage strategy?


I use several tools, including SmartSync Pro. It has a number of options including on the fly verification/validation and a bytewise comparison of all data on the source and target after the fact. It also provides for saving a fixed number of file versions and various other backup/sync features. The sync software has nothing to do with the FS, but should be able to verify that the data is the same on source and target.

EBH



Sep 24, 2017 at 10:55 AM
IndyFab
Offline
• • • • •
Upload & Sell: On
p.4 #11 · p.4 #11 · HD capacity: Your storage strategy?


gdanmitchell wrote:
bs kite wrote:
I'm also (some would think) fanatical about backups. I keep multiple copies of all my files. I've learned the lessons of what happens if I'm not extremely careful with backups and I would go to almost any expense to avoid repeating the losses. (The last serious losses were at least a couple of decades ago, by the way.)

Agree Dan, multiple backups are essential if you value your images, and everything else on your computer .



Sep 24, 2017 at 01:01 PM
skasol
Offline
• • • •
Upload & Sell: Off
p.4 #12 · p.4 #12 · HD capacity: Your storage strategy?


gdanmitchell wrote:
I should probably make this an article at my blog and link to it, but here is the short (!) story.

I'm using a 27" iMac. I do not use the internal iMac storage for photography files. I use an external 8TB Thunderbolt drive that stores raw files and processed photoshop files. I keep all raw and post-processed files online on this drive. I keep almost all raw files, deleting only true disasters in most cases. Hard drive storage is relatively inexpensive, but finding out that you deleted a file that you need is not.

I'm also (some would think) fanatical about
...Show more

Which 8TB Thunderbolt Drive do you use? I am thinking about picking up a couple to do the same, I have a 4TB Thunderbolt that I want to replace as it's getting full.



Sep 24, 2017 at 02:31 PM
Herb
Offline
• • • •
Upload & Sell: On
p.4 #13 · p.4 #13 · HD capacity: Your storage strategy?


jcolwell wrote:
I have three archive drives in my desktop tower (2 TB, 2 TB, 3 TB), with three, external mirrored-drives that are stored in a fireproof box. I have about 500 GB clear on the active archive drive, which will probably be filled this fall. I shot 110 GB of photos in Utah, over the past two weeks.


Shooting 110 GB of photos is exactly the problem....it doesnt take much of an event or travel to add up the storage requirements. As the camera manufacturers increase file sizes the rate with which storage manufacturers must also increase to keep up with them.....I worry that the camera guys are running faster than the storage guys......UGH!



Sep 24, 2017 at 04:55 PM
Paul Mo
Offline
• • • • •
Upload & Sell: Off
p.4 #14 · p.4 #14 · HD capacity: Your storage strategy?


I've run FreeNAS and Open Media Vault (OMV), YMMV, but I can't recommend them. They are, IMO, for tinkerers and too prone to breaking.





Sep 24, 2017 at 06:39 PM
EB-1
Offline
• • • • • • •
Upload & Sell: Off
p.4 #15 · p.4 #15 · HD capacity: Your storage strategy?


jcolwell wrote:
I have three archive drives in my desktop tower (2 TB, 2 TB, 3 TB), with three, external mirrored-drives that are stored in a fireproof box. I have about 500 GB clear on the active archive drive, which will probably be filled this fall. I shot 110 GB of photos in Utah, over the past two weeks.

Herb wrote:
Shooting 110 GB of photos is exactly the problem....it doesnt take much of an event or travel to add up the storage requirements. As the camera manufacturers increase file sizes the rate with which storage manufacturers must also increase to keep up with them..... I worry that the camera guys are running faster than the storage guys......UGH!


Not really. In Q4 2002 hard drives were about 200GB when RAW files (1Ds) were 11MB. In Q2 2015 the hard drives were up to 8TB and the files were about 65MB (5DsR). The HDDs increased nearly 20x compared to 6x for the files over that period. Of course numbers will vary depending on introduction dates, but the media is outpacing the cameras. 12TB HDDs are supposed to start shipping in volume this Q4.

EBH



Sep 24, 2017 at 08:08 PM
daggah
Offline
• •
Upload & Sell: Off
p.4 #16 · p.4 #16 · HD capacity: Your storage strategy?


My main storage solution is unRAID. I have an 18 TB usable (4x6 TB drives, one is parity) media server...it mostly hosts TV shows and movies, but archives my photography as well. I need to organize my backups as I have numerous external drives laying about. I think I have close to 70 TB of drives at home, but none of it is really organized.


Sep 25, 2017 at 01:18 AM
DES-1
Offline
• •
Upload & Sell: On
p.4 #17 · p.4 #17 · HD capacity: Your storage strategy?


I strive to keep my system simple and the drive strategy evolutionary. Late 2012 Mini, it is blazing fast. Can't imagine a home user needing more performance.

For backup images, I use Seagate external 5TB drives partitioned to reflect my 3 active drives. Carbon Copy makes exact bootable copies of each. I have 2 backup drives and rotate their usage.

Drives are:
a) a scratch drive 128GB SSD designated in LR & PS, also holds LR previews, and other non-save items. Never backed up.
b) System, right now a 512GB SSD. Mounted internally in the Mini.
c) Active Photography - a 1TB SSD to hold newest work. LR used for all organization. Folders are by year, and within by date.
d) Repository Photos - a 2 TB rotating drive which holds all older years.

When the Active gets uncomfortably full, I move entire year folders to the Repository.
I still have lots of space but in the future will incrementally upgrade each as drives age.


I use a LR front end workflow, culling images upon input. It's senseless to keep 10 of everything. Gotta train yourself to do this. Costco now has 6TB externals for cheap, but expect by the time my partition must be revised upwards there will be cheap 8-10-12's available (by cheap I mean around 100 bucks).




Sep 25, 2017 at 07:39 AM
MintMar
Offline
• • •
Upload & Sell: Off
p.4 #18 · p.4 #18 · HD capacity: Your storage strategy?


ZFS? Did ZFS make it already to the Linux kernel?


Sep 25, 2017 at 07:54 AM
Monito
Offline
• • • • • •
Upload & Sell: Off
p.4 #19 · p.4 #19 · HD capacity: Your storage strategy?



Without reading pages 2 and 3 of this thread, my strategy:

At least three copies of every file, one each on three disks, one of the disks off-site.
Additionally, use a simple system. By keeping a copy off-site, it protects against fire, theft, and panicked flailing around.

In-system:

500 GB C: System drive; over-ride defaults to keep data off this.
1.5 TB Personal drive: music, code, website source, notes, inspirational images, momentos
1.5 TB Raw photos
1.0 TB Other Photo: work in progress, processed photos, portfolios, notes
3.0 TB Backup for everything. Backed up manually with 8 WinMerge scripts, after important shoots and milestones in work. Sits in a bay with a front opening door for easy removal while system is shut down.

Off-site:
3.0 TB Twin of the backup, kept in safe deposit box at bank, swapped monthly ideally but practically about once every 6 weeks on average.
Additional intention to fit the SDB: A DVD full of as much as fits of portfolio Raws with TIFFs and final print quality JPEGs. A USB stick of a larger set of Raws, TIFFs, & JPEGs. Keep meaning to do this on a systematic basis.

Note: None of the drives are full, so the backup drives are about 70% full currently.

I've been doing this systematically for about a decade; almost as long as I've had DSLRs. (I have some files that are over 20 years old). As drives fill up, I migrate bigger drives down and buy a new pair of biggest drives for the backups. If I were doing such volumes of work that the backup didn't fit on one or two drives I'd probably go to a trio of NAS and do the swap thing offsite somewhere other than a bank.

I only have to retrieve a file or small set of files from the backup about once a year on average (due to careless deletion / overwriting). Never had to rely on the off-site backup, but since it is rotated once a month, it is known to be quite reliable (nothing is 100% reliable; impermanence is the way of all things). Never had a drive fail but a family member did and I had backed her up so almost no loss. Never lost a file due to any bit rot. Have lost about seven photos due to the weakest link: in-camera card writing errors or card download errors.




Sep 25, 2017 at 06:59 PM
kosin
Offline
• • • •
Upload & Sell: On
p.4 #20 · p.4 #20 · HD capacity: Your storage strategy?


Even though external 8TB drives are cheap these days: http://amzn.to/2jYtCB2 I stopped backing up this way and ended up going with the NAS/RAID option



Sep 25, 2017 at 08:09 PM
1       2       3      
4
       5       end




FM Forums | Nikon Forum | Join Upload & Sell

1       2       3      
4
       5       end
    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.