Home · Register · Join Upload & Sell

  

  Previous versions of snapsy's message #12031805 « Using LR's Smart Previews for fast D800/E processing »

  

snapsy
Offline
Upload & Sell: On
Using LR's Smart Previews for fast D800/E processing


Adobe added a new feature called Smart Previews in Lightroom 5. The primary purpose of the feature is to allow offline editing of files, but they also provide a significant other benefit - they dramatically increase the speed for developing/editing large raw images, such as those from the D800/E or Sony A7r.

Smart Previews are lossy, lower-resolution DNG files created from your original raw files. They can be manipulated in LR the same as your original raw file but they compress down to just 5% of the original raw file\'s size. For example, one original D800 raw I compared was 35MB but compressed down to 1.9MB. For LR 5.3 the resolution for a D800/E raw file is 2560x2047 (vs the original resolution of 6144x4912). I have found that 2560x2047 is enough resolution both to confirm critical focus and to perform pixel-accurate editing...and those edits will later be applied to the full-resolution raw when you\'re ready (I\'ll describe the process soon).

The easiest way to create Smart Previews is to select the \"Build Smart Previews\" in the File Handling tab during an import. You can also manually create them by selecting the image(s) in the Library Module and then clicking the \"Original Photo\" link under the histogram (note if you select multiple images the \"Original Photo\" label changes to a number corresponding to the number of files you selected).

Adobe designed Smart Previews to solve the \"offline editing\" problem. The idea is that you import your original raws off an external HD and afterwards you\'re free to disconnect your HD so that you can edit the files offline via their Smart Previews. This is possible because LR stores the Smart Previews in your LR catalog along with all your other metadata/previews, separate from where your raw files are stored. The basic idea is a laptop with a fast SSD holding the catalog and external drive(s) holding the raw files...after creating the Smart Previews you can then take the laptop anywhere without the external drives to do your editing (like on a plane). When you\'re done editing the images and have access to your external HD again you plug the HD back in and LR will automatically apply the edits to the metadata in the catalog associated with the full-sized raw files. Since you edited a lower resolution version of the raw, LR will automatically scale/map those edits to the full-sized version.

Here\'s the rub...since this feature is designed for offline editing, LR automatically decides whether you use the Smart Preview or full-sized raw based upon the availability of the raws. If the drive holding the raws is attached you\'ll always be editing the full-sized raw; if the drive is removed you\'ll be editing the Smart Preview. Unfortunately LR didn\'t provide a way to force editing of Smart Preview when the full-sized raws are available, which means users who want the performance benefits of Smart Previews but don\'t need the offline feature either need to put the images on a drive that can be removed/ejected to force LR into Smart Preview mode or come up with a way to create a mountable filesystem alias to those files so that you can mount/dismount them virtually...I\'ve come up with a process for this and will describe shortly but first the benefits...

Performance Benefits

I ran some benchmarks using 50 D800 NEFs. The total filesize was 1.79GB, for an average filesize of 36.7MB. Using my i7-3770K system with 32GB of memory and HD\'s that do around 150 MB/s sequential reads I got the following results for the LR import:

Import with 1:1 previews, no Smart Previews: 3 minutes 15 seconds
Import with standard previews, no Smart Previews: 15 seconds
Import with minimal previews + Smart Previews: 38 seconds

My pre-LR5 D800 workflow was to import with 1:1 previews, which is very slow for the import but when done allows the files to be viewed at 100% in the develop tab without the usual 3-5 second on-demand demosaic penalty...basically it\'s trading import speed for initial viewing/develop speed. While this solves the initial performance penalty it doesn\'t address the general sluggishness of LR when doing edits of the full-sized raw...and worse, sometimes LR re-renders the raw after major edits or relaunching, incurring an additional 3-5 second demosaic.

My LR5 workflow is to import with minimal previews + Smart Previews. The traditional previews like 1:1 aren\'t performance-beneficial anymore with Smart Previews so minimal previews are now sufficient. When the import is done I then disconnect the drive \"virtually\" and start working with the Smart Previews. The initial render time of each Smart Preview in the develop tab is much faster than the 1:1 render of the full raw since the file is much smaller/lower-res...instead of 3-5 seconds it completes in about 1/4 second. Even better, once that fast render is done the editing tools in the develop tab are much snappier on the Smart Preview than with the full-size raw. What\'s neat about the Smart Previews is that you can export them to TIFF/JPEGs like the full-size raws but just obviously with lower resolution...that way you can create proofs, still without access to the raws (and faster too). Once I\'m done with all my edits I then \"virtually\" reconnect the drive and LR automatically will apply the edits to the full-sized raws on demand.

Virtually Connecting/Disconnecting Drives

Edit: \'awad\' had a much simpler idea than what I described below...simply change the name of the folder containing the raws and LR will revert to the Smart Previews...then change the name back when you want to apply those edits to the raws.

Windows

* Right-click on desktop, and select New->Shortcut from the popup. For Location, type \"%windir%\\system32\\subst.exe s: c:\\my_pics\" (without the quotes). Replace \"c:\\my_pics\" with the base directory of where you store all your images. I\'ve selected drive \"S:\" arbitrarily but you can use any available local drive letter that is not already provisioned. Click Next. For the shortcut name you can type \"Mount Photos\"
* Right-click on desktop, and select New->Shortcut from the popup. For Location, type \"%windir%\\system32\\subst.exe s: /d\" (without the quotes). Click Next. For the shortcut name you can type \"Dismount Photos\"
* Whenever you want to import new photos, copy them to your HD as you would normally but do the import from your subst drive alias (S: in the sample above). For example, if your new photos are at c:\\my_pics\\vacation_photos, you would import the files into LR form s:\\vacation_photos.
* When the import is done you can then virtually \"disconnect\" the alias drive by clicking your \"Dismount Photos\" shortcut and start your LR edits.
* When your LR edits are done you can then virtually \"reconnect\" the alias drive by clicking your \"Mount Photos\" shortcut so that LR can synchronize your edits to the full-sized raws.

Mac OS X

The process for the Mac is a little more involved because OS X doesn\'t have an equivalent to Microsoft\'s subst command. There are probably other ways to do this but the method I use involves using Google\'s free \"bindfs\" utility that lets you mount an alias subdirectory. Here are the steps:

* Install MacPorts, which is an open-source utility that enables the easily installation of command-line utility packages that are distributed online (similar to the package managers available on Linux). This usually gets installed to /opt/local/bin/port.
* Install Google\'s \"bindfs\" by opening a command window and typing \"sudo port install bindfs\". This will download bindfs from an online repository and then install it, usually to /opt/local/bin/bindfs
* You first need to create a dummy directory that will serve as the alias to the actual directory holding your images. It will be this dummy directory that you\'ll dynamically bind/link and unbind/unlink for LR. You can create the dummy directory anywhere except for a directory within the tree where your image files are; I usually create it at the same directory level as my image files. For example if my images are at /data/my_images, I\'ll create a directory called /data/link_to_my_images.
* Now you need to create two small script files in a text editor that you can use to quickly mount and dismount the virtual binding to your files. I created one called bind_create.sh.command and its contents are \"bindfs /Volumes/MyHD/Users/joeblow/data/my_images <space> /Volumes/MyHD/Users/joeblow/data/link_to_my_images\" (replace <space> with an actual space). The next file I called bind_delete.sh.command its contents are \"umount /Volumes/MyHD/Users/joeblow/data/link_to_my_images\"
* Next add execute privilege to these scripts by doing a \"chmod+x bind_create.sh.command\" and \"chmod+x bind_delete.sh.command\" inside a command window.
* And finally you\'re done...whenever you want to mount the virtual volume for importing files into lightrooom/synchronizing Smart Preview edits, double-click the bind_create.sh.command in the Finder. To dismount the volume you can either double-click the bind_delete.sh.command script or simply select the virtual point of the volume in the finder and Eject it.



Dec 30, 2013 at 02:28 AM
snapsy
Offline
Upload & Sell: On
Using LR's Smart Previews for fast D800/E processing


Adobe added a new feature called Smart Previews in Lightroom 5. The primary purpose of the feature is to allow offline editing of files, but they also provide a significant other benefit - they dramatically increase the speed for developing/editing large raw images, such as those from the D800/E or Sony A7r.

Smart Previews are lossy, lower-resolution DNG files created from your original raw files. They can be manipulated in LR the same as your original raw file but they compress down to just 5% of the original raw file\'s size. For example, one original D800 raw I compared was 35MB but compressed down to 1.9MB. For LR 5.3 the resolution for a D800/E raw file is 2560x2047 (vs the original resolution of 6144x4912). I have found that 2560x2047 is enough resolution both to confirm critical focus and to perform pixel-accurate editing...and those edits will later be applied to the full-resolution raw when you\'re ready (I\'ll describe the process soon).

The easiest way to create Smart Previews is to select the \"Build Smart Previews\" in the File Handling tab during an import. You can also manually create them by selecting the image(s) in the Library Module and then clicking the \"Original Photo\" link under the histogram (note if you select multiple images the \"Original Photo\" label changes to a number corresponding to the number of files you selected).

Adobe designed Smart Previews to solve the \"offline editing\" problem. The idea is that you import your original raws off an external HD and afterwards you\'re free to disconnect your HD so that you can edit the files offline via their Smart Previews. This is possible because LR stores the Smart Previews in your LR catalog along with all your other metadata/previews, separate from where your raw files are stored. The basic idea is a laptop with a fast SSD holding the catalog and external drive(s) holding the raw files...after creating the Smart Previews you can then take the laptop anywhere without the external drives to do your editing (like on a plane). When you\'re done editing the images and have access to your external HD again you plug the HD back in and LR will automatically apply the edits to the metadata in the catalog associated with the full-sized raw files. Since you edited a lower resolution version of the raw, LR will automatically scale/map those edits to the full-sized version.

Here\'s the rub...since this feature is designed for offline editing, LR automatically decides whether you use the Smart Preview or full-sized raw based upon the availability of the raws. If the drive holding the raws is attached you\'ll always be editing the full-sized raw; if the drive is removed you\'ll be editing the Smart Preview. Unfortunately LR didn\'t provide a way to force editing of Smart Preview when the full-sized raws are available, which means users who want the performance benefits of Smart Previews but don\'t need the offline feature either need to put the images on a drive that can be removed/ejected to force LR into Smart Preview mode or come up with a way to create a mountable filesystem alias to those files so that you can mount/dismount them virtually...I\'ve come up with a process for this and will describe shortly but first the benefits...

Performance Benefits

I ran some benchmarks using 50 D800 NEFs. The total filesize was 1.79GB, for an average filesize of 36.7MB. Using my i7-3770K system with 32GB of memory and HD\'s that do around 150 MB/s sequential reads I got the following results for the LR import:

Import with 1:1 previews, no Smart Previews: 3 minutes 15 seconds
Import with standard previews, no Smart Previews: 15 seconds
Import with minimal previews + Smart Previews: 38 seconds

My pre-LR5 D800 workflow was to import with 1:1 previews, which is very slow for the import but when done allows the files to be viewed at 100% in the develop tab without the usual 3-5 second on-demand demosaic penalty...basically it\'s trading import speed for initial viewing/develop speed. While this solves the initial performance penalty it doesn\'t address the general sluggishness of LR when doing edits of the full-sized raw...and worse, sometimes LR re-renders the raw after major edits or relaunching, incurring an additional 3-5 second demosaic.

My LR5 workflow is to import with minimal previews + Smart Previews. The traditional previews like 1:1 aren\'t performance-beneficial anymore with Smart Previews so minimal previews are now sufficient. When the import is done I then disconnect the drive \"virtually\" and start working with the Smart Previews. The initial render time of each Smart Preview in the develop tab is much faster than the 1:1 render of the full raw since the file is much smaller/lower-res...instead of 3-5 seconds it completes in about 1/4 second. Even better, once that fast render is done the editing tools in the develop tab are much snappier on the Smart Preview than with the full-size raw. What\'s neat about the Smart Previews is that you can export them to TIFF/JPEGs like the full-size raws but just obviously with lower resolution...that way you can create proofs, still without access to the raws (and faster too). Once I\'m done with all my edits I then \"virtually\" reconnect the drive and LR automatically will apply the edits to the full-sized raws on demand.

Virtually Connecting/Disconnecting Drives

Windows

* Right-click on desktop, and select New->Shortcut from the popup. For Location, type \"%windir%\\system32\\subst.exe s: c:\\my_pics\" (without the quotes). Replace \"c:\\my_pics\" with the base directory of where you store all your images. I\'ve selected drive \"S:\" arbitrarily but you can use any available local drive letter that is not already provisioned. Click Next. For the shortcut name you can type \"Mount Photos\"
* Right-click on desktop, and select New->Shortcut from the popup. For Location, type \"%windir%\\system32\\subst.exe s: /d\" (without the quotes). Click Next. For the shortcut name you can type \"Dismount Photos\"
* Whenever you want to import new photos, copy them to your HD as you would normally but do the import from your subst drive alias (S: in the sample above). For example, if your new photos are at c:\\my_pics\\vacation_photos, you would import the files into LR form s:\\vacation_photos.
* When the import is done you can then virtually \"disconnect\" the alias drive by clicking your \"Dismount Photos\" shortcut and start your LR edits.
* When your LR edits are done you can then virtually \"reconnect\" the alias drive by clicking your \"Mount Photos\" shortcut so that LR can synchronize your edits to the full-sized raws.

Mac OS X

The process for the Mac is a little more involved because OS X doesn\'t have an equivalent to Microsoft\'s subst command. There are probably other ways to do this but the method I use involves using Google\'s free \"bindfs\" utility that lets you mount an alias subdirectory. Here are the steps:

* Install MacPorts, which is an open-source utility that enables the easily installation of command-line utility packages that are distributed online (similar to the package managers available on Linux). This usually gets installed to /opt/local/bin/port.
* Install Google\'s \"bindfs\" by opening a command window and typing \"sudo port install bindfs\". This will download bindfs from an online repository and then install it, usually to /opt/local/bin/bindfs
* You first need to create a dummy directory that will serve as the alias to the actual directory holding your images. It will be this dummy directory that you\'ll dynamically bind/link and unbind/unlink for LR. You can create the dummy directory anywhere except for a directory within the tree where your image files are; I usually create it at the same directory level as my image files. For example if my images are at /data/my_images, I\'ll create a directory called /data/link_to_my_images.
* Now you need to create two small script files in a text editor that you can use to quickly mount and dismount the virtual binding to your files. I created one called bind_create.sh.command and its contents are \"bindfs /Volumes/MyHD/Users/joeblow/data/my_images <space> /Volumes/MyHD/Users/joeblow/data/link_to_my_images\" (replace <space> with an actual space). The next file I called bind_delete.sh.command its contents are \"umount /Volumes/MyHD/Users/joeblow/data/link_to_my_images\"
* Next add execute privilege to these scripts by doing a \"chmod+x bind_create.sh.command\" and \"chmod+x bind_delete.sh.command\" inside a command window.
* And finally you\'re done...whenever you want to mount the virtual volume for importing files into lightrooom/synchronizing Smart Preview edits, double-click the bind_create.sh.command in the Finder. To dismount the volume you can either double-click the bind_delete.sh.command script or simply select the virtual point of the volume in the finder and Eject it.



Dec 30, 2013 at 01:03 AM





  Previous versions of snapsy's message #12031805 « Using LR's Smart Previews for fast D800/E processing »

 




This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.