Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

FM Forum Rules
Wedding Resource List
  

FM Forums | Wedding Photographer | Join Upload & Sell

1
       2       end
  

Archive 2013 · Using LR's Smart Previews for fast D800/E processing

  
 
snapsy
Online
• • • • • •
Upload & Sell: On
p.1 #1 · p.1 #1 · Using LR's Smart Previews for fast D800/E processing


Adobe added a new feature called Smart Previews in Lightroom 5. The primary purpose of the feature is to allow offline editing of files, but they also provide a significant other benefit - they dramatically increase the speed for developing/editing large raw images, such as those from the D800/E or Sony A7r.

Smart Previews are lossy, lower-resolution DNG files created from your original raw files. They can be manipulated in LR the same as your original raw file but they compress down to just 5% of the original raw file's size. For example, one original D800 raw I compared was 35MB but compressed down to 1.9MB. For LR 5.3 the resolution for a D800/E raw file is 2560x2047 (vs the original resolution of 6144x4912). I have found that 2560x2047 is enough resolution both to confirm critical focus and to perform pixel-accurate editing...and those edits will later be applied to the full-resolution raw when you're ready (I'll describe the process soon).

The easiest way to create Smart Previews is to select the "Build Smart Previews" in the File Handling tab during an import. You can also manually create them by selecting the image(s) in the Library Module and then clicking the "Original Photo" link under the histogram (note if you select multiple images the "Original Photo" label changes to a number corresponding to the number of files you selected).

Adobe designed Smart Previews to solve the "offline editing" problem. The idea is that you import your original raws off an external HD and afterwards you're free to disconnect your HD so that you can edit the files offline via their Smart Previews. This is possible because LR stores the Smart Previews in your LR catalog along with all your other metadata/previews, separate from where your raw files are stored. The basic idea is a laptop with a fast SSD holding the catalog and external drive(s) holding the raw files...after creating the Smart Previews you can then take the laptop anywhere without the external drives to do your editing (like on a plane). When you're done editing the images and have access to your external HD again you plug the HD back in and LR will automatically apply the edits to the metadata in the catalog associated with the full-sized raw files. Since you edited a lower resolution version of the raw, LR will automatically scale/map those edits to the full-sized version.

Here's the rub...since this feature is designed for offline editing, LR automatically decides whether you use the Smart Preview or full-sized raw based upon the availability of the raws. If the drive holding the raws is attached you'll always be editing the full-sized raw; if the drive is removed you'll be editing the Smart Preview. Unfortunately LR didn't provide a way to force editing of Smart Preview when the full-sized raws are available, which means users who want the performance benefits of Smart Previews but don't need the offline feature either need to put the images on a drive that can be removed/ejected to force LR into Smart Preview mode or come up with a way to create a mountable filesystem alias to those files so that you can mount/dismount them virtually...I've come up with a process for this and will describe shortly but first the benefits...

Performance Benefits

I ran some benchmarks using 50 D800 NEFs. The total filesize was 1.79GB, for an average filesize of 36.7MB. Using my i7-3770K system with 32GB of memory and HD's that do around 150 MB/s sequential reads I got the following results for the LR import:

Import with 1:1 previews, no Smart Previews: 3 minutes 15 seconds
Import with standard previews, no Smart Previews: 15 seconds
Import with minimal previews + Smart Previews: 38 seconds

My pre-LR5 D800 workflow was to import with 1:1 previews, which is very slow for the import but when done allows the files to be viewed at 100% in the develop tab without the usual 3-5 second on-demand demosaic penalty...basically it's trading import speed for initial viewing/develop speed. While this solves the initial performance penalty it doesn't address the general sluggishness of LR when doing edits of the full-sized raw...and worse, sometimes LR re-renders the raw after major edits or relaunching, incurring an additional 3-5 second demosaic.

My LR5 workflow is to import with minimal previews + Smart Previews. The traditional previews like 1:1 aren't performance-beneficial anymore with Smart Previews so minimal previews are now sufficient. When the import is done I then disconnect the drive "virtually" and start working with the Smart Previews. The initial render time of each Smart Preview in the develop tab is much faster than the 1:1 render of the full raw since the file is much smaller/lower-res...instead of 3-5 seconds it completes in about 1/4 second. Even better, once that fast render is done the editing tools in the develop tab are much snappier on the Smart Preview than with the full-size raw. What's neat about the Smart Previews is that you can export them to TIFF/JPEGs like the full-size raws but just obviously with lower resolution...that way you can create proofs, still without access to the raws (and faster too). Once I'm done with all my edits I then "virtually" reconnect the drive and LR automatically will apply the edits to the full-sized raws on demand.

Virtually Connecting/Disconnecting Drives

Edit: 'awad' had a much simpler idea than what I described below...simply change the name of the folder containing the raws and LR will revert to the Smart Previews since it can't find the raws anymore...then change the name back when you want to apply those edits to the raws.

Windows

* Right-click on desktop, and select New->Shortcut from the popup. For Location, type "%windir%\system32\subst.exe s: c:\my_pics" (without the quotes). Replace "c:\my_pics" with the base directory of where you store all your images. I've selected drive "S:" arbitrarily but you can use any available local drive letter that is not already provisioned. Click Next. For the shortcut name you can type "Mount Photos"
* Right-click on desktop, and select New->Shortcut from the popup. For Location, type "%windir%\system32\subst.exe s: /d" (without the quotes). Click Next. For the shortcut name you can type "Dismount Photos"
* Whenever you want to import new photos, copy them to your HD as you would normally but do the import from your subst drive alias (S: in the sample above). For example, if your new photos are at c:\my_pics\vacation_photos, you would import the files into LR form s:\vacation_photos.
* When the import is done you can then virtually "disconnect" the alias drive by clicking your "Dismount Photos" shortcut and start your LR edits.
* When your LR edits are done you can then virtually "reconnect" the alias drive by clicking your "Mount Photos" shortcut so that LR can synchronize your edits to the full-sized raws.

Mac OS X

The process for the Mac is a little more involved because OS X doesn't have an equivalent to Microsoft's subst command. There are probably other ways to do this but the method I use involves using Google's free "bindfs" utility that lets you mount an alias subdirectory. Here are the steps:

* Install MacPorts, which is an open-source utility that enables the easily installation of command-line utility packages that are distributed online (similar to the package managers available on Linux). This usually gets installed to /opt/local/bin/port.
* Install Google's "bindfs" by opening a command window and typing "sudo port install bindfs". This will download bindfs from an online repository and then install it, usually to /opt/local/bin/bindfs
* You first need to create a dummy directory that will serve as the alias to the actual directory holding your images. It will be this dummy directory that you'll dynamically bind/link and unbind/unlink for LR. You can create the dummy directory anywhere except for a directory within the tree where your image files are; I usually create it at the same directory level as my image files. For example if my images are at /data/my_images, I'll create a directory called /data/link_to_my_images.
* Now you need to create two small script files in a text editor that you can use to quickly mount and dismount the virtual binding to your files. I created one called bind_create.sh.command and its contents are "bindfs /Volumes/MyHD/Users/joeblow/data/my_images <space> /Volumes/MyHD/Users/joeblow/data/link_to_my_images" (replace <space> with an actual space). The next file I called bind_delete.sh.command its contents are "umount /Volumes/MyHD/Users/joeblow/data/link_to_my_images"
* Next add execute privilege to these scripts by doing a "chmod+x bind_create.sh.command" and "chmod+x bind_delete.sh.command" inside a command window.
* And finally you're done...whenever you want to mount the virtual volume for importing files into lightrooom/synchronizing Smart Preview edits, double-click the bind_create.sh.command in the Finder. To dismount the volume you can either double-click the bind_delete.sh.command script or simply select the virtual point of the volume in the finder and Eject it.


Edited on Dec 30, 2013 at 02:30 AM · View previous versions



Dec 30, 2013 at 01:03 AM
SloPhoto
Offline
• • • •
Upload & Sell: Off
p.1 #2 · p.1 #2 · Using LR's Smart Previews for fast D800/E processing


Listen to the preacher people!

I have very very little reason to edit with full raws. The smart preview dng files are all I need to visualize the end result (besides NR).

My "working drive" is a very fast nas (I max out gigE), but I simply unplug my Ethernet cable when I want to start editing and LR automatically works with the smart previews. I plug back in before I do the export. Works like a charm.

If you had an external esata or usb3/thunderbolt drive it would likely be just as convenient.



Dec 30, 2013 at 01:56 AM
awad
Offline
• •
Upload & Sell: Off
p.1 #3 · p.1 #3 · Using LR's Smart Previews for fast D800/E processing


whoa whoa whoa. you made that way too complicated.

all you have to do is change the name of the folder that contains the RAW images. I change my folder name from "RAWs" to "aRAWs" and LR "loses" the files and forces the smart previews to show up.



Dec 30, 2013 at 02:21 AM
snapsy
Online
• • • • • •
Upload & Sell: On
p.1 #4 · p.1 #4 · Using LR's Smart Previews for fast D800/E processing


awad wrote:
whoa whoa whoa. you made that way too complicated.

all you have to do is change the name of the folder that contains the RAW images. I change my folder name from "RAWs" to "aRAWs" and LR "loses" the files and forces the smart previews to show up.


haha, you're right. That works too, thanks! I updated the OP with your idea.



Dec 30, 2013 at 02:25 AM
playerofwar
Offline
• • •
Upload & Sell: Off
p.1 #5 · p.1 #5 · Using LR's Smart Previews for fast D800/E processing


Love it! Looks very usefull for less powerfull systems!
I do think the way you edit the SP's and then reallocate the raws, you will see a difference in editing/colours. It just can't be that a 2mb file delivers all the information of a 35mb file...
Did you check this out?



Dec 30, 2013 at 02:36 AM
heikoM
Offline
• • • •
Upload & Sell: Off
p.1 #6 · p.1 #6 · Using LR's Smart Previews for fast D800/E processing


I am a computer dummy, but I think this is revolutionary...

heiko



Dec 30, 2013 at 03:00 AM
SloPhoto
Offline
• • • •
Upload & Sell: Off
p.1 #7 · p.1 #7 · Using LR's Smart Previews for fast D800/E processing


playerofwar wrote:
Love it! Looks very usefull for less powerfull systems!
I do think the way you edit the SP's and then reallocate the raws, you will see a difference in editing/colours. It just can't be that a 2mb file delivers all the information of a 35mb file...
Did you check this out?

They are low resolution dng files, which are 8-bit log scaled files. You will only see a big difference when pushing low iso d800(or equivilent) files several stops.

It is perfectly fine for culling and all but the most precise editing.



Dec 30, 2013 at 03:29 AM
form
Offline
• • • •
Upload & Sell: Off
p.1 #8 · p.1 #8 · Using LR's Smart Previews for fast D800/E processing


If significant differences between results are found with d800 files - which I use specifically for the dynamic range and edit by boosting shadows in high dr scenes - then this is not a practical tool for the d800 files for me. Instead, it would be very useful for culling first and possibly working with the canon files that are less flexible and therefore less likely to look very different from the smart preview versions.


Dec 30, 2013 at 10:17 AM
snapsy
Online
• • • • • •
Upload & Sell: On
p.1 #9 · p.1 #9 · Using LR's Smart Previews for fast D800/E processing


form wrote:
If significant differences between results are found with d800 files - which I use specifically for the dynamic range and edit by boosting shadows in high dr scenes - then this is not a practical tool for the d800 files for me. Instead, it would be very useful for culling first and possibly working with the canon files that are less flexible and therefore less likely to look very different from the smart preview versions.


I push shadows extensively and the differences are relatively small and easily usable for editing. Here is an example of a D800 raw underexposed by 5 stops and pushed in LR5, exported to JPEG from both the original raw and the SmartPreview...pretty much a torture test.

Orig JPEG exported from full-size raw
+5EV JPEG exported from full-size raw
+5EV JPEG exported from Smart Preview



Dec 30, 2013 at 10:39 AM
Fred Miranda
Offline
Admin
Upload & Sell: On
p.1 #10 · p.1 #10 · Using LR's Smart Previews for fast D800/E processing


snapsy wrote:
I push shadows extensively and the differences are relatively small and easily usable for editing. Here is an example of a D800 raw underexposed by 5 stops and pushed in LR5, exported to JPEG from both the original raw and the SmartPreview...pretty much a torture test.

Orig JPEG exported from full-size raw
+5EV JPEG exported from full-size raw
+5EV JPEG exported from Smart Preview


Great sample. As a Canon shooter, this shadow recovery never ceases to amaze me. Do you see similar results with the A7R when compared to the D800E?



Dec 30, 2013 at 01:10 PM
DblDrgn
Offline
• •
Upload & Sell: Off
p.1 #11 · p.1 #11 · Using LR's Smart Previews for fast D800/E processing


I've been unplugging my RAWs and editing from smart previews for a couple of months now, it's much better than editing RAW files. I'm very pleased with performance in the Lightroom Develop module now, everything is very snappy and switching between images is a lot faster.


Dec 30, 2013 at 05:14 PM
S-Man23
Offline
• • •
Upload & Sell: Off
p.1 #12 · p.1 #12 · Using LR's Smart Previews for fast D800/E processing


1) I usually like to import 1:1 previews. Should I stop that if I'm doing smart previews?

2) The smart previews surely take up a lot of space on the OS drive. Can you delete them once you've exported final images?

Thanks for the useful info. I've been wanting to try using smart previews but have not taken the time to look into it much.



Dec 30, 2013 at 05:27 PM
awad
Offline
• •
Upload & Sell: Off
p.1 #13 · p.1 #13 · Using LR's Smart Previews for fast D800/E processing


S-Man23 wrote:
1) I usually like to import 1:1 previews. Should I stop that if I'm doing smart previews?

2) The smart previews surely take up a lot of space on the OS drive. Can you delete them once you've exported final images?

Thanks for the useful info. I've been wanting to try using smart previews but have not taken the time to look into it much.


i do 1:1 and smart previews. you can set LR to automatically delete previews after a certain time.



Dec 30, 2013 at 05:31 PM
awad
Offline
• •
Upload & Sell: Off
p.1 #14 · p.1 #14 · Using LR's Smart Previews for fast D800/E processing


snapsy wrote:
haha, you're right. That works too, thanks! I updated the OP with your idea.


don't get me wrong, im SUPER impressed with your work around. i dont even know what half that stuff meant.



Dec 30, 2013 at 05:32 PM
S-Man23
Offline
• • •
Upload & Sell: Off
p.1 #15 · p.1 #15 · Using LR's Smart Previews for fast D800/E processing


awad wrote:
i do 1:1 and smart previews. you can set LR to automatically delete previews after a certain time.


It deletes the smart previews or the 1:1 previews?



Dec 30, 2013 at 07:20 PM
snapsy
Online
• • • • • •
Upload & Sell: On
p.1 #16 · p.1 #16 · Using LR's Smart Previews for fast D800/E processing


Fred Miranda wrote:
Great sample. As a Canon shooter, this shadow recovery never ceases to amaze me. Do you see similar results with the A7R when compared to the D800E?


The A7r has similar performance although it has a bit more color cast in the shadows. Here's an A7r in the same scene pushed +5EV, with -3 shadow tint correction applied in LR5.

Sony A7r ISO 100 +5EV 36MP



Dec 30, 2013 at 10:45 PM
snapsy
Online
• • • • • •
Upload & Sell: On
p.1 #17 · p.1 #17 · Using LR's Smart Previews for fast D800/E processing


awad wrote:
don't get me wrong, im SUPER impressed with your work around. i dont even know what half that stuff meant.


Even a seemingly clever workaround is a bad one if it's more complicated than it needs to be. Yours is a much better solution.



Dec 30, 2013 at 10:48 PM
Scott Mosher
Offline
• • • •
Upload & Sell: Off
p.1 #18 · p.1 #18 · Using LR's Smart Previews for fast D800/E processing


I've been editing like this for a while. If you know an image will be a problem, I tag the image with a yellow color. Once I "reconnect" the images with the previews, I have LR only show me the yellow images and I edit those further.


Dec 31, 2013 at 12:03 AM
RustyBug
Offline
• • • • • • •
Upload & Sell: On
p.1 #19 · p.1 #19 · Using LR's Smart Previews for fast D800/E processing


tagrb


Jun 16, 2015 at 03:25 PM
Ziffl3
Offline
• • • • •
Upload & Sell: Off
p.1 #20 · p.1 #20 · Using LR's Smart Previews for fast D800/E processing


what?..... talk about old news....


Jun 16, 2015 at 04:12 PM
1
       2       end




FM Forums | Wedding Photographer | Join Upload & Sell

1
       2       end
    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.