Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | Pro Digital Corner | Join Upload & Sell

  

Archive 2013 · .XMP file extension?

  
 
nugeny
Offline
• • • • •
Upload & Sell: On
p.1 #1 · p.1 #1 · .XMP file extension?


In some, few NEF files, I see, in addition to the NEF file, there is another file, same number but attached is .XMP, Example:
DSC5105
DSC5105.XMP
What is this .XMP file for? just to take up place in the harddisk?

Now, while shooting, I don't do any thing different.



Mar 02, 2013 at 12:56 PM
BruceF99
Offline
• •
Upload & Sell: Off
p.1 #2 · p.1 #2 · .XMP file extension?


http://www.adobe.com/products/xmp/

It is a sidecar file that typically contains edit information from Lightroom, Photoshop, etc.



Mar 02, 2013 at 01:13 PM
nugeny
Offline
• • • • •
Upload & Sell: On
p.1 #3 · p.1 #3 · .XMP file extension?


BruceF99 wrote:
http://www.adobe.com/products/xmp/

It is a sidecar file that typically contains edit information from Lightroom, Photoshop, etc.


I di understand that it contains the metadata. But why not every image? In fact, it is attached to the minority of my NEF file. I work exclusively with CS5 and soon will be cs6



Mar 02, 2013 at 03:54 PM
Z_man
Offline
• •
Upload & Sell: Off
p.1 #4 · p.1 #4 · .XMP file extension?


The .xmp file contains editing data from Photoshop and Lightroom. You only have .xmp files when you have edited a photo.


Mar 02, 2013 at 04:46 PM
nugeny
Offline
• • • • •
Upload & Sell: On
p.1 #5 · p.1 #5 · .XMP file extension?


Z_man wrote:
The .xmp file contains editing data from Photoshop and Lightroom. You only have .xmp files when you have edited a photo.


I think I got it now. I pick a raw image, PP and save my edited version in Tiff file and very rarely i look back at the original raw. So now i have 3 "original files" to fall back to: raw, tiff and XMP.Really safe.
Thanks.
Bob



Mar 02, 2013 at 05:00 PM
RustyBug
Offline
• • • • • • •
Upload & Sell: On
p.1 #6 · p.1 #6 · .XMP file extension?


As I understand it (hopefully correctly) ... the XMP file is NOT an image file, it is only a copy of the set of instructions (for raw converter to use) of what changes to make that you want made to the corresponding raw image file (i.e. based on your LR/PS editing of the raw file). Thus you really only have 2 "original files" the raw (now with a set of sidecar instructions contained in the corresponding .xmp file) and the tiff file is now a "master file" that you have produced.

I do have a question ...
Why does a tiff file GROW to such a large size compared to the raw?
And what is the primary distinction/advantage for a tiff file vs. a raw that would warrant working with such a large file?

For instance, my raw is 10MB, the .xmp is 6KB and the tiff is 180MB. The file is one image layer, with one B&W layer and a mask (i.e. small potatoes), yet there is a rather significant difference between 10MB + 6KB vs. 180MB.



Mar 02, 2013 at 07:56 PM
Eyeball
Offline
• • • •
Upload & Sell: Off
p.1 #7 · p.1 #7 · .XMP file extension?


RustyBug wrote:
As I understand it (hopefully correctly) ... the XMP file is NOT an image file, it is only a copy of the set of instructions (for raw converter to use) of what changes to make that you want made to the corresponding raw image file (i.e. based on your LR/PS editing of the raw file).


That is correct. The XMP is actually a text file that documents the "edits" that were produced by the raw processor so that they can be reproduced the next time you open that RAW file in the raw development software. It is a text file and you can look at the contents if you like in just about any type of text editor, including MS Word.

The use of XMP files is not limited to just photography. It is a general standard in the IT industry for transmitting meta data (data about data). It is used in many different computer applications. The content of the XMP depends on the application that uses it. Here is more info:

http://en.wikipedia.org/wiki/Extensible_Metadata_Platform

When using a raw developer (Lightroom, for example), the software will often give you the alternatives of storing the parametric edits in a central database or as the individual XMPs or both.

RustyBug wrote:
Why does a tiff file GROW to such a large size compared to the raw?


To understand this, it's helpful to run the numbers. Let's start with the TIF file since that's the easiest to explain.

For a 16-bit, uncompressed TIF with only one image layer, you can calculate the file size in bytes this way:

Vertical dimension in pixels * Horizontal dimension in pixels * 3 color channels * 2 bytes

For my Canon 5D2, this would be 3744*5616*3*2 = 126,157,824 bytes.
In computer science "kilo", "mega", and so on use a factor of 1024 (2 to the 10th power) rather than 1000 so 126,157,824 bytes would be 123,201 kB or 120.31MB.

The actual size will probably be slightly higher for additional overhead like an embedded ICC color profile. It will be much bigger if there are additional layers. It will get smaller if one of the TIF lossless compressions options are used when the file is written.

Now for a RAW file. Most camera manufacturers use some form of lossless compression (and I believe Nikon even has a lossy alternative) for their RAW files. To start, however, we can do a rough calculation for an uncompressed raw file:

Vertical dimension in pixels * Horizontal dimension in pixels * 2 bytes * # bits recorded / 16 bits

For my Canon 5D2, this would be 3744*5616*2*14/16 = 36796032 bytes = 35933.6 kB = 35.1 MB.

Note that there is no multiplication by 3 color channels. That is because with a bayer sensor, there is only one photosensor for each pixel, whether it be a red, green, or blue sensitive photosensor. Later during raw development, the values from these photosensors are interpolated to produce RGB values for each photosensor site.

Taking a look at a few of my 5D2 raw files, they range in size from about 22,500 kB to about 29,500 kB, which sounds about right taking into account the lossless compression that Canon is applying when the file is written. Roughly speaking, lossless compression looks for repeating patterns in the data and uses "shorthand" to represent those repeating patterns in the stored file. The more random the data is, the less the file can be compressed. If there is something in the file that causes strings of repeated values (like many photosensor values clipped to full black or full white) that will allow for more compression in the stored file.

Hope that helps. If anyone catches any errors in what I wrote, please let me know.

RustyBug wrote:
And what is the primary distinction/advantage for a tiff file vs. a raw that would warrant working with such a large file?


There is currently no software that I know of that allows true, pixel-level editing of a raw file. In other words, it is not possible for you to edit and save the edited file in the raw format. Raw development software instead saves the edits as parametric metadata apart from the raw file, either in a central database or in an additional file (like the XMP previously mentioned).

When you need to use editing software that uses rendered RGB files (like Photoshop), RAW is no longer an option and a lossless file format like TIF is usually desired if the user wants to preserve the maximum amount of image information (and even TIF may not preserve all of the information contained in the original RAW).

You normally use editing software that works on rendered files because it provides features that current raw developers don't provide. That may include things like layers and sophisticated cloning/healing.

The gap between capabilities of raw developers (like Lightroom) and rendered editors (like Photoshop) has been slowly closing but I think it will be a while yet before we see all of the functionality of something like PS put into LR. One of the limiting factors is the quantity of parametric edits that begins to be generated by the raw developer and the performance of storing, reading, and changing that information during the editing process. Storing an exposure or white balance setting is pretty simple and doesn't take up much space, but when you start pixel-level edits like cloning, the quantity of generated instructions increases significantly. The mouse location and movement needs to be constantly recorded along with the brush size, feathering, opacity, and so on. Do much work with cloning, healing, and gradients in LR and you will see your XMP file increase significantly and those functions in LR are currently very limited compared to their counterparts in PS.



Edited on Mar 03, 2013 at 11:08 AM · View previous versions



Mar 03, 2013 at 08:51 AM
RustyBug
Offline
• • • • • • •
Upload & Sell: On
p.1 #8 · p.1 #8 · .XMP file extension?


Eyeball wrote:
For a 16-bit, uncompressed TIF with only one image layer, you can calculate the file size in bytes this way:

Vertical dimension in pixels * Horizontal dimension in pixels * 3 color channels * 2 bytes

For my Canon 5D2, this would be 3744*5616*3*2 = 126,157,824 bytes.

Vertical dimension in pixels * Horizontal dimension in pixels * 2 bytes * # bits recorded / 16 bits

For my Canon 5D2, this would be 3744*5616*2* 14*16 = 36796032 bytes = 35933.6 kB = 35.1 MB.


Dennis ... way cool. You know me, always diggin' the numbers.

Looks like your formula is not the same as your description ... i.e. double check the # bits recorded / 16 bits vs. "14*16". Should that be (14/16) or in the case of an 8bit camera (8/16)?

So, it sounds like the raw, while the name might suggest original really isn't the "complete" original data. It sounds like tiff is as "complete data" as is possible. So, does the tiff file data actually exist in the capture, or is the tiff file a "reverse engineer" / "reverse calculation" of the data previously consolidated from the lossless raw to produce a "complete data" tiff file?

Also, I tested the formula against my numbers and it didn't quite come up the same numbers as my tiff file size indicates @ 180MB. So I "tweaked" on the formula by changing the operator from multiply by two, to raise to the power of 2 and the my numbers came in line with 180MB for my tiff file. Whereas using the operator of multiplication is 13.5 (14MP camera) vs. 81. Is there a difference between the camera vs. PS @ how it makes such calculations?

(3000 x 4500) ^2 x (8/16)
(3000 x 4500) ^2 x 3

In either case, the dramatic size growth difference for me would be explained by formula's factor of 6 (.5 vs 3) ... in addition to the amount of "unconsolidation" from raw to tiff.
For a 14 bit camera, the "growth" factor would be 3.4 (.875 vs. 3).

Hmmm, so is the calculation for determining the number of PIXELS using the multiplication operator, while determining the number of BYTES (not to be confused with bits @ factor of 8x) uses the exponential operator.

Thanks ... this helps.

Okay ... so, my camera captures 14MP, the raw file consolidates SOME of this data and the file size varies. Tiff "unconsolidates" the data and the file size will be larger. Changing from an 8 bit to a 16 bit is an ^2 @ bytes vs. bits.

I think I've got it now ... maybe a bit crudely, but I think I'm good.

Thanks.

Edited on Mar 03, 2013 at 12:01 PM · View previous versions



Mar 03, 2013 at 10:53 AM
Eyeball
Offline
• • • •
Upload & Sell: Off
p.1 #9 · p.1 #9 · .XMP file extension?


RustyBug wrote:
Looks like your formula is not the same as your description ... i.e. double check the # bits recorded / 16 bits vs. "14*16". Should that be (14/16) or in the case of an 8bit camera (8/16)?


I figured I would make a mistake somewhere. You are correct. It was a typo. I have corrected it.

And yes, if it was an 8-bit camera you could either use the 8/16 adjustment or just eliminate the byte multiplier and adjustment factor since 8 bits = 1 byte. A 12-bit camera would use an adjustment factor of 12/16.

This adjustment factor is an assumption on my part. Much of the raw format info is proprietary. I am just assuming that the manufacturer will eliminate the unnecessary bits in a full byte as part of the compression process.

RustyBug wrote:
So, it sounds like the raw, while the name might suggest original really isn't the "complete" original data. It sounds like tiff is as "complete data" as is possible. So, does the tiff file data actually exist in the capture, or is the tiff file a "reverse engineer" / "reverse calculation" of the data previously consolidated from the lossless raw to produce a "complete data" tiff file?


You can kind of think of the process like this:

A little data (the raw photosensor data) + known characteristics of the sensor and camera = a lot of data (the 16-bit RGB file).

A simple, imperfect analogy would be how you can take a zip code of 92688 and, using a zip code table, get "Rancho Santa Margarita, California". In the case of a raw file though you are using mainly formulas rather than a look-up table.

Even with the increased amount of data that is generated by this "data + formula" process, the 16-bit TIF file is still a larger "container", if you will, for two reasons:
- It probably has more bits than the camera's raw file (usually 12 or 14 these days).
- The TIF file represents 3 channels of full RGB color data per pixel compared to the raw file, which has just one true color channel of data plus some additional information added by the interpolation of neighboring photo sensors.

The TIF file is never going to contain more information than the raw file (unless you add stuff to it later). A 16-bit TIF file is a larger "container" but just like pouring a small glass of water into a larger glass, you still have the same amount of water.

In this case of raw development though, you may actually end up with less "water", even though 14-bit raw data would apparently have no problem fitting into the TIF 16-bit container. The reason is a gamma adjustment is usually made during the raw development process, causing the values to "bunch up" on the high end. If this "bunching up" is severe enough, tonal values can be lost to the precision limits of the 16 bits. This is why that if you need to make big exposure adjustments (particularly those impacting highlights) it is usually a good idea to do that in the raw developer rather than later in a RGB editor like Photoshop.

If you really, really needed a TIF for some reason that retained the maximum amount of information from a raw file, the way to do that would be to use a linear gamma during the conversion process. That would take some twiddling to do in Lightroom but in Canon's DPP software, there is a checkbox to do that.

So in summary and to try to answer your questions more directly:
- The TIF file will never be more "complete" than the raw file in terms of the real tonal information it contains and it may actually be somewhat less "complete" due to the gamma adjustment.
- I think you can consider the TIF file to be more of an "engineering" of the raw data rather than the "reverse engineering". "Reverse engineering" to me would be to take the TIF and try to derive the formulas to get back to the raw data. That is actually similar to what companies like Adobe have to do, since the real formulas and processes are usually kept a secret by the camera manufacturers.



Mar 03, 2013 at 11:58 AM
Eyeball
Offline
• • • •
Upload & Sell: Off
p.1 #10 · p.1 #10 · .XMP file extension?


RustyBug wrote:
Also, I tested the formula against my numbers and it didn't quite come up the same numbers as my tiff file size indicates @ 180MB.


That surprises me. What camera is it?
The calculation of the TIF file size in bytes should always be pretty close to the formula (Vpixels * Hpixels * 3 * 2) assuming it is uncompressed, 16 bits, and with only the one image layer.

The TIF size calculation is independent of the camera except for the vertical and horizontal image dimensions in pixels.

The 8/12/14 bit adjustment factor only comes into play when estimating the raw file size and as I said earlier, I'm making some assumptions when using that adjustment factor.

Edited on Mar 03, 2013 at 12:10 PM · View previous versions



Mar 03, 2013 at 12:07 PM
RustyBug
Offline
• • • • • • •
Upload & Sell: On
p.1 #11 · p.1 #11 · .XMP file extension?


Cool ... pretty heavy stuff for a Sunday morning. A lot to digest and retain and much that I may later "data dump" into practicality, but it certainly helps take out the mystique of what you hear from differing opinions.

Anyway ... sorry for the thread hijack, but hopefully the OP (et al) finds it helpful as well in his understanding of raw, .xmp, tiff.

Good stuff, thanks again.

Camera ... Kodak SLR/c.

Hmmm, just opened and saved a couple files from raw to tiff .... 13.7MB to 77.2 MB.
Not sure what might have happened to cause that 180MB file. In that case, your multiplication would be correct formula over my "tweaking" to generate a formula for the 180MB. I'll have to watch closer if that occurs again.

Strange, I open the 180MB tiff file ... it shows 77.2M in PS ... is that odd.

Then when I save as tiff ... 180MB.
If I select "discard layers", it saves @ 77.2MB. Then if I save as again without checking "discard layers", it saves @ 180MB again.





Edited on Mar 03, 2013 at 12:42 PM · View previous versions



Mar 03, 2013 at 12:10 PM
nugeny
Offline
• • • • •
Upload & Sell: On
p.1 #12 · p.1 #12 · .XMP file extension?


RustyBug wrote:
Cool ... pretty heavy stuff for a Sunday morning. A lot to digest and retain and much that I may later "data dump" into practicality, but it certainly helps take out the mystique of what you hear from differing opinions.

Anyway ... sorry for the thread hijack, but hopefully the OP (et al) finds it helpful as well in his understanding of raw, .xmp, tiff.

Good stuff, thanks again.

Camera ... Kodak SLR/c.

Hmmm, just opened and saved a couple files from raw to tiff .... 13.7MB to 77.2 MB.
Not sure what might have happened to cause that 180MB file. In that
...Show more

that is still 6x the file size. I wonder if it would make anyn difference if I save it in PDS file?



Mar 03, 2013 at 12:41 PM
RustyBug
Offline
• • • • • • •
Upload & Sell: On
p.1 #13 · p.1 #13 · .XMP file extension?


+1 @ 6x


Mar 03, 2013 at 12:45 PM
Eyeball
Offline
• • • •
Upload & Sell: Off
p.1 #14 · p.1 #14 · .XMP file extension?


Yep, 77.2 MB should be the number.

Things that could cause additional growth in 16-bit TIF file size:
- ICC profile
- Additional layers
- Additional channels (alpha channel for transparency, for example)
- Notes
- Spot colors



Mar 03, 2013 at 12:45 PM
RustyBug
Offline
• • • • • • •
Upload & Sell: On
p.1 #15 · p.1 #15 · .XMP file extension?


Okay, so I deleted the B&W layer (with mask) from the 180MB file and then saved as tiff.

It saved @ 77.2MB.
Repeated the B&W adjustment layer ... back to 180MB.
Same with a Brightness/Contrast adjustment layer ... 180MB

How in the world does a simple adjustment layer cause a file to grow from 77.2 to 180MB?

Went back and added three more adjustment layers (color, bright, sat) and the file save grew to 257MB.
I thought adjustment layers were supposed to be very "data efficient" ... why is tiff being effected so strongly by the adjustment layers?




Edited on Mar 03, 2013 at 01:37 PM · View previous versions



Mar 03, 2013 at 12:51 PM
Eyeball
Offline
• • • •
Upload & Sell: Off
p.1 #16 · p.1 #16 · .XMP file extension?


nugeny wrote:
that is still 6x the file size. I wonder if it would make anyn difference if I save it in PDS file?


A PSD file may be slightly smaller due to differences in the lossless compression used but I doubt if you will see a significant improvement. It could even be worse if you use "maximize compatibility mode" since that adds a flattened layer to the file.

There just isn't much you can do if you need 16 bits without lossy compression. Lossless compression can only squeeze so much. Beyond that you need to reduce bit-depth or use lossy compression like Jpeg.

For me it's:
- Keep my original RAWs
- Keep layered TIFs during PS editing and during a reasonable period where I might want to come back to the edited TIF. Once that period is over, I delete the TIF. Worse comes to worse, I can always start over from the RAW.

If I was in the business (like a wedding photographer), I might consider saving a Jpeg version of my final, edited TIF before deleting the TIF so I had an archival copy of exactly what went out to the client.



Mar 03, 2013 at 12:58 PM
Mr Joe
Offline
• • • •
Upload & Sell: Off
p.1 #17 · p.1 #17 · .XMP file extension?


@Eyeball - I always wondered about the math behind these file size differences. Makes perfect sense. Thank you!


Mar 03, 2013 at 09:08 PM
Imagemaster
Offline
• • • • • • •
Upload & Sell: On
p.1 #18 · p.1 #18 · .XMP file extension?


If like me, you have no use for .xmp files and just find them annoying:

Open a RAW image in Camera Raw.

In the Tool Bar, click on the Open Preferences Dialog icon.

In the resulting window, go down to DNG File Handling and put a check mark in the box for Ignore sidecar ".XMP" files.



Apr 12, 2013 at 10:43 PM
nolaguy
Offline
• • • •
Upload & Sell: Off
p.1 #19 · p.1 #19 · .XMP file extension?


*opening Excel to try to map and understand this stuff*

Dennis, Rusty, Robert, thanks so much for this tag-team effort. You gave me a much appreciated headache.




Apr 20, 2013 at 03:07 PM





FM Forums | Pro Digital Corner | Join Upload & Sell

    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.