It is making me consider converting the uncompressed ARW to a lossless compressed DNG....
-Tim
Fred Miranda wrote:
I expect to be close to double so 75-85MB depending on the image. It's nice to have the option as I would only use uncompressed to more serious work.
I hadn't seen the effect so non-subtle before, this changes my feelings on the subject. I have seen this type of thing before with my pictures but (from what I could tell) it was only on JPGs.
Oh jeez - I see it now in my photos from last weekend, even in raw in LR. Well, so much for that. I hope there's an update for the base A7.
Here's another example - it's worse here in JPG than it is in raw, but I am guessing that is because the JPG nonlinearities are just exacerbating the effect.
If this posterization comes from 11 bits gamma, how would it be possible at all to view jpg images that are with 8 bits gamma without 3 bits / 8 times worse posterizaion? The gamma curves don't differ that much as to more than neglect an 8 times larger effect?
The posterization here also occurs in areas that are so dark that the raw values have to come from the lowest part of the compression curve, where the tonal resolution of 11 bits gamma compressed equals 13 bits linear.
p.10 #10 · Uncompressed 14 bit for A7sii and A7rii
alundeb wrote:
If this posterization comes from 11 bits gamma, how would it be possible at all to view jpg images that are with 8 bits gamma without 3 bits / 8 times worse posterizaion? The gamma curves don't differ that much as to more than neglect an 8 times larger effect?
The posterization here also occurs in areas that are so dark that the raw values have to come from the lower parts of the compression curve, where the tonal resolution of 11 bits gamma compressed equals 12-13 bits linear.
There is effectively a really strong tone curve between raw data and a processed JPG. Editing JPG can produce obvious posterization but processing RAW and outputting JPG there really shouldn't be any posterization introduced by the JPG.
Worth noting it is also possible to provoke posterization even from "good" RAW data in something like a polarized sky through uncontrolled color transforms causing gamut or channel clipping along the way. So just because there is posterization in the processed output does not necessarily mean the RAW data itself was the source of the posterization.
p.10 #11 · Uncompressed 14 bit for A7sii and A7rii
kwalsh wrote:
There is effectively a really strong tone curve between raw data and a processed JPG. Editing JPG can produce obvious posterization but processing RAW and outputting JPG there really shouldn't be any posterization introduced by the JPG.
Worth noting it is also possible to provoke posterization even from "good" RAW data in something like a polarized sky through uncontrolled color transforms causing gamut or channel clipping along the way. So just because there is posterization in the processed output does not necessarily mean the RAW data itself was the source of the posterization.
So in sebboh's example, the posterization in the dark blues is a result of the raw data being limited to 13 bits linear?
That would be devastating for all 12 bits raw files out there....
p.10 #12 · Uncompressed 14 bit for A7sii and A7rii
tsdevine wrote:
It is making me consider converting the uncompressed ARW to a lossless compressed DNG....
-Tim
Exactly what I plan to do. On top of that you can make a backup of the original raw files on another drive and compressed those with an ordinary zip compression tool.
p.10 #13 · Uncompressed 14 bit for A7sii and A7rii
I'm in a bit of confusion here. In earlier discussions regarding compressed raw and posterization, there seemed to be a consensus among the experts that posterization would be difficult to see even in stretched images. Now sebboh comes with an unprocessed image that shows posterization in the part of the compressed tone curve that has best tonal resolution, and suddenly the compression is responsible for this posterization?
Another suggestion: There has been talk about noise reduction i raw files applied to the blue channel in Sony cameras. If that is the case, it would make a plausible explanation for Sony files to be more prone to posterization in blue skies.
p.10 #14 · Uncompressed 14 bit for A7sii and A7rii
The only way to tell if there is posterization in the RAW file is to look at the numerical values in RAW digger or some such.
You can't see RAW file; you can only see a rendering of it, and channel clipping or out of gamut info and other things in between the RAW file and the screen or print make the image posterized..
p.10 #15 · Uncompressed 14 bit for A7sii and A7rii
alundeb wrote:
So in sebboh's example, the posterization in the dark blues is a result of the raw data being limited to 13 bits linear?
That would be devastating for all 12 bits raw files out there....
that was always my question, i never saw such issues with 12bit raws.
is there evidence that sony does cooked in noise reduction to the blue channel from the raw files? i've never seen the effect in anything other than blue, maybe i should find a red or green wall to try to replicate this on...
p.10 #16 · Uncompressed 14 bit for A7sii and A7rii
sebboh wrote:
that was always my question, i never saw such issues with 12bit raws.
is there evidence that sony does cooked in noise reduction to the blue channel from the raw files? i've never seen the effect in anything other than blue, maybe i should find a red or green wall to try to replicate this on...
If you do, I think it would be a good idea to have the target out of focus.
Noise reduction combined with reduced bith depth / compression would give a double effect, as noise dithering masks the compression.
In your example, the posterization is alternate clean and noisy bands. They are likely clean around a stable quantized value, and noisy around the transition between two quantization steps. More noise would have the effect that we wouldn't see the clean bands, and wouldn't see this as posterization.
p.10 #17 · Uncompressed 14 bit for A7sii and A7rii
tsdevine wrote:
It is making me consider converting the uncompressed ARW to a lossless compressed DNG....
That may be a good idea anyway. Although .ARW look like TIFF files and Sony probably meant them to be (all of .CR2, .NEF and .DNG are TIFF) they are not quite right. Obviously Adobe and other raw readers can handle them (as does my software now!) but the TIFF standard is quite explicit that items appear in the numerical order of their tag values, and some are out of order in the .ARWs from my RX1.
That, and maybe somewhere else they've been sloppy that I didn't notice, could mean that those files will not be readable by some hypothetical program someone writes in the future based on standards. In other words, .ARW is not an archive format. Canon's .CR2 at least stands a fighting chance because there are so many of them and because, so far, they appear to be valid TIFF.
Another point: if .DNG becomes the only way most people store Sony raws, it might get rid of the assumption that's out there that everyone will keep actual camera raws. At the moment I can't bring myself to delete the files that come directly out of the camera in case two moons should rise one day, I get short-listed for some competition, and the judges want to see a .CR2 or .ARW.
p.10 #19 · Uncompressed 14 bit for A7sii and A7rii
tsdevine wrote:
It is making me consider converting the uncompressed ARW to a lossless compressed DNG....
Fred Miranda wrote:
That will probably be a good workflow for our new uncompressed RAW files in order to save space. DNG is a better format anyways.
Compression aside, I'm not convinced DNG it's a better format at all - you lose any manufacturer specific metadata, and I simply don't buy the idea that it's a safer format for archival because it is "open" - the source code for DCRAW contains a complete description of how to interpret Sony ARW files, so they are effectively just as open as DNGs.
I would store the original RAWs on a compressed drive, or zip them and store JPEG previews alongside.
DavidBM wrote:
You can't see RAW file; you can only see a rendering of it, and channel clipping or out of gamut info and other things in between the RAW file and the screen or print make the image posterized..
You can open up each channel as a greyscale image and examine them as if they are images, that's how I would go about looking for posterisation in a RAW file.
sebboh wrote:
is there evidence that sony does cooked in noise reduction to the blue channel from the raw files?
I haven't seen any - it would be very easy to detect this by looking at a Fourier transform.
Matt Grum wrote:
Compression aside, I'm not convinced DNG it's a better format at all - you lose any manufacturer specific metadata, and I simply don't buy the idea that it's a safer format for archival because it is "open" - the source code for DCRAW contains a complete description of how to interpret Sony ARW files, so they are effectively just as open as DNGs.
I would store the original RAWs on a compressed drive, or zip them and store JPEG previews alongside.
You can open up each channel as a greyscale image and examine them as if they are images, that's how I would go about looking for posterisation in a RAW file.
I haven't seen any - it would be very easy to detect this by looking at a Fourier transform.
