Lee Saxon wrote:
If you'll look back you'll see that I specifically excluded real features such as IBIS or PDAF pixels. I'm talking about things like having an unusable grip, lossy raw compression, a battery they couldn't possibly think is big enough.
I don't have time to go searching back through everyone's post history so I was just responding to what was written in this thread.
There is always a slow rollout of certain features and the intentional obsolesce of older models. Often this is the case with video codecs and LOG tonecurves. However I don't think this was the case with the shutter/grip, which seemed to be in reaction to A7 criticisms, lossy RAW compression (unless they were playing a very long game since it's been around many years) or the battery - which is designed to be standard across the E-mount line, and so has to be sized for the smallest bodies.
GMPhotography wrote:
Being a optimist one can hope they may have addressed a few of these things without saying anything. If they say something that's a admission something is wrong. No OEM admits there wrong.
They don't have to admit they're wrong - they could just say "improves image quality with long exposures", similar statements have been made in the past. The fact that nothing is mentioned on this front strongly suggests this update will be for uncompressed RAW only, as stated in the announcement.
GMPhotography wrote:
Not a engineer but I do have a thought. To compress a Raw file that would take processing power to actual complete that task. If you shot a bigger file that is uncompressed than its just pushing the raw data without compressing it. I wonder and we are assuming that the uncompressed raw will tax the system let's say but what if the opposite is true, it's not taxing the system any harder because it does not have to go through the compression routine. Just thinking out loud.
The lossy compression will be done with dedicated parallel processing hardware and will thus be extremely fast, so omitting it ought to have a negligible effect on image processing speed or power consumption. It will however mean the buffer is effectively halved, and card write times doubled.
kwalsh wrote:
Yes the 11 bit nonlinear coding is the culprit but this would not follow a patch/stripe pattern for something like posterization of blue sky.
The algorithm is described as "11+7" meaning 11 bit non-linear companding and 7 bit delta encoding.
The first step, 11 bit companding, is *always* lossy. It is applied to every sensor value converting a 14 bit linear value from the sensor to a 11 bit non linear value. This in itself is not a bad idea if done "right" since we don't need so many levels in the highlights. The problem is Sony was too aggressive here, especially for low values, and in fact there is no portion of the curve that ever represents deltas as small as the original 14 bit linear data. Effectively the lowest values are only ever equivalent to 13 bit linear data.
The second step, 7 bit delta encoding, is only *sometimes* lossy. When there isn't high contrast in a stripe then the 7 bit delta encoding actually perfectly represents the "original" 11 bit companded values. It is only when there is high contrast that the 7 bit deltas are scaled and now the recovered 11 bit non linear values have additional quantization/rounding. For the case of a uniform blue sky the 7 bit delta encoding would be "lossless" during this step. For something like star trails with very high contrast this delta encoding step is very lossy.
Important to recall also that for something like a blue sky posterization is caused by the low valued channels - red and green. Detail, including "false detail" from posterization, comes from the least exposed channels. So a blue sky, or deep blue water, is going to be subject to posterization in the red and green raw data. That data for the Sony cRAW format is effectively only 13 bit data, not the 14 bit data you'd get from lossless compression like Nikon.
How much of a difference is 13 bit red channel vs 14 bit red channel in a blue sky really going to make in the end? Well, most of the time very little but in some cases - including digiloyd's notorious lake photo - the lossy RAW data itself clearly shows posterization and the number of red levels would in fact double if the compression had be lossless. So it certainly would help, going from 200 to 400 unique color values is not a trivial thing.
So again, for low contrast shots it is only the 11 bit companding step that is "lossy" while the 7 bit delta step is actually "lossless". That 11 bit step is only very slightly lossy, but it is always lossy. The 7 bit step is sometimes entirely lossless but in certain cases can be very lossy....Show more →
Thanks for the explanation Ken. I was under the impression that the 11 bit comression was done for the patch reference pixel only, and after the delta encoding. If implemented differently, the same encoding scheme could have avoided potential posterization of uniform areas. They could theorethically have encoded the delta values in 14 bits linear space and compensated for the error in the 11 bits companding when assigning the delta values.
alundeb wrote:
Thanks for the explanation Ken. I was under the impression that the 11 bit comression was done for the patch reference pixel only, and after the delta encoding.
Ah, I see what you were thinking. I haven't examined the DCRAW source code directly myself, but lexa's post on the Raw Digger site analyzing the cRAW format states that the 7 bit deltas are in the 11 bit non linear space rather than the original 14 bit linear space.
If implemented differently, the same encoding scheme could have avoided potential posterization of uniform areas. They could theorethically have encoded the delta values in 14 bits linear space and compensated for the error in the 11 bits companding when assigning the delta values.
Indeed, and actually in this theoretical case only the two max/min pixels per stripe would be subject to the 11 bit lossy companding. The remaining 14 pixels could express their true original values (in the case of very low contrast in the stripe). For higher contrast you would of course scale the deltas, and you would have to scale them "sooner" because they aren't companded but in the end I think the delta scaling would be much like the effective scaling that occurs in the companding. Hmmm....
Well, for whatever reason that's not the way they did it - they do the deltas on the 11 bit companded values if the Raw Digger post is to be believed.
kwalsh wrote:
I haven't examined the DCRAW source code directly myself, but lexa's post on the Raw Digger site analyzing the cRAW format states that the 7 bit deltas are in the 11 bit non linear space rather than the original 14 bit linear space.
I have examined the DCRAW source code directly, the detlas are indeed based on the 11 bit nonlinear values.
kwalsh wrote:
Indeed, and actually in this theoretical case only the two max/min pixels per stripe would be subject to the 11 bit lossy companding. The remaining 14 pixels could express their true original values (in the case of very low contrast in the stripe). For higher contrast you would of course scale the deltas, and you would have to scale them "sooner" because they aren't companded but in the end I think the delta scaling would be much like the effective scaling that occurs in the companding. Hmmm....
The delta values would have to be nonlinear, but essentially delta encoding the original 14-bit values would be possible, and would give greater precision for the same compression ratio...
...but I would wonder how often you get a variation of less than 127 ADUs within a 32-pixel wide area of the image. Not very often would be my guess. When I have a minute I'll grab a RAW file to actually check!
It will be really interesting to ABX-test the uncompressed format against the two older ones (gamma, gamma/lossy).
I know I can find situations where it will show a visible difference, the question is how far I have to push things before it shows. I'm willing to bet quite a lot of money that it's pretty far, and the risk of me losing is really small.
But I guess the option to have it (uncompressed, linear) is good. I do wish they keep an option for gamma-compression though. I'm used to large files, and photography is NOTHING compared to editing raw video - which can take up hundreds of gigabytes of intermediate storage to render rather short series of footage with grading and effects - but efficiency is more important to me.
Now, we're not limited by memory size or storage size. We're mainly limited by memory and storage throughput - bandwidth.
It's actually faster (and more energy/battery efficient!) to move a few thousand MP's worth of image data by compressing-moving-decompressing than by moving the full bulk of uncompressed data, since computation at the dispatch and endpoints is cheaper than bus bandwidth. At least as soon as you move outside the processor level 1-3 caches in a computer, and certainly in FPGA's and ASIC's. This means that editing is faster with compressed data, just as gaming video cards accelerate work by compressing memory streams as it sends it from the processors to the memory and back you gain maybe ten percent of performance without losing data fidelity.
Matt Grum wrote:
I have examined the DCRAW source code directly, the deltas are indeed based on the 11 bit nonlinear values.
The delta values would have to be nonlinear, but essentially delta encoding the original 14-bit values would be possible, and would give greater precision for the same compression ratio...
...but I would wonder how often you get a variation of less than 127 ADUs within a 32-pixel wide area of the image. Not very often would be my guess. When I have a minute I'll grab a RAW file to actually check!
What you really need to do is to run reverse MTF analysis, so you know the possible contrast ratios within that spatial frame.
For the other part (gamma-coding), you need to look at the minimum value-step available in that frame. One ADU delta is invaluable at blackpoint linear, but totally useless (more than one hundred times smaller than the photon shot noise at ISO100) in the highlights.
Given gamma encoding, the noise vs delta changes with level. The problem with the way Sony encodes lossily is not that the dynamic range within the frame is limited, but that the minimum available delta is bigger than the the minimum accurate information step in the darker parts of the strip. That gives posterization of individual levels - when you push the shadow beyond the slope of the gamma encoding.
That last sentence is important, since that was what the software engineers / signal processing engineers at Sony overlooked. That their sensors were getting so good that they contained usable detail information well below the step of their reduced-bit gamma coding.
In a Canon sensor, that would STILL not be the case. Read noise (at lower ISOs) and/or shot noise (at ISOs >400) would overwhelm the minimum level step in the shadows where this quantization/posterization problem is visible with this kind of encoding.
Yeah, if we consider the posterization effect from the 11 bit gamma they use as a "signal" that can be seen in images, the photon shot noise from the A7s pixels at 18% of saturation should be about ten times that "signal". It could be possible to see after extreme stretching, but at a signal/noise ratio of -20 dB I would think the noise dominates.
Anyone else see the potential ambiguity in the press release?
Either
...selectable Compressed (11+7) and Uncompressed (14-Bit) RAW image capture"
or
... selectable (Compressed and Uncompressed) 14-Bit RAW image capture"
Matt Grum wrote:
I have examined the DCRAW source code directly, the detlas are indeed based on the 11 bit nonlinear values.
Thanks!
The delta values would have to be nonlinear, but essentially delta encoding the original 14-bit values would be possible, and would give greater precision for the same compression ratio...
Well they wouldn't have to be nonlinear, but it might make sense to do so for more sensible compression performance. For example, the compression algorithm could be:
1. Find min and max 14-bit values in the stripe
2. Covert to 11-bit and store
3. For remaining pixels compute difference in 14-bit space between pixel value and the min value as translated back to 14-bit from the stored 11-bit min value. This step preserves the exact 14-bit value of these non min/max pixels despite the fact that the min and max pixels themselves did not preserve their own exact values.
4. Scale or otherwise compand this delta as necessary.
In step 4 there is no need for this delta to be stored with a non linear compander, it could be stored linearly using the simple power of 2 scaling as necessary. I'm not sure that is necessarily the best option, but it is one option. Or maybe that's what you meant by non-linear (i.e. scaling by factors of two as necessary).
...but I would wonder how often you get a variation of less than 127 ADUs within a 32-pixel wide area of the image. Not very often would be my guess. When I have a minute I'll grab a RAW file to actually check!
Well you would in the very lowest ADU counts - something like the red channel in a uniform sky. But of course in almost all other cases it would be quite a bit larger and you'd then need to scale or compand the deltas at which point you haven't gained anything from the different delta encoding approach.
In the end just a less aggressive gamma curve - one that preserved the LSB for the lowest ADU values instead of always throwing away the LSB - would be just fine in my book. At least for the posterization of skies.
AJSJones wrote:
Anyone else see the potential ambiguity in the press release?
Either
...selectable Compressed (11+7) and Uncompressed (14-Bit) RAW image capture"
or
... selectable (Compressed and Uncompressed) 14-Bit RAW image capture"
Yeah a little ambiguous. On the DPR announcement they mentioned that their understanding was the choices were the original 11+7 lossy compression or completely uncompressed 14 bit (not even losslessly compressed).
Mystik wrote:
There’s a stark difference between deliberate planned obsolescence and real technological improvements and developments that allow for better performance at a given price point over time.
Purposely holding back features is suicidal in competitive consumer electronics market segments, unless there’s some sort of collusion between the competitors, as innovation is where the value proposition lies in the consumer electronics space. Those who do not innovate in this space, die. If you, as a player in a consumer electronics space, deliberately do not put your best innovations into your current products, your competition will.
Holding back innovations only works at best, within a duopoloy, where two competitors indirectly collude to not innovate. Game theory comes into play here, as the competitive value of innovating will be immediately made null when the single other competitor reacts, thus eliminating any incentive to innovate and lengthening product life cycles. This is precisely the reason that within the duolopoly of Canon and Nikon, neither have made real efforts to go mirrorless until Sony disrupted the market and the market for prosumer cameras truly (in my mind) became a competitive one. Sony has disrupted this space by specifically deciding to innovate where Canon and Nikon did not, and now, none of them have any incentive to hold back innovations. Should be interesting times in the world of cameras.
Planned obsolescence does occur in the realm of competitive consumer electronics…but in an area that most of us don’t suspect: durability. You don’t really see this with prosumer cameras as these are specifically designed to be durable (because the demands of photographers require them to be), but you do see them most blatantly in smartphones. Glass screens surrounded by thinner bezels, non-replaceable batteries, proprietary screws, metal casings that warp when you pry them open, software updates that becomes increasingly more resource heavy etc etc . Smartphone makers have cleverly conditioned us to believe that its normal to replace $500+ cell phones every other year.
Holding back innovations might work though IF say Sony has such a technology lead that the competition will not be able to immediately respond. It seems we certainly see this with sensor technology and Sony. How many years has Sony's sensor tech shown a clear advantage over Canon?
Another point about planned obsolescence, Sony and cameras is that it's not just a matter of initial durability but, perhaps just as important, product support and parts availability down the road. CaNikon might say support their higher end models with parts/ service for 8 or even 10 years. How long does Sony support their models? If they (Sony) choose not to produce enough parts for expected service beyond a few years, that's also planned obsolescence.
alundeb wrote:
Yeah, if we consider the posterization effect from the 11 bit gamma they use as a "signal" that can be seen in images, the photon shot noise from the A7s pixels at 18% of saturation should be about ten times that "signal". It could be possible to see after extreme stretching, but at a signal/noise ratio of -20 dB I would think the noise dominates.
Blah, I did the math wrong. The photon shot noise is only 3 times the quantization step in 11 bit nonlinear right above 18% grey.
sebboh wrote:
i don't know about the a7r or a7rII, but my a7 and rx1 always have corrections turned off and i seem to have posterization issues much more frequently than on other cameras i've used. pretty much every time there is a large chunk of blue sky in the frame.
I noticed this too, especially on the RX1, even if the files weren't push super hard. And I do wonder if Sony will put out an rx1 follow up. Is ISO 64 possible through FW?
nehemiahphoto wrote:
I noticed this too, especially on the RX1, even if the files weren't push super hard. And I do wonder if Sony will put out an rx1 follow up. Is ISO 64 possible through FW?
Is this blue sky posterization visible only in JPEG? I have a bunch of big blue sky shots on the A7s, and I have not noticed it much even with pushing (JPEG shows minor banding).
Thanks, Fred for posting that. Now, the file size goes from 24MB to 48MB and that's for A7s II. How big will the uncompressed file size be for the A7r II ?
thanks ken, matt, and theSuede! that makes much more sense then the previous explanations i'd heard.
nehemiahphoto wrote:
I noticed this too, especially on the RX1, even if the files weren't push super hard. And I do wonder if Sony will put out an rx1 follow up. Is ISO 64 possible through FW?
hiepphotog wrote:
Is this blue sky posterization visible only in JPEG? I have a bunch of big blue sky shots on the A7s, and I have not noticed it much even with pushing (JPEG shows minor banding).
no, shows up in raw viewing in LR as well as direct export to tiff.
AGeoJO wrote:
Thanks, Fred for posting that. Now, the file size goes from 24MB to 48MB and that's for A7s II. How big will the uncompressed file size be for the A7r II ?
I expect to be close to double so 75-85MB depending on the image. It's nice to have the option as I would only use uncompressed to more serious work.
?