I think we are making two assumptions here.
(1) RAW data is the sensor output as is.
(2) The sensor data is linear ( meaning 2x exposure will double the data value )
Most of the quantitative analysis are based on the assumptions. But I am presenting here that one or both of them may not be true.
Rico and I have access to the most primitive raw data via different methods. Rico uses dcdraw while I use dng tools. We see the same thing. There is offset in low range. And the dark-frame data histogram shows weird missing data points ! For 1Ds2, the value of 127 all the time. For 1D3, there seem to be multiple ones. ( I don't have 1D3 DF data. I observed Rico's posting ) That suggests me that the data may not the raw sensor data as is. ( looks like, artificially manipulated )
I did the simple experiment. I took the shots of the white card board at the incremental stops ( 1/250, 1/125, 1/60, 1/30 ). The corresponding raw data is expected to show the linear correlation with the exposures. As the data is before the Bayer interpolation, you see basically two peaks - Green at the higher position and Red and Blue merged together.
It seems that somehow the raw data is "processed" before saved. And Reverse-processing needs to be applied to restore the original raw data by the RAW converter programs. Why ?!?!
And my conclusion is that the Linearized data should be used for the analysis.
Maybe all this is related to their trying to keep the proprietary nature of their data intact? Making it difficult for you all to figure out would equally make it difficult for others who wish to make money off their success in doing so. Who knows...
I would say that every manufacturer knows everything about each others processes. The so called proprietary processes are only to BS the general public. If you have the money to spend the disection of a cameras hardware and firmware is a relative easy thing to do. Even the composition of the sensor is open to analysis. The curcuitry is duck soup. A camera or three, and three to six months salary for a good engineer should do it. None of us have the resources to expend on the reverse engineering but you can bet that Nikon and Sony and probably the other manufactures do and have. A good digital analyser piece of hardware runs about $40,000 and even if a manufacture develops its own machine code, that too can be cracked with the analyser. The only thing that a manufacture can really do is keep a jump on the release of the product knowing that the compitition than has to play catch up. Their secrets are very short lived. Been there, done that. never liked it.
Tom_W wrote:
...
Can this data be translated into something (like stops) that us photobrained drones can understand?
Since the task is photon counting, "bits" and "f-stops" operate on the same scale. With its greater DR, the 1D3 can extend the highlights over the 1Ds2 by approx one stop, or extend shadow definition by one stop. The default split of additional DR is up to Canon, and how they place middle gray. An indirect benefit of greater DR is extending color balance into the lower Kelvins without having the blue channel collapse into the noise.
Jeff wrote:
Maybe all this is related to their trying to keep the proprietary nature of their data intact? Making it difficult for you all to figure out would equally make it difficult for others who wish to make money off their success in doing so. Who knows...
There must be other reasons than just protecting the stuff. Protection doesn't seem to be the intent. The linearization table is official tag of public DNG spec.
Pondria, would you please explain the linarisation to me. If you have already explained it then I must have missed it (it's a l-o-n-g thread). Also, can you explain the "linearity" values in your table above - I don't see the relationship between them and any of the other data values in the tables.
Have I correctly understood that the linearisation is now thought to be a fairly minor thing once we take into account the fixed offset at the bottom end of the scale for the raw sensor data, whereas initially it looked like it had major significance ?
In your post on page 21 you displayed a table of sensor data in half-stop bins, and you subsequently realised that there should have been an offset applied to this data. Have you now got a corrected table that shows the data spread after the offset is removed, and can you present it here for us ?
Does linearisation apply after the offset is taken into account, or is that part of the linearisation process ?
Rico, your charts show some gaps in the data values for some of the different sensors. I vaguely recall something like that being published in the past and that it had something to do with the process used to adjust the effective ISO rating in-camera. Do you know if the image files you analysed were all shot at the basic ISO 100 ?
Alan321 wrote:
Pondria, would you please explain the linarisation to me.
(1) They say sensors are pretty much linear except the low range.
(2) And In our Canon sensor data, there are non-negligible offsets. For instance it is about 100 for 1Ds2. In high range values 100 is negligible. In low range, 100 is very significant. We are taking about log scale.
Let me give you simplified example.
You take the shots of the white blank wall with the Exposure -4, -3, -2, -1, 0, 1, 2.
You expect that the the raw data will give you something like,
16, 32, 64, 128, 256 ... ( this is linear )
BUT, Canon raw data give you
116, 132, 164, 228, 356, ...
Linearization is the process to take out the offset and rescale.
Alan321 wrote:
Rico, your charts show some gaps in the data values for some of the different sensors. I vaguely recall something like that being published in the past and that it had something to do with the process used to adjust the effective ISO rating in-camera. Do you know if the image files you analysed were all shot at the basic ISO 100 ?
Rico asked us for ISO100 files.
By the way, I thought Alan was on something wit ISO. I quickly check the dark frame for ISO 100, 200, 400, 800 and 1600.
Interesting. The histograms show the following
1. The deep valley @ 127 is consistent at all ISOs.
2. The the position and width of the peak measure at 50% peak height is the same for all ISO, 127 +/- 3 or 4.
3. The peak height is getting reduced as the values are leaking about the the right hand side due to increased noise as the ISO gets higher.
Pondria, I have to jump into this very interesting discussion. The reason is, I don't see any evidence here that the RAW data is not linear. The intensity values are all offset by the BlackLevel, which should be included in the RAW file as a tag. Obviously it is 127 for the 1DIII. Subtracting an offset value doesn't destroy the principle of linearity. It just moves the line From your cardboard exposure experiment, the values corrected for BlackLevel will be:
R-B: 74, 146, 290, 582
G: 127, 255, 510, 1044
That's still pretty linear, with the exception of the highest Green value.
Hope this helps, please forgive if it doesn't
Alundeb, you are right that the raw data seems to become linear by correcting the offset. But it leaves us with two issues.
1. The RAW data doesn't seem to be "as-is" given by the sensor. It requires certain process to restore back what the sensor generates.
2. Quantitative analysis ( like noise level or DR ) should be done after the data is properly processed.
Good.
The output from the sensor is processed by the CCD imaging analog front end to:
1) Remove the DC offset used to charge the cell
2) Remove the thermal noise floor and channel offsets by applying the signal from some optically black cells, in a feedback loop.
3) Adjust the ISO sensitivity by varying the gain of a programmable gain amplifier.
Now we have a linear signal that we for simplicity can call the output from the sensor.
It has been explained before why the offset is applied (because the electrical noise may introduce a small negative component into a value that should be positive or zero). Instead of adding a sign bit, they use the offset method, and all RAW converters are aware of that, or, maybe not always?
Now, you have shown us we have a problem with the avaliable software like DPP 3.0, that seems to handle the lower range incorrectly. I belive that is a problem with the offset correction in the software (the 14 bit format is new), and not that there is some other hidden nonlinearity in the RAW data.
With your excellent tools, you can prove what dynamic range there actually is by extending your cardboard test with some more stops in both ends. If your tools also could modify the RAW files, it would be extremely interesting to see what happened in DPP with the offset subtracted. But I know, that is too much to ask. But do you follow the idea?
There must be other reasons than just protecting the stuff. Protection doesn't seem to be the intent. The linearization table is official tag of public DNG spec.
Might another reason be to cover up a guilty secret ?
For example, because DPP offers the option of conversion to a 16-but .tif file I assumed that DPP was capable of 16-bit processing. Its not. DPP's 16-bit .tiffs are "fakes" in the sense that they contain only 8-bit data scattered about a 16-bit container whilst other raw converters [I've tried ACR, UFRaw and RawThreapee] appear to produce genuine 16-bit files.
[Comment derived from checking the 16-bit histograms of the .tiffs with ImageJ and its 16-bit histogram plugin].
The notch at 127 appears to be a delibert notch filter, the question is, is it the same in all cameras or does it move around due to the tolerance of the components used? OP amps are notoriously bad about offset drift and usually have individual adjustment pots in critical applications. If the technician doing the final test does not adjust it properly or if the design does not allow him to adjust it you wind up with an offset. Also is the placement correct, or is there an error in the filter design or is it there for stripping out some type of internal interference. (banding?) The notch may also be an error in the offset correction curcuitry which usually is designed to drive the OP anp to zero DC offset, The only way to know this is sampling the output from several cameras and see if the 127 notch moves. There are so many unknowns that we are basically speculating on what the design engineer had in mind. The quality of the engineer and the time and resources he had allocatted to him, Even if we find errors, the odds of Canon fixing minor errors given their history is very small, though hopefully they would be corrected in subsequent production and the next model.
Wanted to add that the 1DIIIs apparent focusing problem with temperture could be due to the same OP amp drift in the focusing curcuit. Component temperture coificient tolerance could be due to any one of the components being out of spec.
A/D converters have a certain nonlinearity in the relationship between input voltage and output value. It can even be non-monotonic in critical areas, dependent of the method of A/D conversion. In that case, we have the phenomenon called "missing values".
One example: If output values of 126, 127 and 128 correspond to input signal amplitudes of 79, 80 and 81 uV in theory, the actual input voltages in an imperfect converter could map to 79.8 , 80.6 and 80.4 uV. In that case, the value 127 would never occur in the output.
The type of ultra-fast A/D converters used in digital cameras are vulnerable to the phenomenon of missing values. A 14-bit converter much more so than a 12-bit.
The exact locations of these missing values will vary from copy to copy of the camera. For one specific camera the missing values will be fairly consistent, but not 100%
The missing value of 127 in Pondria's camera could be located at the Black Level offset by coincidence, or it could be a critical area for that type of converter.
Pondria wrote:
OK, can anyone send me the dark frame data of his 1Ds2 ? We need to see if the notch position is camera dependent.
Larry sent me his 1Ds2 Dark Frame data. Thank you, Larry. It looks very nice. The shape and the center position of the curve is the same. But his camera doesn't have the notch @ 127. Look how nice it looks. I want to throw away my camera http://www.sesee.com/Photo/Exports/larry-df.gif
