fredmiranda.com

Moderated by: Fred Miranda

New fredmiranda.com Mobile Site

New Feature: SMS Notification alert

New Feature: Buy & Sell Watchlist

FM Forums | Canon Forum | Join Upload & Sell

1 2 3 … 11

13 14 end

Archive 2007 · 1D3 14bit sample

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #1 · 1D3 14bit sample

rico wrote:
The original Canon 1Ds has a dynamic range of about 10 bits. This claim is based on test conditions and personal interpretation which are explained in this report. In the first step, I confirmed that RAW data was linear.

Interesting. I am not sure if DCDRaw compensates for the offset that I mentioned above. For 1Ds2 and 1D3, there was an offset to be compensated to make the raw data linear.

Besides the linear aspect of this data, notice that the sensor is saturated at value 3631, which is only 89% of the advertised 2^12.

1D3 goes up to 15384 or 94% of 2^14. And 1Ds2 does up to 3711 or 91%.

This saturation occurs about 1 stop beyond the "blinkies" during LCD review.

This is why in my DR method, I try to test the real saturation by pulling it in with negative exposure compesation.

Thank you for the test ! I need evening time to digest

Jun 26, 2007 at 08:18 AM

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #2 · 1D3 14bit sample

Greenfields,
I think along the similar line.
In the low 3 stop range in log scale ( talking about literally 0, 1, 2, 3, 4, 5, 6, 7 ), there is not enough data points to form a histogram of any meaningful looking lines. So, I guess the triangular shape is just an artifact of having not enough data points.

Jun 26, 2007 at 08:23 AM

greenfields
Offline
•
Upload & Sell: Off

p.12 #3 · 1D3 14bit sample

Pondria states:

"Besides the linear aspect of this data, notice that the sensor is saturated at value 3631, which is only 89% of the advertised 2^12."

When exploring 5D raw files with dcraw I agree that it appears as though not all of the 12-bit dynamic range is actually used. No matter how low the exposure [max shutter speed with cap on in dark room] I never have anything below bin 128 and no matter how severely an image is overexposed there is never anything above bin 3692. I have not tried to see whether these limits change with temperature. A 12 bit range should run from 0 to 4095.

This may be dcraw: If dcraw's default settings clips the highlights and shadows I would be interested in knowing how to change those settings.

However, the DPP raw histogram never exceeds +3.25 on its scale of EV [referring to my last post + 3.4 would correspond to bin 4095. Bin 3692 would correspond to 3.253] So DPP does the same.

The DPP histogram is presumably a luminance histogram. In principle, you can't get this from the individual channel histograms because the pixels which contribute to a particular bin will generally be in different locations for each channel. In practice this might usually work just because, as Kodak found when researching automatic printing years ago, a large proportion images "integrate to grey". My guess [pure speculation] is that this is what DPP does to produce a histogram which stops short of the +3.4 value. This could make the histogram incirrect for images which did not integrate to grey.

To produce an image DPP then has to apply a Bayer interpolation algorithm to four component channels [two green plus red and blue]. I wonder if the difference bwteen 3692 and 4095 is intended to allow for the possibility that all four channels could be saturated.

As for the lower limit: More speculation: Could that reflect the operation of the noise reduction algorithms which operate before the raw data is saved. For example, subtracting thermal noise from the masked margins of the sensor [that's one reason the effective sensor size is always less than the sensor size] might leave no values below 128. That explanation fails to explain why the raw histogram in DPP certainly does show data below this bin which would correspond to -1.156 EV on the histogram's scale.

The 128 bin lowe limit seems to be a dcraw default setting.

Jun 26, 2007 at 12:56 PM

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #4 · 1D3 14bit sample

Greenfields, you quoted what Rico said

Jun 26, 2007 at 01:24 PM

Paul Gardner
Offline
• • •
Upload & Sell: Off

p.12 #5 · 1D3 14bit sample

Pondria wrote:
Greenfields, you quoted what Rico said

Here is what I said:
1D3 goes up to 15384 or 94% of 2^14. And 1Ds2 does up to 3711 or 91%.

For the lower limit offset, 1Ds2 seems to go down to 100 while 1D3 goes down to 990ish. Those are 2% or 6 % offset respectively.

So in summary, the very raw data from the sensors occupies little less than 90% of the full range available, clipped at the top and at the bottom. The offset and scaling ( minor ) seem to be adjusted in the Linearization process to yield fully saturated data with...Show more →

One more reason for Canon to give us a RAW dump with NO processing. Let the graphic software boys do a complete decode.

Jun 26, 2007 at 02:05 PM

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #6 · 1D3 14bit sample

24 hours of silence

Is everyone taking a break ?

Jun 27, 2007 at 01:51 PM

Jeff
Offline
• • • • • •
Upload & Sell: On

p.12 #7 · 1D3 14bit sample

My job is done here (sound of swiping hands off)...

Actually, I need to re-shoot two files for Rico to take a look at for his little super-statistics analysis.

Jun 27, 2007 at 02:28 PM

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #8 · 1D3 14bit sample

Jeff wrote:
My job is done here (sound of swiping hands off)...

.

Hey stay in

My feel now is that looking into the numbers doesn't help much in terms of the original goal to see the 14bit effect. Yes, data are captured in 14bit length. But that's pretty much it. Your DR test data tells us one story. We need more. Especially when the output image files are 8bit, it will be difficult to see any effect easily.

Jun 27, 2007 at 04:58 PM

SoundHound
Offline
• • • • •
Upload & Sell: On

p.12 #9 · 1D3 14bit sample

I guess that part of this analysis is reverse engineering the processing software that Canon uses. So inconvenient that Canon hasn't released it but not likely-after all Coke still has that secret formula locked up in a safe.

Jun 28, 2007 at 06:49 AM

greenfields
Offline
•
Upload & Sell: Off

p.12 #10 · 1D3 14bit sample

Pondria Wrote:

"For the lower limit offset, 1Ds2 seems to go down to 100 while 1D3 goes down to 990ish. Those are 2% or 6 % offset respectively."

Canon DPP seems to go right down to the first bin - even if it does only seem to go up to the upper limit.

I admit I don't understand why, but wonder whether dcraw applies its own threshold which begs two questions: Why ? Can it be changed ?

Does anyone know how to get a genuine 16-bit histogram [even if its in the form of a text file which can be plotted in a charting program] from a 16-bit .tif ?

That could lead to some interesting tests

Jun 28, 2007 at 12:25 PM

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #11 · 1D3 14bit sample

greenfields wrote:
Pondria Wrote:

"For the lower limit offset, 1Ds2 seems to go down to 100 while 1D3 goes down to 990ish. Those are 2% or 6 % offset respectively."

Canon DPP seems to go right down to the first bin - even if it does only seem to go up to the upper limit.

I admit I don't understand why, but wonder whether dcraw applies its own threshold which begs two questions: Why ? Can it be changed ?

DPP doesn't show us the very raw data. What we see is after the offset is corrected.

Does anyone know how to get a genuine 16-bit histogram [even if its in the form of a text file which can be plotted in a charting program] from a 16-bit .tif ?

That could lead to some interesting tests

I can get it in linear scale which has two problems.
- Log scale is a lot easire to see
- R,G,B channels are not separated. You can see potentially 3 peaks for one Luminosity peak. Rico is ahead of me as he can separate out channels. May be I can try this weekend

Jun 28, 2007 at 01:15 PM

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #12 · 1D3 14bit sample

Pondria wrote:
Greenfields, you quoted what Rico said

Here is what I said:
1D3 goes up to 15384 or 94% of 2^14. And 1Ds2 does up to 3711 or 91%.

For the lower limit offset, 1Ds2 seems to go down to 100 while 1D3 goes down to 990ish. Those are 2% or 6 % offset respectively.

So in summary, the very raw data from the sensors occupies little less than 90% of the full range available, clipped at the top and at the bottom. The offset and scaling ( minor ) seem to be adjusted in the Linearization process to yield fully saturated data with...Show more →

For the low limit, it seems that every horizontal line has the dedicated offset.
From the dng file for a 1D3 file;
BlackLevelDeltaV: 2600 entries
Delta [0] = 262305/256
Delta [1] = 262187/256
Delta [2] = 262230/256
Delta = 262287/256
Delta [4] = 262170/256
Delta [5] = 262167/256
Delta [6] = 262227/256
...
And the values after the division is around 1000, which was observed.

Jun 30, 2007 at 01:18 PM

greenfields
Offline
•
Upload & Sell: Off

p.12 #13 · 1D3 14bit sample

I've learned a lot from this thread but there's still at bit to go.

Pondira wrote:

"DPP doesn't show us the very raw data. What we see is after the offset is corrected."

Not sure what you mean by correcting the offset.

A histogram generated from a dcraw conversion using the switch settings:dcraw.exe -D -4 -t 0 -o 0 -v has no values in the bins below 128 [at least on the 5D. This is hard to understand because it means that only the upper 5ev of the sensor's dynamic range is being converted by dcraw - if the ADC is linear. There is no reason to think that the ADC won't be linear, except at the extremes of the dynamic range. The simplest explanation is that [perhaps because of the switch settings] dcraw is simply trumcating the shadow data and that there are better switch settings to use.

Can't see how DPP could work with just the top 5 EV: [assuming that's the effect of the 128 bin lower limit]. If the ADC was so non linear that the range from bin 128 covered the whole of the sensor's dynamic range, interpolatiion artefacts [combing] should be visible - especially where quantization is visible in the shadows - in the histogram when the data is "stretched" to occupy the 12 bit range displayed by DPP

As for my own question on getting histograms from a 16-bit .tif: The free ImageJ program has a plugin which is claimed to generate tabular histograms from 16-bit .tif files[including individual channels] which can be exported as .txt files then displayed in Excel or a charting application like dplot. However, I've not been able to get the ImageJ plugin to work properly: the reslts are obviously wrong [values appear in to few bins to be credible]. Perhaps it struggles with large files or I'm not using ImageJ correctly.

Jul 04, 2007 at 07:02 AM

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #14 · 1D3 14bit sample

greenfields wrote:
I've learned a lot from this thread but there's still at bit to go.

Pondira wrote:

"DPP doesn't show us the very raw data. What we see is after the offset is corrected."

Not sure what you mean by correcting the offset.

A histogram generated from a dcraw conversion using the switch settings:dcraw.exe -D -4 -t 0 -o 0 -v has no values in the bins below 128 [at least on the 5D.

Exactly that is what's been observed with every canon camera that I looked into.

It seems that the analog OP amp (

) between the sensor and the ADC has dc offset.
For 1D3, it is 1000 count. ( or 6% of the full range ). For 1Ds2, it is about 100. And you say it is 128 for 5D.

This offset is different for each single row ! So, the raw file has the array of the offset values for each row.

RAW data is NOT linear until you correct the offset, meaning that you subtract 128 from the value for 5D case. To be more precise, you subtract the offset for the row that the pixel belongs.

Jul 04, 2007 at 12:27 PM

rico
Offline
• • • • •
Upload & Sell: Off

p.12 #15 · 1D3 14bit sample

greenfields wrote:
...
Can't see how DPP could work with just the top 5 EV: [assuming that's the effect of the 128 bin lower limit].

The lower limit is not the result of bit truncation but, rather, a simple arithmetic bias. As such, loss of dynamic range introduced by bias is a small fraction in theory. In practice, the sensors I inspected don't even utilize that (reduced) range of the linear encoding. A key measure for the DR calculation is the variation of the pixel values, specifically in the dark frame. I have completed my statistical study for 1Ds and 1D3, and will try to post it tonight. I hope to include Pondria's 1Ds2 as well.

Jul 04, 2007 at 05:16 PM

Jeff
Offline
• • • • • •
Upload & Sell: On

p.12 #16 · 1D3 14bit sample

Is it still tonight where you are?

Jul 05, 2007 at 06:13 PM

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #17 · 1D3 14bit sample

Jeff wrote:
Is it still tonight where you are?

My theory is that he was ready with 1D3 and 1Ds data. He was trying to add 1Ds2 data from me at the last moment. And his home-made SW crashed

Jul 05, 2007 at 09:41 PM

rico
Offline
• • • • •
Upload & Sell: Off

p.12 #18 · 1D3 14bit sample

The new Canon 1D3 has a dynamic range of +1.4 bits over the original 1Ds, while the 1Ds2 dominates the 1Ds by a lesser +0.5 bits. For those who like to work with absolute bits, here are the four-channel measures:

1D3 DR (bits) = 10.50 = (10.50 + 10.49 + 10.50 + 10.50) / 4 1Ds2 DR (bits) = 9.61 = ( 9.60 + 9.62 + 9.60 + 9.62) / 4 1Ds DR (bits) = 9.14 = ( 9.14 + 9.18 + 9.20 + 9.05) / 4

Relative differences are quite reliable statistically, but the absolute figures include the noise tolerance of a certain viewer (me).

I have prepared this study programmatically, and invite Linux-savvy persons to download the project, either to audit my methods or to measure other cameras. For bandwidth reasons, I'm not supplying the RAW files used in this study. You can easily download dcraw source from Dave Coffin's site. This project archive (3KB) is http://patternassociates.com/rico/snr/eos1snr.tar.gz

Project title is "Exploring Dynamic Range of Canon DSLR Sensors". Here is the description:

For each camera, calculate signal-to-noise ratio and create graphics.

A camera is described by dark-frame and bright-frame images, both RAW. All Bayer channels are extracted as 16-bit linear values with Dave Coffin's utility `dcraw'. For each channel, produce a decile distribution of values, a plot of the low-order noise, an image of same, and stats.

For calculation of SNR within a given channel, the bright frame supplies the signal amplitude, while the dark frame supplies the noise amplitude. Signal amplitude is the bright frame 9th decile minus the dark frame mean, while noise amplitude is the standard deviation of all values in the dark frame. SNR is then expressed (in log base-2) as signal minus two times the noise. Based on image testing, this calculation delivers a sensible baseline for dynamic range. The SNR reported for the entire Bayer sensor is the channel SNR averaged.

Thanks to Pondria and Jeff for hounding me to complete the mission.

They also provided test frames: a step of unexpected difficulty, as described below. I uncovered a bug in 1Ds RAW generation that may explain the infamous chroma noise of that camera, and which certainly bolloxed my earlier 1Ds DR measurement in this thread: https://www.fredmiranda.com/forum/topic/543552/27#4761925

For that reason, I now confirm dark-frame data with the decile distribution and a plot. The plot should show something very close to a Gaussian (bell) curve because the source is thermal noise.

The bright frame is designed to saturate a good portion of the imager - without burning a hole in it! I requested two stops over the LCD "blinkies", in sunlight to ensure all channels were filled. The dark frame is designed to capture nothing over the entire frame, which should be trivial with a lens cap. Exposure for both was ISO 100, 1/250sec.

Here is the grand decile tabulation, the contents of which are used in the calculations. I didn't bother to label each Bayer channel:

Frame	Pixel value by decile
Frame	low	1st	2nd	3rd	4th	5th	6th	7th	8th	9th	high
1Ds dark (bad)	0	0	0	0	0	0	1	2	2	4	14
	0	0	0	0	0	0	0	1	2	3	15
	0	0	0	0	0	0	0	1	1	3	14
	0	0	0	0	0	0	1	1	2	3	15
1Ds bright	1038	3626	3628	3629	3629	3630	3630	3631	3631	3631	3631
	593	2344	2788	3017	3174	3301	3378	3446	3486	3531	3733
	950	3519	3561	3575	3589	3600	3609	3617	3624	3630	3639
	1013	3619	3622	3623	3624	3624	3629	3630	3631	3632	3632
1Ds dark	12	26	27	28	29	30	31	32	32	34	45
	3	17	18	19	20	20	21	22	23	24	35
	3	18	20	21	21	22	23	24	25	26	38
	11	26	28	29	30	31	31	32	33	35	47
1Ds2 bright	181	1060	1406	3829	3829	3829	3829	3829	3829	3829	3829
	222	1914	2577	3829	3829	3829	3829	3829	3829	3829	3829
	222	1919	2580	3829	3829	3829	3829	3829	3829	3829	3829
	175	1163	1759	3534	3829	3829	3829	3829	3829	3829	3829
1Ds2 dark	112	125	126	126	128	129	129	130	130	131	146
	117	125	126	126	128	128	129	129	130	131	183
	111	125	126	126	128	128	129	129	130	131	150
	117	125	126	126	128	128	129	129	130	131	182
1D3 bright	8022	11443	12612	13656	14590	15375	16121	16133	16133	16133	16133
	15859	16133	16133	16133	16133	16133	16133	16133	16133	16133	16133
	16064	16133	16133	16133	16133	16133	16133	16133	16133	16133	16133
	12812	16133	16133	16133	16133	16133	16133	16133	16133	16133	16133
1D3 dark	998	1017	1020	1022	1023	1024	1026	1027	1028	1030	1074
	995	1018	1021	1022	1023	1024	1026	1027	1028	1030	1054
	996	1017	1020	1022	1023	1024	1026	1027	1028	1030	1057
	999	1018	1021	1022	1023	1024	1026	1027	1028	1030	1054

Next are representative noise plots, including two from the 1Ds:

The dark-frame bias for 1D3 and 1Ds2 is clearly evident in both decile form and graphically. The exact power of two indicates that biasing of RAW data is being performed digitally. The old 1Ds is also digitally biased, but the origin is 0. This is a big mistake! The purpose of bias is to allow the noise curve to be correctly represented in the unsigned integer format of the RAW encoding. Because negative numbers are not expressible, the low side of the 1Ds curve has been truncated or, more properly, been mapped to zero. Numerically speaking, this bug erroneously improved the 1Ds DR by one bit. The notches seen in the 1D3 and particularly in the 1Ds2 plots are also camera bugs, but don't affect the DR calcs.

The improved 1Ds dark frame involved a highly diffused flash source that elevated the noise mean enough to prevent negative numbers. Uniform exposure was achieved by removing the lens, and using an extension tube to better collimate the light.

The noise plots are not too interesting, except to show impressive banding on my ancient 1Ds. Further investigation invites a visit to spectral analysis, from where few return alive. Here is Pondria's 1Ds2 blue channel, with blurring:

Now for a parade of caveats.

First is camera sample variation - 'nuf said. Second, full-value pixels may affect the noise characteristics for dark pixels in a scenic image: measuring this sounds like a bear. Third, in-camera smoothing can artificially improve DR to your favorite astronomically-high number of bits. Of course, scenic detail will be non-existent. While this is a good topic of investigation, I hope my results can stand because the 1 Series cameras are said to avoid such cheap tricks (especially at low ISO). Fourth, there is factor (although small between these cameras) if pixel density is to be considered. To elaborate, one bit of DR is gained for every 4x of resampling, whether in time (image stacking) or in space (image supersampling). For example, my test would claim a 3MP FF sensor achieves +1 bits of DR over a 12MP FF sensor, from a geometric standpoint.

Finally, a moment to reflect (ha, ha) whether the 1D3 (and its 10.5 bits) is all dressed up with nowhere to go:

https://www.fredmiranda.com/forum/topic/543552/7#4728510

Refs:
http://cybercom.net/~dcoffin/dcraw/ RAW decoder `dcraw' by Dave Coffin

Jul 06, 2007 at 05:20 AM

Tom_W
Offline
• • • • •
Upload & Sell: Off

p.12 #19 · 1D3 14bit sample

That's going to take some time to read and understand, Rico.
Can this data be translated into something (like stops) that us photobrained drones can understand?

PS - thanks!

Jul 06, 2007 at 06:29 AM

Pondria
Offline
• • • • • •
Upload & Sell: Off

p.12 #20 · 1D3 14bit sample

Tom_W wrote:
That's going to take some time to read and understand, Rico.
Can this data be translated into something (like stops) that us photobrained drones can understand?

PS - thanks!

Volunteering to be the commentator. ( correct me if I'm wrong )
The Model is to detect (1) the raw data range captured and (2) the noise level. The ratio is DR.

(1) For 12bit sensor, the range is not exactly 0-4095. Very top and bottom ranges are not utilized. You get 11.5 stops of range instead of full 12 stops.

(2) Then, Rico *defines* the noise level as the 2 x Standard Deviation of all the pixels in the dark-frame ( no light induced signal ). That was around 4 to 8 or say 1.5 stops.

(1) - (2) = 11.5 - 1.5 = 10. So, the DR is 10 stops.

This is the simplified version of Rico's methodology.

Jul 06, 2007 at 02:04 PM

1 2 3 … 11 12 13 14 end
FM Forums \| Canon Forum \| Join Upload & Sell 1 2 3 … 11 12 13 14 end