p.2 #1 · Panasonic Diffractive Color Filter/Splitter patent
Taylor Sherman wrote:
What? no. We are measuring W, Y, and C here, not R, G and B. The way you are adding the noise is as if each pixel was measuring R, G, and B and then using that to come up with it's value of W, Y, or C.
...
Check the "assumptions" explaining my terminology at the top of my model post: the incoming light is made up of R+G+B photons of Red, Green, and Blue light hitting each pixel. You are making the mistake of assuming that W represents about the same number of photons as R, which isn't right: the "White" filtered pixels sees White = Red+Green+Blue (W=R+G+B) photons, thus has noise dW = sqrt(W) = sqrt(R+G+B), etc. This is critical for allowing me to compare photon/noise counts at the end: "sqrt(W+C)" isn't "the noise of twice the photons" as R, it's sqrt(R+G+B + B+G), or ~sqrt(5*R).
p.2 #4 · Panasonic Diffractive Color Filter/Splitter patent
I just got the time to read the full Nature Photonics article, which is far more interesting/informative than the marketing drivel of the press release.
The article introduction emphasizes the usefulness of this invention for increasingly small (~1 micron) pixel sizes, which means cellphone/P&S sensors.
On color separation: It may increase noise due to the use of a matrix operation, causing a potential concern about the degradation of resolution after colour correction. This will be discussed later.
The specific impact on color resolution is not quantified in the paper, but can be estimated from the color-untangling matrix given in the paper by a method similar to my preceding "toy model".
The "later discussion" is that the color resolution of the device might be improved by adding more varieties of color deflector than the basic Red/Blue deflection scheme: Using a combined G- and M-deflector and B- and Y-deflector instead of the combination of an R-deflector and white, and white and B-deflector, the area ratio responsible for colour separation would rise by 100%, making it possible to improve the colour purity.
The paper claims that, so far as spatial resolution goes, the results "maintain the same level of resolution" as Bayer-filtered sensors.
p.2 #5 · Panasonic Diffractive Color Filter/Splitter patent
mpmendenhall wrote:
Check the "assumptions" explaining my terminology at the top of my model post: the incoming light is made up of R+G+B photons of Red, Green, and Blue light hitting each pixel. You are making the mistake of assuming that W represents about the same number of photons as R, which isn't right: the "White" filtered pixels sees White = Red+Green+Blue (W=R+G+B) photons, thus has noise dW = sqrt(W) = sqrt(R+G+B), etc. This is critical for allowing me to compare photon/noise counts at the end: "sqrt(W+C)" isn't "the noise of twice the photons" as R, it's sqrt(R+G+B + B+G), or ~sqrt(5*R).
Got it, thanks for the explanations. That's pretty interesting. I kind of have to keep going back and looking at it because part of me doesn't want to believe it (the part that thinks more light is better). I suppose your example is one way of explaining why sensors don't use WYC filter packs! If all of our digital processes were set up to deal with that color space then I guess it would make sense, but when the first thing you do is convert to RGB, you're amplifying the noise.
p.2 #6 · Panasonic Diffractive Color Filter/Splitter patent
Yeah, there is still a part of me which doesn't want to believe it either For example, what about all this talk of Canon using "thin" CFAs to improve high ISO noise performance, and thus losing out on colour fidelity and resolution?
p.2 #7 · Panasonic Diffractive Color Filter/Splitter patent
Honor to those who try in the quest for the holy grail of lossless color separation.
It would be neat if it was possible to make a stacked sensor with "light tubes" down to deflectors and photodiodes at different height levels. No "W+R" and "W+B" pixels, only "W-R", "R", "W-B" and "B".
p.2 #8 · Panasonic Diffractive Color Filter/Splitter patent
Taylor Sherman wrote:
If all of our digital processes were set up to deal with that color space then I guess it would make sense, but when the first thing you do is convert to RGB, you're amplifying the noise.
There's no way to re-work the digital process to avoid this, because in the end you have to deal with human eyes (which are fundamentally RGB devices): you eventually have to produce output that causes light with the correct proportions of R, G, and B to reach a viewer. When information about these proportions is lost at the start of the process by partially scrambling the counts, there's no way to recover it by different processing in-between.
p.2 #9 · Panasonic Diffractive Color Filter/Splitter patent
mpmendenhall wrote:
There's no way to re-work the digital process to avoid this, because in the end you have to deal with human eyes (which are fundamentally RGB devices): you eventually have to produce output that causes light with the correct proportions of R, G, and B to reach a viewer. When information about these proportions is lost at the start of the process by partially scrambling the counts, there's no way to recover it by different processing in-between.
Is that true? RGB is the only synthesis method which works for human eyes?
I have a hard time believing that a WYC sensor wired up to a corresponding WYC display would produce an image that didn't look right (when properly calibrated). The Y and C outputs would still tickle our cones in the right proportion to induce a valid impression, wouldn't they?
p.2 #10 · Panasonic Diffractive Color Filter/Splitter patent
Taylor Sherman wrote:
Is that true? RGB is the only synthesis method which works for human eyes?
The color response of human eyes is based on the chemical sensors in you eye that respond with a particular sensitivity to various portions of the spectrum (roughly the "blue," "green," and "red" parts, though with a lot overlap). Display technology can use a wide variety of different colors/pigments/methods to show you an image, but, ultimately, what eye cares about is the differing proportions of excitation of the "red", "green", and "blue" receptors in your eye that give you the perception of color. When colors of light are "mixed" hitting the camera sensor and uncertainty (noise) introduced, information about the precise ratio of R/G/B is irrecoverably lost --- so, regardless of how the image is processed/displayed afterwards, your eye will see the R/G/B signal muddled by noise.
p.2 #11 · Panasonic Diffractive Color Filter/Splitter patent
Taylor Sherman wrote:
Is that true? RGB is the only synthesis method which works for human eyes?
I have a hard time believing that a WYC sensor wired up to a corresponding WYC display would produce an image that didn't look right (when properly calibrated). The Y and C outputs would still tickle our cones in the right proportion to induce a valid impression, wouldn't they?
No, a positive tristimuli of White-Yellow-Cyan can never combine to make a green color. It would not be able to make blue or red color either... It only "kind of" works in a sensor because you can subtract channel values from each other.
The only way that you could use subtraction in a light emitting device would be to use phased light like in a laser, and I'm not entirely sure that I would want that in a screen....
p.2 #12 · Panasonic Diffractive Color Filter/Splitter patent
mpmendenhall wrote:
Here's a simple toy mathematical model to demonstrate what happens with color errors.
Suppose we have a camera with 3 pixels, and light consisting of R+G+B photons is hitting each pixel.
The "usual" scheme is to use an R, G, or B filter on each of the three pixels. This directly gives the color signal one wants, with the usual counting statistics noise proportional to the square root of the number of photons for each channel:
R, dR = sqrt(R)
G, dG = sqrt(G)
B, dB = sqrt(B)
Now, suppose we instead use white, yellow, and cyan filters (not exactly what's happening here, similar). Then, what we directly measure at the three pixels (with its counting statistics noise) is:
W = R+G+B, dW = sqrt(R+G+B)
Y = R+G, dY = sqrt(R+G)
C = G+B, dC = sqrt(G+B)
From this, we can un-tangle the colors to get back to R, G, B:
but now the color errors are about sqrt(5) to sqrt(6) times bigger (assuming roughly equal counts in each channel). We've collected about 7/3 times as much light (so the luminance signal is much better), but the color noise is as bad as if we only had 1/5 the light compared to the R,G,B filter scenario!...Show more →
Well even in bayer we don't have "red pixels" measuring just red, and green pixels measuring just green etc. The color filters have overlapping range and a correction matrix is used in bayer as well to do the untangling. So unlike the simplified equations above, in bayer we actually have:
R' = alpha1 * R - alpha2 * G - alpha3 * B
and so on
The basic equations would similar for bayer and this new scheme. However the difference would be in the values of alphas and that difference will determine the impact on noise.
p.2 #13 · Panasonic Diffractive Color Filter/Splitter patent
Taylor Sherman wrote:
What? no. We are measuring W, Y, and C here, not R, G and B. The way you are adding the noise is as if each pixel was measuring R, G, and B and then using that to come up with it's value of W, Y, or C.
Starting from pixels that actually measure these things directly, we have
W = W, dW = sqrt(W)
Y = Y, dY = sqrt(Y)
C = C, dC = sqrt(C)
So, we have increased the noise sqrt(2) to sqrt(3) times over what it was in-sensor.
So we've got a 2-3x improvement in signal, with a sqrt(2)-sqrt(3) increase in noise. It's a win. ...Show more →
No.
Add in the saturation factor, same problem as in the Foveon. The "red" you mark as the resulting R' is actually something close to pale warm red. The G' is close to a slightly greenish neutral. The B' is like an overcast sky, not really blue - but close. To get saturation up to even sRGB standard primaries, you'd need to amplify the channel differences by a factor of at least two, quadrupling the loss in color fidelity and doubling the noise effect sum.
The main strength this method has over something like the Foveon is that you can actually tune the cutoff wavelengths, giving a much lower metameric failure rate.
Otherwise, it's not really a solution for anything that does not have a fixed lens (or pixels larger than ~1.1-1.4µm for that matter) - the adaptation problems are to high. And unfortunately the losses at F2.0 angles is more than 50%, making the system total a theoretical maximum T-stop of ~T2.0 (with a F1.0 lens)
But in a small-sensor solution like in smaller compacts and smart-phone modules, it's brilliant. The color quality is not much worse than the current 1.1µm cell solutions, since the CFA layer has to be very thin when the pixels are small (otherwise you get color smearing/leakage effects).
p.2 #14 · Panasonic Diffractive Color Filter/Splitter patent
mpmendenhall wrote:
Here's a simple toy mathematical model to demonstrate what happens with color errors.
Suppose we have a camera with 3 pixels, and light consisting of R+G+B photons is hitting each pixel.
The "usual" scheme is to use an R, G, or B filter on each of the three pixels. This directly gives the color signal one wants, with the usual counting statistics noise proportional to the square root of the number of photons for each channel:
R, dR = sqrt(R)
G, dG = sqrt(G)
B, dB = sqrt(B)
Now, suppose we instead use white, yellow, and cyan filters (not exactly what's happening here, similar). Then, what we directly measure at the three pixels (with its counting statistics noise) is:
W = R+G+B, dW = sqrt(R+G+B)
Y = R+G, dY = sqrt(R+G)
C = G+B, dC = sqrt(G+B)
From this, we can un-tangle the colors to get back to R, G, B:
but now the color errors are about sqrt(5) to sqrt(6) times bigger (assuming roughly equal counts in each channel). We've collected about 7/3 times as much light (so the luminance signal is much better), but the color noise is as bad as if we only had 1/5 the light compared to the R,G,B filter scenario!...Show more →
Second problem with your analysis is that you are talking about noise rather than SNR. What we care about is the SNR not noise. Noise goes up when you have more light but the signal also goes up. In your analysis you are looking at the increased noise but not accounting for the increased signal and as a result coming up with incorrect conclusions.
p.2 #15 · Panasonic Diffractive Color Filter/Splitter patent
curious80 wrote:
Second problem with your analysis is that you are talking about noise rather than SNR. What we care about is the SNR not noise. Noise goes up when you have more light but the signal also goes up. In your analysis you are looking at the increased noise but not accounting for the increased signal and as a result coming up with incorrect conclusions.
The R',G',B' "signals" that I extract are the same magnitude as the R,G,B signals (you should be able to verify this by following through the calculations from the start --- I'm not trying to pull a fast one here), so their increased noise is directly comparable. You are correct that signal-to-noise (e.g. dR/R vs. dR'/R') is what actually matters; in the terminology I've used, R' = R but dR' = ~sqrt(5)*dR, so the SNR is indeed impacted by the amount I indicate.
p.2 #16 · Panasonic Diffractive Color Filter/Splitter patent
curious80 wrote:
Well even in bayer we don't have "red pixels" measuring just red, and green pixels measuring just green etc. The color filters have overlapping range and a correction matrix is used in bayer as well to do the untangling.
I agree, absolutely. I omitted this (along with several other simplifying assumptions) from my "toy model" for clarity, because including too many details just obscures the main point under messy algebra. My "RGB" Bayer sensor is an idealized best case, where real sensors have more overlap (often to increase light throughput), and correspondingly worse color sensitivity.
The Nature Photonics paper shows the actual spectral sensitivity curves for, e.g., the "W+R" and W-R" pixels produced by Panasonic's scheme. The actual amount of color deflection is quite a bit less than perfect --- the results are really more like "W-R/4" and "W+R/4". Just like extra mixing in RGB filters, this makes the color separation even worse than the simplified "ideal" case.
p.2 #17 · Panasonic Diffractive Color Filter/Splitter patent
mpmendenhall wrote:
The R',G',B' "signals" that I extract are the same magnitude as the R,G,B signals (you should be able to verify this by following through the calculations from the start --- I'm not trying to pull a fast one here), so their increased noise is directly comparable. You are correct that signal-to-noise (e.g. dR/R vs. dR'/R') is what actually matters; in the terminology I've used, R' = R but dR' = ~sqrt(5)*dR, so the SNR is indeed impacted by the amount I indicate.
But you also have to factor in the fact that the R that you are calculating in bayer is once every 4 pixels. So you have a total of R signal per 4 pixels. In this case you will get an R' signal for every pixel. So over 4 pixels you are getting 4 times the signal.
p.2 #18 · Panasonic Diffractive Color Filter/Splitter patent
mpmendenhall wrote:
I agree, absolutely. I omitted this (along with several other simplifying assumptions) from my "toy model" for clarity, because including too many details just obscures the main point under messy algebra. My "RGB" Bayer sensor is an idealized best case, where real sensors have more overlap (often to increase light throughput), and correspondingly worse color sensitivity.
The Nature Photonics paper shows the actual spectral sensitivity curves for, e.g., the "W+R" and W-R" pixels produced by Panasonic's scheme. The actual amount of color deflection is quite a bit less than perfect --- the results are really more like "W-R/4" and "W+R/4". Just like extra mixing in RGB filters, this makes the color separation even worse than the simplified "ideal" case....Show more →
W-R/4 is interesting. I think I should go over the paper in detail
p.2 #19 · Panasonic Diffractive Color Filter/Splitter patent
curious80 wrote:
But you also have to factor in the fact that the R that you are calculating in bayer is once every 4 pixels. So you have a total of R signal per 4 pixels. In this case you will get an R' signal for every pixel. So over 4 pixels you are getting 4 times the signal.
In my "toy model" above, I am consistently using the full light information available from a 3-pixel sensor (either with R,G,B filters or W,M,C) with 3x R+G+B incident on each pixel. There is no additional light that I throw out of the calculations. I may have a mistake somewhere, but this isn't it.
p.2 #20 · Panasonic Diffractive Color Filter/Splitter patent
mpmendenhall wrote:
In my "toy model" above, I am consistently using the full light information available from a 3-pixel sensor (either with R,G,B filters or W,M,C) with 3x R+G+B incident on each pixel. There is no additional light that I throw out of the calculations. I may have a mistake somewhere, but this isn't it.
I am not saying you are throwing away the light in your calculations. What I am saying is that in any 2x2 pixel patch, in the ideal bayer case you will have light R at pixel (0,0) but then you will have no red at pixel (0,1), pixel(1,0) and pixel (1,1).
In the modified scheme you will get R' at pixel (0,0) which will correspond in its magnitude with the R from the bayer case. But then at pixel (0,1) you would again have another R' and then at pixel (1,0) another R' and then at (1,1) another R'. That means that in a 2x2 pixel patch the total red signal would be about 4 times the bayer case. However the measurements would be more noisy as you illustrated.
).