DxO's summarized score has subjective weighting and thus open to debate but the individual DxO data points are extremely valuable for those looking for a comprehensive, objective view on how a sensor performs. There are tidbits buried in the DxO data that are simply not available anyplace else.
One underappreciated (and largely unreviewed) aspect of camera/sensor performance is spectral (color) response. This describes how cleanly the sensor delineates colors, which has direct bearing on color noise for the final rendered image, particular for photographs that undergo a fair amount of post-processing/color adjustment. Some camera makers compromise on the sensor's color filter array in order to get better High ISO sensitivity, at the expense of color separation and thus noise (for all ISOs). If you're up for a bit of a surprise, compare DxO's color response data for any contemporary Nikon camera against any contemporary Canon camera. The data is rather shocking and translates to actual visible results in photographs for those of us who use both systems.
All I know is that I trust my eyes and the prints. The small sensors can't deliver what I want much of the time. Resolution is more important to me than DR or noise, but DXO does not seem to care about that. I really wish we had some 30-40MP FX/FF bodies to take advantage of some of the newest lenses.
wickerprints wrote:
You've misunderstood what I'm getting at here. The issue is not one of how the sensor records the data. The problem with saying "sensor A has 2 stops more DR than sensor B" is that it doesn't really tell us whether that dynamic range is *useful*. That statement doesn't capture, for example, how the sensor might clip the highlights, or whether there's pattern noise that makes recovering information in the shadows useless. I'm not saying it's entirely meaningless, but it's not something that can just be blurted out without qualification, either.
Well you do make a point about the pattern noise since that can make the theoretical DR less than the engineering DR (however, it should also be said that it has been the Canons that have been suffering more from the pattern stuff than the Nikons and Sonys recently ). I'm not sure I agree with you on the clipping highlights bit, I mean look at ISO160 and 200 they are the same thing really, same DR, it's just how you shift around what you call the rated ISO really. I suppose some sensor might do something weird and have some odd sudden drop off or whatnot but I don't think that has been the case. Film and digital sensors have a different highlight roll-off.
"Appear to be," "reasonable," "hard to know exactly what to make of them," and "seem to be [in] the same ballpark" are not exactly quantitative statements, are they? Perhaps you need to read about Millikan's oil drop experiment to appreciate the significance of what I am saying, and the extreme care with which we must avoid fooling ourselves.
The hard to know what to make of them was ONLY referring the overall sensor rating so don't mix that comment in with the rest. Each body varies a bit and so on so it's hard to get absolutely 100% matching results, but from what I've tested and some others have tested most of their numbers have seemed to be pretty solid. Of course I wouldn't go crazy about it if one camera has a fractional difference, etc. since there are in reality error bars around all the data points.
And for the record, not only have I read about Millikan's oil drop experiment, I've actually carried it out .
I am NOT saying that DxO's measurements are wrong. That would imply I have arrived at a different set of results. I am saying that their entire methodology is suspect because what I *can* see is done so sloppily, and for the sake of a largely unquestioning audience, that one would be negligent to assume they only slipped up in that one respect. It's not necessarily junk data, but it absolutely is junk science.
I don't think it is fair to jump to that conclusion just because they try to mash together everything into a final overall number.
I'm really quite perplexed at the eagerness with which some of you are trying to defend DxO's work. Not once have I made any claim of a systematic bias for or against a particular brand of camera based on their data analysis, so if you think that I'm against DxO because they appear to be boosting one manufacturer's results over another, that couldn't be further from the truth.
Maybe because you are laughing them off as total jokes and pretty much blasting the whole thing?
digitalbug30d wrote:
any idiot knows dynamic range lowers as the iso goes up...and you know why....wait for it its darker outside...just like you lose colors when its dark..
I guess I am to only one that uses high iso in the dark...DXO would have you thinking iso6400 is just peachy at noon...
if DXO is so smart then they should tell us the DR of a black cat in a coal mine at midnight...
Here's a rule of thumb for Canon users: Any review that even dares to suggest that a Canon product is inferior to the competition is completely irrelevant and should be treated as utter junk.
For example, DxO does the same tests on all cameras.
In these tests (which are the same for all cameras), Canon’s latest sensors lag behind the competition.
Therefore, the DxO numbers are complete and utter junk! 'Nough said.
skibum5 wrote:
Well you do make a point about the pattern noise since that can make the theoretical DR less than the engineering DR (however, it should also be said that it has been the Canons that have been suffering more from the pattern stuff than the Nikons and Sonys recently ). I'm not sure I agree with you on the clipping highlights bit, I mean look at ISO160 and 200 they are the same thing really, same DR, it's just how you shift around what you call the rated ISO really. I suppose some sensor might do something weird and have some odd sudden drop off or whatnot but I don't think that has been the case. Film and digital sensors have a different highlight roll-off....Show more →
What I am speaking about regarding highlight clipping, is that digital sensors do not have the same dynamic range across the visible spectrum--highlights don't get clipped at the same point. Thus, there may be differences between sensors in terms of how they handle highlights that are not captured by saying "this sensor has 12.5 stops of dynamic range." It is entirely conceivable that one sensor could be more tolerant of an exposure pulled in post compared to another sensor of the same measured dynamic range, because it handles highlight clipping in a more perceptually pleasing way. And this speaks to your statement about film vs. digital, in that negative film is generally very tolerant of overexposure compared to a digital sensor, not so much because it has more dynamic range, but because the way it reacts to overexposure is more uniform across the visible spectrum. (It's also nonlinear but that doesn't pertain to the point I'm making).
Also, regarding the pattern noise--the anecdotal evidence from various photographers supports the hypothesis that Nikon bodies tend to exhibit less pattern noise at the same ISOs compared to Canon bodies.
The hard to know what to make of them was ONLY referring the overall sensor rating so don't mix that comment in with the rest. Each body varies a bit and so on so it's hard to get absolutely 100% matching results, but from what I've tested and some others have tested most of their numbers have seemed to be pretty solid. Of course I wouldn't go crazy about it if one camera has a fractional difference, etc. since there are in reality error bars around all the data points.
And for the record, not only have I read about Millikan's oil drop experiment, I've actually carried it out ....Show more →
Do you know why I made the reference to Millikan's oil drop experiment, and how it pertains to your above reasoning?
I don't think it is fair to jump to that conclusion just because they try to mash together everything into a final overall number.
We are just going to have to disagree. What you think is fair and what I think is fair is not going to be reconciled by debating the issue further. My position has been made clear--a misrepresentation of such a nature as that put forth by DxOMark is grounds for distrusting the entire analysis. As in the analogy I furnished earlier, if I find that the contractor didn't catch a problem, then I have a reasonable suspicion that they did not properly oversee the quality of workmanship in general, and I am going to be all the more vigilant in making sure that everything else was done correctly.
Maybe because you are laughing them off as total jokes and pretty much blasting the whole thing?
Who said anything about "laughing them off as total jokes?" I take their lack of rigor quite seriously--enough to bother to write multiple posts on the topic. This is not some kind of cavalier dismissal on my part. Simply because you disagree with my position does not give you the right to think that my position is not a reasoned one.
I would say, DxOMark is more relevant than those who trash it's research results.
fd9_ wrote:
Hey guys,
One of the most common search results I get when comparing DSLR's is a website called Snapsort, which uses data collected by DxOMark to compare specs. I'm wondering if these specs are actually meaningful in the real world.
Right now I'm looking at the comparison chart for both of the newest entry-level DSLR's from Canon and Nikon (the 600D and D5100, respectively). According to Snapsort, the Nikon D5100 has:
"More than 20% better image quality"
"Distinguishes more than 2.5x more colors"
"Has 2.1 f-stops more dynamic range"
These claims seem a little odd, considering that both of these cameras are in the same class and released around the same time. Claiming that the Nikon has "20% better image quality" is pretty bold. Is it accurate through? Also, since I'm primarily a landscape photographer, having 2.1 f-stops of more dynamic range seems pretty significant. That means if I shoot a scene with a large amount of dynamic range, I'm going to get noticeably better results with the Nikon compared to the Canon. Or am I?
By the way, I've read that the Nikon inherits the same sensor from the D7000, while the Canon inherits the same (or almost the same) sensor as the 7D. While I've read that the D7000 has better low-light performance, the only measurement that really sparks my interest is the dynamic range....Show more →
The most important advantage of the Nikon is that it can use 169 lenses, compared to only 162 for the Canon.
Maybe we should be discussing snapsort as well as DxOMark. Yes, I know they didn't flag this as an important feature, just trying to inject a little levity into something that in fact is a serious issue (testing and rating methods).
...In conclusion, I think we can all agree that DxOMark's "Overall Score" is misleading, particularly because it consolidates multiple benchmarks into a single number which has no real unit of measurement, much the same way a PC benchmarking program might average together your RAM, HDD speed, and graphics card to derive a "Total Score" that approximates your computer's performance. To make matters worse, Snapsort uses this arbitrary number to derive a percentage of which one camera is better than another (for example, when they claimed that the D5100 has "20% better image quality" then the T3i).
However, I think that DxOMark can still be used to draw accurate comparisons of individual cameras. But there is a subtle catch here. In order to make a proper comparison using the data that DxOMark provides, one must first understand what the benchmarks mean. For example, the Nikon sensor has a higher dynamic range at base ISO then the Canon sensor. From my understanding, that doesn't mean I'm going to get a better looking picture from the Nikon camera (which is what I originally thought). Instead, what it means is that I'll be able to push the shadow details a little harder in post processing. Hence, the dynamic range benchmark is only relevant if I'm going to push the shadows hard to begin with. (By the way, for those of you who are interested, here is a recent thread about this topic - https://www.fredmiranda.com/forum/topic/996962/0).
So I think it's important to understand that while the DxOMark benchmarks can be useful if understood properly, they can also be misleading to the novice reader. And I think that's the biggest problem here.
Please let me know if I've said anything that is inaccurate.
fd9_ wrote:
...In conclusion, I think we can all agree that DxOMark's "Overall Score" is misleading, particularly because it consolidates multiple benchmarks into a single number which has no real unit of measurement, much the same way a PC benchmarking program might average together your RAM, HDD speed, and graphics card to derive a "Total Score" that approximates your computer's performance. To make matters worse, Snapsort uses this arbitrary number to derive a percentage of which one camera is better than another (for example, when they claimed that the D5100 has "20% better image quality" then the T3i).
However, I think that DxOMark can still be used to draw accurate comparisons of individual cameras. But there is a subtle catch here. In order to make a proper comparison using the data that DxOMark provides, one must first understand what the benchmarks mean. For example, the Nikon sensor has a higher dynamic range at base ISO then the Canon sensor. From my understanding, that doesn't mean I'm going to get a better looking picture from the Nikon camera (which is what I originally thought). Instead, what it means is that I'll be able to push the shadow details a little harder in post processing. Hence, the dynamic range benchmark is only relevant if I'm going to push the shadows hard to begin with.
So I think it's important to understand that while the DxOMark benchmarks can be useful if understood properly, they can also be misleading to the novice reader. And I think that's the biggest problem here.
Please let me know if I've said anything that is inaccurate. ...Show more →
The reason shadows can be pushed so far on the D5100/D7000 is because they're cleaner to begin with (less read-noise), so depending on your scene that can mean better IQ even without pushing. As for pushing, the D7000's shadows can be pushed 2 1/2 stops further than Canon's on a relative noise basis and 3-4 stops further if you account for the absence of banding.
digitalbug30d wrote:
its a dumb chart think about it....
It is not a dumb plot because the noise can occur at different stages and when the amp noise at one stage is enough to be relevant you end up with not getting a straight plot and with most cameras and certainly all Canon camera so far, this is very much the case.
wickerprints wrote:
What I am speaking about regarding highlight clipping, is that digital sensors do not have the same dynamic range across the visible spectrum--highlights don't get clipped at the same point. Thus, there may be differences between sensors in terms of how they handle highlights that are not captured by saying "this sensor has 12.5 stops of dynamic range." It is entirely conceivable that one sensor could be more tolerant of an exposure pulled in post compared to another sensor of the same measured dynamic range, because it handles highlight clipping in a more perceptually pleasing way. And this speaks to your statement about film vs. digital, in that negative film is generally very tolerant of overexposure compared to a digital sensor, not so much because it has more dynamic range, but because the way it reacts to overexposure is more uniform across the visible spectrum. (It's also nonlinear but that doesn't pertain to the point I'm making).
Also, regarding the pattern noise--the anecdotal evidence from various photographers supports the hypothesis that Nikon bodies tend to exhibit less pattern noise at the same ISOs compared to Canon bodies.
Do you know why I made the reference to Millikan's oil drop experiment, and how it pertains to your above reasoning?
We are just going to have to disagree. What you think is fair and what I think is fair is not going to be reconciled by debating the issue further. My position has been made clear--a misrepresentation of such a nature as that put forth by DxOMark is grounds for distrusting the entire analysis. As in the analogy I furnished earlier, if I find that the contractor didn't catch a problem, then I have a reasonable suspicion that they did not properly oversee the quality of workmanship in general, and I am going to be all the more vigilant in making sure that everything else was done correctly.
Who said anything about "laughing them off as total jokes?" I take their lack of rigor quite seriously--enough to bother to write multiple posts on the topic. This is not some kind of cavalier dismissal on my part. Simply because you disagree with my position does not give you the right to think that my position is not a reasoned one. ...Show more →
Well tossing out the baby with the bathwater is not all that reasoned IMO and, quickly, I don't think there has been much of a difference in the nature of highlight roll off between the DSLR sensors.
Well dpreview publishes an equally useless %age score e.g. the new D5100 review says that it scores 76% - whatever that means (its actually even more useless because it tries to capture much more than just IQ - including build quality, focusing accuracy, performance etc and even totally subjective measures like "value"). But most of us know that we need to ignore that number and look through the actual review to find out the information that we need. Same is true for DxOMark. It is true that unlike dpreview, DxOMark tries to make a big deal out of its sensor rankings and sensor scores. And in that sense they are guilty of somewhat misleading the users. Nevertheless that doesn't change the usefulness of the data that they provide and there is lots of information to gain if one is willing to dive down into all the graphs.
skibum5 wrote:
Well tossing out the baby with the bathwater is not all that reasoned IMO and, quickly, I don't think there has been much of a difference in the nature of highlight roll off between the DSLR sensors.
You are alleging that I am dismissing the data outright. That is not the case. I am saying that one is justified in being skeptical of the data and methodology because they have demonstrated a significant degree of misrepresentation of that data. So for you to call that "tossing out the baby with the bathwater" is inappropriate, because you are operating under the assumption that the "baby" even exists to begin with--whereas I am saying, let's see if there is in fact a baby in the bath. If you want to call that unreasonable, then there's nothing I can say or do to rebut what is essentially your unscientific opinion of my skepticism.
As for the nature of the highlight roll off, again, I am not saying any differences are significant. I am merely postulating various reasons why simply stating a comparison of dynamic ranges doesn't necessarily reflect how well the resulting images can cope with pulled or pushed exposures in post. Even you can and have agreed that differences in the nature of read noise, for example, is not something that is faithfully captured with this metric. Whether those differences are significant in two compared bodies is not the point.
My critique is not based on the conclusions that DxO draws, or the accuracy of the data per se, as you seem to be implying. Rather, it is a critique of their methodology and presentation under an air of scientific authority that is not in fact as rigorous as they make it appear; consequently, that misrepresentation subjects them to additional scrutiny. In plain terms, because they make themselves appear scientific, that opens them up to being held to actual scientific standards.
Finally, you still have not been able to answer my question as to why I referred to the oil drop experiment. In fact, I am curious to see if ANYONE can explain why I referred to it.
thedigitalbean wrote:
Yes it does mean something. It means that if you need to shoot something at base ISO and then push the shadows 4 stops in your RAW converter, the D5100 will come out cleaner than the Canon cameras. Some people need to do this, others don't. You decide for yourself if you do.
Having used all of those cameras in question, yes, the Nikon D700 and D5100 do have a dynamic range advantage, namely that at base ISO read noise is virtual absent, meaning you can push exposure digital afterwards will relative impunity.
Interesting the way the 5D2 rolls off at the low-ISO range - perhaps this is due to the noted banding issue that some attribute to amplifier noise or processing noise or some such thing. In the 800-6400 range, the 5 has the advantage, but that flattening of the curve below 400 or so subtracts from the camera's advantage.
I'm assuming that the 550D is the Rebel T2i, which uses a similar sensor to the 7D. Knowing that, and knowing that the 5D2 has a respectable noise advantage over the 7D above ISO 400, the chart is sensible based on what I am familiar with. Can't speak for the Nikon since I don't have one.
That chart strikes me as a bit strange, because if it is to be believed, what it tells me is that the D5100 basically has one real ISO. Think about it. The relationship between ISO and DR is such that the slope is nearly constant--so in terms of DR, you may as well shoot at ISO 100 and push your exposure however many stops in post that you are underexposed. For example, if the D5100 has 13 stops DR @ ISO 200, and 12 stops @ ISO 400, then you could just shoot -1 EV @ ISO 200, capture the 13 stops DR, then push it in post and you will have essentially lost nothing. You may even have succeeded in gaining some highlight latitude.
So in a sense, while the chart looks favorable for the D5100 (and it *is* favorable compared to the Canon bodies, mind you), a more functionally useful performance would be reflected by a curved slope, because this means that there would be some real advantage to increasing the sensitivity. Note that in the 5D2, for example, the curve begins to behave linearly around ISO 1600, which makes sense because in fact, ISOs at 3200 and beyond aren't true increases in sensitivity. The camera is underexposing and then pushing the result. This is reflected in the linear slope of the curve. Thus, much the same could be said for the Nikon body, but this behavior extends even to the low ISO range, which means that if I were to use this body, I would probably stick to ISO 100-200 even if, say, ISO 1600 were required; underexpose, then push. The chart doesn't give me much reason to pick ISO 1600 at the outset.
Of course, we would have to see if the empirical results support this hypothesis. It would be quite easy to test.
). I'm not sure I agree with you on the clipping highlights bit, I mean look at ISO160 and 200 they are the same thing really, same DR, it's just how you shift around what you call the rated ISO really. I suppose some sensor might do something weird and have some odd sudden drop off or whatnot but I don't think that has been the case. Film and digital sensors have a different highlight roll-off.
