One of the most common search results I get when comparing DSLR's is a website called Snapsort, which uses data collected by DxOMark to compare specs. I'm wondering if these specs are actually meaningful in the real world.
Right now I'm looking at the comparison chart for both of the newest entry-level DSLR's from Canon and Nikon (the 600D and D5100, respectively). According to Snapsort, the Nikon D5100 has:
"More than 20% better image quality"
"Distinguishes more than 2.5x more colors"
"Has 2.1 f-stops more dynamic range"
These claims seem a little odd, considering that both of these cameras are in the same class and released around the same time. Claiming that the Nikon has "20% better image quality" is pretty bold. Is it accurate through? Also, since I'm primarily a landscape photographer, having 2.1 f-stops of more dynamic range seems pretty significant. That means if I shoot a scene with a large amount of dynamic range, I'm going to get noticeably better results with the Nikon compared to the Canon. Or am I?
By the way, I've read that the Nikon inherits the same sensor from the D7000, while the Canon inherits the same (or almost the same) sensor as the 7D. While I've read that the D7000 has better low-light performance, the only measurement that really sparks my interest is the dynamic range.
Saying a camera has "more than 20% better image quality" is like saying, "this sandwich is 20% more delicious." What does "percent better" actually MEAN?
As for dynamic range, you must understand that it is a RANGE--hence the word "range" in the term. It doesn't tell you where the noise floor or the saturation point is located--it simply tells you the useful width. For example, I could have two dice, one numbered from 1 to 6 inclusive, and the other numbered from 101 to 106, and both have the same range, but they don't represent the same set of values. And just because a camera has more DR doesn't mean that you will be able to successfully exploit that property in your images.
Now, I'm not saying the D5100 is not the better camera. I'm merely saying that the claims being put forth are not sufficiently rigorous to support that conclusion.
I could go on, but I will summarize my response as follows: DxOMark = junk numbers. Snapsort = junk conclusions based on junk analysis. Garbage in, garbage out. People need to stop and think about what they read and observe. There is an egregious lack of critical thinking skills among the general public these days.
wickerprints wrote:
Saying a camera has "more than 20% better image quality" is like saying, "this sandwich is 20% more delicious." What does "percent better" actually MEAN?.
Well, if I were to eat two identical sandwiches, but only one of them had bacon, then I would say the sandwich with bacon would be 20% more delicious.
I'm still curious about the dynamic range, though. Are you saying that the numbers are just plain wrong or are you saying that they're just misleading? I'm not quite sure I completely understand your analogy with the dice.
wickerprints wrote:
I could go on, but I will summarize my response as follows: DxOMark = junk numbers. Snapsort = junk conclusions based on junk analysis. Garbage in, garbage out. People need to stop and think about what they read and observe. There is an egregious lack of critical thinking skills among the general public these days.
fd9_ wrote:
Well, if I were to eat two identical sandwiches, but only one of them had bacon, then I would say the sandwich with bacon would be 20% more delicious.
My good sir, adding bacon to the sandwich would make it far MORE than 20% more delicious.
"More than 20% better image quality"
"Distinguishes more than 2.5x more colors"
"Has 2.1 f-stops more dynamic range"
By the way, I've read that the Nikon inherits the same sensor from the D7000, while the Canon inherits the same (or almost the same) sensor as the 7D. While I've read that the D7000 has better low-light performance, the only measurement that really sparks my interest is the dynamic range.
Those numbers only have relevance if you understand Dxomark's testing methodology and they matter to you. I think Dxomark's DR testing methodology is solid, and would be tempted to believe them. Having shot with the D7000 and 7D, the D7000 is undoubtedly a much higher IQ camera.
That said, do I need 2.5x more colors than the 7D? No. I didn't even notice them. I'm sure someone will be able to appreciated the extra color depth much as I was able to appreciate the much cleaner low-iso (and high-iso) D7000 files.
Could someone explain (perhaps in a little more detail) as to why the numbers are junk? I'm still curious about the dynamic range in particular, and the 2 extra f-stops. If I'm shooting the same scene with the same lens and using the exact same settings, won't 2 extra f-stops make a noticeable difference in the picture, in terms of seeing the highlights and/or shadows?
fd9_ wrote:
Could someone explain (perhaps in a little more detail) as to why the numbers are junk? I'm still curious about the dynamic range in particular, and the 2 extra f-stops. If I'm shooting the same scene with the same lens and using the exact same settings, won't 2 extra f-stops make a noticeable difference in the picture, in terms of seeing the highlights and/or shadows?
Probably not as they would be barely noticeable on monitor unless you zoomed in. You would have to dig them out (lighten shadows & darken HLs with adjustment masks) to bring them into the printable and visible range. These types of images usually look rather flat as the extremes need to be "compressed" towards the midrange. Also, the extremes of dynamic range tend to have terrible colors.
fd9_ wrote:
Could someone explain (perhaps in a little more detail) as to why the numbers are junk? I'm still curious about the dynamic range in particular, and the 2 extra f-stops. If I'm shooting the same scene with the same lens and using the exact same settings, won't 2 extra f-stops make a noticeable difference in the picture, in terms of seeing the highlights and/or shadows?
I'm not sure why. I definitely saw at least 2 more stops of shadow range on my D7000 compared to my 7D, and best of all they were free of banding that normally shows up on Canon's current non 1-series bodies when you attempt to use more than 7 stops of dynamic range. There's sample raws on the internet if you want to try it yourself.
DXOMark is a nice technical site, but it never influences any purchase I make. I base any decisions on my real world use and how it delivers my requirements. I've never needed to push a shot 2 stops and never will.
it's as relevent as you wish to make it. DxO has no personal axe to grind with anyone. it's an analysis suite that works off the same criteria/baseline for all makers. an attempt at a level playing field where there has ben much fudging of accurate statements. there are always those who will scoff at a study if theirs is showing up on the short end of the score. take it for what you want.
wickerprints is right to an extent about reading and observing. as to it being junk i don't feel that it is at all. you need to give concrete reasoning why it doesn't work. but in the end we all see what we want to see, hear what we want to hear a nd in the end think what we want to think.
For me, DxO is just another piece of information. On that basis, it's definitely relevant. Whether or not it's correct (i.e. aligns with my opinion) is a different matter.
sjms wrote:
it's as relevent as you wish to make it. DxO has no personal axe to grind with anyone. it's an analysis suite that works off the same criteria/baseline for all makers. an attempt at a level playing field where there has ben much fudging of accurate statements. there are always those who will scoff at a study if theirs is showing up on the short end of the score. take it for what you want.
The problem is not bias, at least none that I'm aware of. The problem is that the analysis itself has very fundamental flaws in methodology, and the conclusions that are drawn are not meaningful. It makes no sense to combine performance data in the manner that they do to obtain a numerical "score." It is as nonsensical as saying "20 miles/hour + 11 liters = 31." Quantities that are measured in fundamentally different ways cannot be mashed together with some pseudoscientifically chosen weight function to derive some overall score.
Furthermore, their results don't stand up to the most basic challenge, which is that in practice, people use a wide variety of lenses with different kinds of optical attributes, and that the performance of the sensor is never observed in isolation. The lens+sensor is operated as a system. Even if you wanted to tease the two apart in order to understand the contribution of each, then you must qualify your conclusions by stating that the measured performance cannot be taken as an indication of real-world results.
wickerprints is right to an extent about reading and observing. as to it being junk i don't feel that it is at all. you need to give concrete reasoning why it doesn't work. but in the end we all see what we want to see, hear what we want to hear a nd in the end think what we want to think.
Feelings and desires have nothing to do with science; only facts and evidence. Any entity with such a cavalier approach to analyzing and summarizing data and purports to present it in an authoritative, "scientific" manner, casts enormous doubt on the validity of the entire process. It simply is not enough to say "people will believe what they want to believe" and shrug one's shoulders. "Cargo cult science" is the term I used for DxOMark, and it is absolutely justified; if one is to have an honest discussion of such things, it becomes necessary to call out the actions of those who are being dishonest, whether or not they do so intentionally.
"The work of science is to substitute facts for appearances, and demonstrations for impressions." --John Ruskin
jcolwell wrote:
For me, DxO is just another piece of information. On that basis, it's definitely relevant. Whether or not it's correct (i.e. aligns with my opinion) is a different matter.
Correctness isn't a function of agreeableness. Correctness is simply a matter of measuring and presenting data accurately, honestly, and completely.
Say you have a contractor remodel your house; they hire plumbers, electricians, carpenters, etc. to rework the dwelling. You find that the drywall was installed incorrectly, and the contractor didn't notice or didn't care. He says, "well, everything else is fine, so what's the problem?" Wouldn't you suspect the quality of the rest of the work he had overseen? Wouldn't you want to redouble your efforts to inspect and ensure everything was done properly?
Science is the same way. One mistake is enough to cast the entire analysis in doubt. And we're not even talking about an error here; we're talking about very, very fundamental flaws in methodology that even a reasonably competent high school student could point out. As a result, one would be remiss in failing to question everything else that was done.
I'm continually astounded by the willingness of people to accept definitive conclusions that are presented in a seemingly authoritative and scientific manner, without asking the most simple of questions as to whether the analysis makes any sense. And yet, there is so much distrust of actual science. I suppose this has some relation to the lack of proper math and science education in the US. People want hard and fast answers. They don't want results that ask more questions than they answer.
wickerprints wrote:
The problem is not bias, at least none that I'm aware of. The problem is that the analysis itself has very fundamental flaws in methodology, and the conclusions that are drawn are not meaningful. It makes no sense to combine performance data in the manner that they do to obtain a numerical "score." It is as nonsensical as saying "20 miles/hour + 11 liters = 31." Quantities that are measured in fundamentally different ways cannot be mashed together with some pseudoscientifically chosen weight function to derive some overall score.
Furthermore, their results don't stand up to the most basic challenge, which is that in practice, people use a wide variety of lenses with different kinds of optical attributes, and that the performance of the sensor is never observed in isolation. The lens+sensor is operated as a system. Even if you wanted to tease the two apart in order to understand the contribution of each, then you must qualify your conclusions by stating that the measured performance cannot be taken as an indication of real-world results.
Feelings and desires have nothing to do with science; only facts and evidence. Any entity with such a cavalier approach to analyzing and summarizing data and purports to present it in an authoritative, "scientific" manner, casts enormous doubt on the validity of the entire process. It simply is not enough to say "people will believe what they want to believe" and shrug one's shoulders. "Cargo cult science" is the term I used for DxOMark, and it is absolutely justified; if one is to have an honest discussion of such things, it becomes necessary to call out the actions of those who are being dishonest, whether or not they do so intentionally.
"The work of science is to substitute facts for appearances, and demonstrations for impressions." --John Ruskin...Show more →
and i totally agree with your last 2 statements. you, for due diligence, need to come up with demonstratable analysis and facts to show where where theirs is flawed
Look at the underlying data and test methods used by DXO, I ignore their overall rating. The overall rating numbers they give as results are based on someone's opinion of whats better. One person might think more DR is the most important thing, while another wants better high ISO performance, while yet another wants the ultimate IQ at base ISO.
Different users might give a different answer to the same test results based on what they want most. Another thing to consider is available lenses, new and used.
If you are free to pick a system, select the one that has the results you value most, and be prepared for the next new round of models to have a different manufacturer with the ratings you want most.
One example, is the Canon DSLR's like the 5D MK II versus the Nikon D700. One has high resolution While the other does better with high ISO. One does video, while the other has a bit better Autofocus system.
If all cameras were identical, performance and feature wise, there would be no incentive to make improvements, I like to see manufacturers come out with improved models and features that lead the field in some or all areas. The other manufacturers will catchup or pass them on the next round of new models, or will lose market share.
