The DxO Mark stuff is relevant if used correctly. The correct way to use it is not to compare individual summary scores, but to study the charts to gain an appreciation for how well the cameras can perform.
Take dynamic range, for example. When reduced to a single DR score it generally applies at only the base ISO and gives no indication of how relatively good or bad the DR is at higher ISOs. Some cameras that are better at base ISO are also worse at higher ISO. Comparing the DR charts is far more revealing unless we only shoot at base ISO.
There are many factors way beyond what DxO Mark measures that also determine how good a camera is for a particular purpose or an individual person. Ergonomics, features, lens options, etc., etc. However, at least the DxO Mark stuff is measured in a similar way for valid comparison with other cameras, and that cannot be said for a great many camera reviews and user opinions.
sjms wrote:
and i totally agree with your last 2 statements. you, for due diligence, need to come up with demonstratable analysis and facts to show where where theirs is flawed
I already told you what was wrong. If you couldn't understand the problem, that's not because I didn't demonstrate it, since other members have understood where I have pointed out the flaws.
If you take measurements of, say, the noise of a sensor, and then take measurements of the dynamic range, you cannot "average" the two measurements together using some arbitrarily chosen weighting function to get an overall score that makes any sense. The units are not even the same. If you can't grasp something as basic as that, we have nothing further to discuss.
Then, as Alan pointed out, you can't even average measurements of something taken at various ISOs, without diluting the meaning of the relative performance of two different sensors, because one may do well at lower ISOs whereas the other may do better at high ISOs. One should at least take into consideration the relative frequency at which one uses different ISOs.
Finally, my point is that if an analysis makes such simple errors for the sake of distilling down their data in a way that a layperson can appreciate, then not only is this an act of intellectual dishonesty, but it calls into suspicion everything they have done. In other words, if I can't trust them to avoid pasting some global "score" on the entire camera, I shouldn't trust whether they were competent enough to properly measure the data they captured in the first place.
Other reviewers are not necessarily immune to the same flaws and criticisms. Photozone.de is another site I strongly distrust, but for somewhat different reasons. But you really want to understand why I go after DxO so ferociously? It's because they purport to be scientific about their evaluation. At least other reviewers will do something silly like give some number of "stars" or a thumbs up/down, or a recommendation. While their reviews may hold technical data, they don't have a pretense of being able to boil down everything they looked at into some nonsensical numerical score, for which an audience might look at and arrive at some comically naive statement like,
"20% better image quality."
Indeed, I would be laughing if it weren't so utterly pathetic.
Monito wrote:
The fallacy of doing it by the numbers only: Eyes vs. Numbers.
One line from Michael's article is so very true:
"My fear though is that the web forum fanboys are going to jump all over these DxO results and make inappropriate conclusions. But, what else is new? Pros and serious amateurs will likely continue to make their judgments and life will go on. Twas ever thus."
wickerprints wrote:
Saying a camera has "more than 20% better image quality" is like saying, "this sandwich is 20% more delicious." What does "percent better" actually MEAN?
Agree here. It's relatively meaningless.
As for dynamic range, you must understand that it is a RANGE--hence the word "range" in the term. It doesn't tell you where the noise floor or the saturation point is located--it simply tells you the useful width. For example, I could have two dice, one numbered from 1 to 6 inclusive, and the other numbered from 101 to 106, and both have the same range, but they don't represent the same set of values. And just because a camera has more DR doesn't mean that you will be able to successfully exploit that property in your images.
Disagree here. You are not making good sense here. I mean come image they stored a CD as the values 65536 to 131071 instead of 0 to 65535? who cares (other than they'd be wasting a lot of space in this case, although they would have to since you could then just declare that 0 bits set means 65536 and all 16bits set means 65535, who cares?)?
I could go on, but I will summarize my response as follows: DxOMark = junk numbers. Snapsort = junk conclusions based on junk analysis. Garbage in, garbage out. People need to stop and think about what they read and observe. There is an egregious lack of critical thinking skills among the general public these days.
The overall ratings on DxO are close to junk, although not out and out junk. It's just hard to know exactly what to make of them. Their normalized individual numbers appear to generally be reasonably decent though. Their data seems to be same ballpark as what others get when they try to do similar measurements, for the ones that are easy for everyone else to do.
It's funny you just write off the entire site's numbers as junk and then talk about egregious lack of critical thinking skills.
wickerprints wrote:
I already told you what was wrong. If you couldn't understand the problem, that's not because I didn't demonstrate it, since other members have understood where I have pointed out the flaws.
If you take measurements of, say, the noise of a sensor, and then take measurements of the dynamic range, you cannot "average" the two measurements together using some arbitrarily chosen weighting function to get an overall score that makes any sense. The units are not even the same. If you can't grasp something as basic as that, we have nothing further to discuss.
Then, as Alan pointed out, you can't even average measurements of something taken at various ISOs, without diluting the meaning of the relative performance of two different sensors, because one may do well at lower ISOs whereas the other may do better at high ISOs. One should at least take into consideration the relative frequency at which one uses different ISOs.
Finally, my point is that if an analysis makes such simple errors for the sake of distilling down their data in a way that a layperson can appreciate, then not only is this an act of intellectual dishonesty, but it calls into suspicion everything they have done. In other words, if I can't trust them to avoid pasting some global "score" on the entire camera, I shouldn't trust whether they were competent enough to properly measure the data they captured in the first place....Show more →
So why not just look at their normalized individual plots You are throwing out the baby with the bathwater and being equally as bad as what you decry!
gonna say NO...unless you only want to misuse a hobby with numbers...instead of pictures,it astounds me people will pay 2k for a lens
based soley on numbers...not an image...bunch of dorks...
skibum5 wrote:
Disagree here. You are not making good sense here. I mean come image they stored a CD as the values 65536 to 131071 instead of 0 to 65535? who cares (other than they'd be wasting a lot of space in this case, although they would have to since you could then just declare that 0 bits set means 65536 and all 16bits set means 65535, who cares?)?
You've misunderstood what I'm getting at here. The issue is not one of how the sensor records the data. The problem with saying "sensor A has 2 stops more DR than sensor B" is that it doesn't really tell us whether that dynamic range is *useful*. That statement doesn't capture, for example, how the sensor might clip the highlights, or whether there's pattern noise that makes recovering information in the shadows useless. I'm not saying it's entirely meaningless, but it's not something that can just be blurted out without qualification, either.
The overall ratings on DxO are close to junk, although not out and out junk. It's just hard to know exactly what to make of them. Their normalized individual numbers appear to generally be reasonably decent though. Their data seems to be same ballpark as what others get when they try to do similar measurements, for the ones that are easy for everyone else to do.
"Appear to be," "reasonable," "hard to know exactly what to make of them," and "seem to be [in] the same ballpark" are not exactly quantitative statements, are they? Perhaps you need to read about Millikan's oil drop experiment to appreciate the significance of what I am saying, and the extreme care with which we must avoid fooling ourselves.
It's funny you just write off the entire site's numbers as junk and then talk about egregious lack of critical thinking skills.
I am NOT saying that DxO's measurements are wrong. That would imply I have arrived at a different set of results. I am saying that their entire methodology is suspect because what I *can* see is done so sloppily, and for the sake of a largely unquestioning audience, that one would be negligent to assume they only slipped up in that one respect. It's not necessarily junk data, but it absolutely is junk science.
I'm really quite perplexed at the eagerness with which some of you are trying to defend DxO's work. Not once have I made any claim of a systematic bias for or against a particular brand of camera based on their data analysis, so if you think that I'm against DxO because they appear to be boosting one manufacturer's results over another, that couldn't be further from the truth.
digitalbug30d wrote:
gonna say NO...unless you only want to misuse a hobby with numbers...instead of pictures,it astounds me people will pay 2k for a lens
based soley on numbers...not an image...bunch of dorks...
I've paid way more than that. I prefer to consider that I'm a bunch of bananas, or maybe grapes (go Habs!).
Here's a graph comparing the dynamic range of the D5100, 550D, and 5D Mark II:
It's interesting to see how well the D5100 performs at under ISO 200. After ISO 200, all three cameras seem to perform similarly. Whether or not this actually means anything is another story...
It's interesting to see how well the D5100 performs at under ISO 200. After ISO 200, all three cameras seem to perform similarly. Whether or not this actually means anything is another story...
Yes it does mean something. It means that if you need to shoot something at base ISO and then push the shadows 4 stops in your RAW converter, the D5100 will come out cleaner than the Canon cameras. Some people need to do this, others don't. You decide for yourself if you do.
Having used all of those cameras in question, yes, the Nikon D700 and D5100 do have a dynamic range advantage, namely that at base ISO read noise is virtual absent, meaning you can push exposure digital afterwards will relative impunity.
any idiot knows dynamic range lowers as the iso goes up...and you know why....wait for it its darker outside...just like you lose colors when its dark..
I guess I am to only one that uses high iso in the dark...DXO would have you thinking iso6400 is just peachy at noon...
if DXO is so smart then they should tell us the DR of a black cat in a coal mine at midnight...
jcolwell wrote:
Also, see Rabbit and Owl count megapixels http://tinyurl.com/cyt5uy
That's fun! I remember all the arguments from a decade ago, but never saw them summarized in this enjoyable parody.
digitalbug30d wrote:
any idiot knows dynamic range lowers as the iso goes up...and you know why....wait for it its darker outside...just like you lose colors when its dark..
I guess I am to only one that uses high iso in the dark...DXO would have you thinking iso6400 is just peachy at noon...
if DXO is so smart then they should tell us the DR of a black cat in a coal mine at midnight...
actually they probably could if they put their mind to it.
wickerprints wrote:
... The lens+sensor is operated as a system...
I would add software to the system. One cannot assume that using a single RAW developer 'levels the playing field'. Some cameras' raw files work better with certain developers than with others.
You are throwing out the baby with the bathwater and being equally as bad as what you decry!
