skibum5
Offline
• • • • • •
Upload & Sell: Off
|
 p.4 #12 · DxOMark - Is it relevant? | |
wickerprints wrote:
You would have realized this if you had understood why I mentioned the oil drop experiment--but your response below shows you did not and are simply grasping at straws.
Well one of the straws I grasped WAS the one you were going after, how am I supposed to mind read which exact straw, alone, that you wanted? Quicker to just toss out a few than to play games back and forth with you.
Why not? The summary statistics are nonsense. The individual data points are absolutely open to scrutiny. They haven't shown me or anyone else that their raw data is trustworthy, or given anyone a reason to accept them. The fact that I am so skeptical of DxO doesn't imply that I am by contrast "trusting" of other review sites. Again, flawed reasoning on your part.
Because you should know that overall rating is just something that pretty much every single popular site does, it is just standard practice to slap on some overall rating for those who insist on that for whatever reason, and has little relation to the way anything else is done and it is not so hard to look past that and see whether the foundation is solid or not. And my point was not that you re trusting of other review sites but that you basically won't be able to accept anything unless it is published to the levels of a scientific journal and that you are reading much too much into the fact that they presented and overall number.
It doesn't hurt to be practical and take what we are given and try to make the most out of it. With just a quick look one can see that the overall rating on DxOMark is silly for the most part but that the individual plots appear to hold lots of promise and are something worthy of at least looking into.
Look at the fact that others have taken that overall number (that they supposedly don't pretend means anything) and arrive at a conclusion of "20% better image quality." That alone means it is being understood incorrectly, and that there is an implied precision that does not actually exist. The proof of the pudding is in the eating--if people are thinking "Camera A got a 78 but B got a 75, therefore A is better," then nobody needs to prove that DxO is being willfully misleading or posing as scientists--the damage is done.
Yes so that site does silly things but what does that have to do with the DxO plots?
I've decried the overall rating on DxO Mark myself plenty of times before on various forums. They are not posing their overall rating as some scientific construct although I'm sure it is misleading but all the same you need to be able to separate that out and not read too much into what they were trying to do there and then automatically entering into everything else with extra doubt because of that. And plenty of times the most serious works have been cited in wacky ways and that is on the wacky user not on the work itself.
You know about the wrong value for air viscosity that Millikan used, but that wasn't the crux of the story. As Feynman pointed out, subsequent experiments deviated from his incorrect value in a line that trended toward the true value. The key here is the fact that the error wasn't resolved immediately, but only over time, which indicated that the researchers were operating under the misapprehension that Millikan's value was correct, and when their results weren't in agreement, they would try to find reasons for that difference. Had the researchers been truly unbiased, the error in the obtained values would have been randomly distributed around the true value, and this is not the case.
...Show more →
yes and I mentioned all of that, in brief form (along with the air viscosity point, heaven forbid)
This is why one must be extremely cautious about using other people's data to support your own, or vice versa, because you open yourself up to confirmation bias. The oil drop experiment is a bit of an embarrassment in the history of science--it is a lesson in seeing the truth for what it is, rather than what others found.
No one that I know tried to come at it from the perspective of trying to prove DxOMark correct and one person actually had seemed to be negatively biased towards them, if anything. I don't know anyone who was saying they were taking DxO as a given and trying to replicate their results or who had any particularly expectation about anything going in. Nobody was going around trying to repeatedly scan over the RAW file and collect data and re-plot and go back again and re-plot just trying to find the little boxes that would give a reading that would give closest match to the DxO data. Many people had already made some of their plots before DxO data had been released too.
I really don't think this is something you need worry about in this case. Making DR plots is pretty basic and I don't know anyone who would have wanted to sit around wasting hours of time recollecting again and again to try to best fit DxOMark. Every proudly announced any difference they found. Here and there, there were discrepancies and I certainly shouted about my different results for part of the very high iso findings for one camera. Body to body, especially with 7D and 20D, there were differences. And for the people who chose to avg over only one small fixed block they could by chance hit on a bad patch for one camera. I'm sure the DxO results are not perfect and that if they sampled 100 copies across different manufacturing dates and did full sensor averages things might slightly shift but nothing jumps out with their plots as being particularly off, the general relative trends seem to be pretty solid.
It's not like the whole cold fusion debacle either where a number of schools quickly reported that they had replicated it and it was almost treated as an embarrassment for the schools not able to quickly replicate the findings (I vaguely that it was embarrassing that one Boston area school had not been able to replicate it.... well at least until it turned out that whole cold fusion thing was proven to be junk and they were the one who got the experiment correct).
One thing to keep in mind though is that while everyone fights over SNR and DR, the values, aside from low ISO DR for the new SONY type, within the same sensor size have generally not been much to bother about one way of the other for any of the later generation cameras (early on Canon did have sizable and realistically very noticeable advantage) and stuff like pattern banding, which gets laughed at, would actually be more practicably noticeable in real world usage than minor SNR/engineering DR differences (although, as stated, with the new SONY design and the linear plot, that can make a noticeably difference in usage too) which get all the attention.
Again, that is not to say the DxO data is invalid. Nor should it be taken to mean that the methodology of measurement is wrong. Skepticism is not synonymous with dismissal. It simply means, I don't see enough evidence to believe what they've written, in light of the dubious nature of their analysis. I need more convincing, and showing others' data is not necessarily the way to do it.
You are putting way to much weight on the fact they also presented an overall rating and trying to make it sound like that their was the height and end and be all of their analysis, you are reading too much into it, it was just a somewhat silly overall number they made up, as they state, trying to combine all aspects at once into an overall general purpose jack of all trades number (leaving out resolution though) for those who insist on some overall number. Sure there are plenty who will just see than number and run with it and yeah it's probably better if they didn't publish the overall number but they probably, perhaps foolishly all the same, felt they had to since everybody else does it. But you are putting too much weight into the fact that they decided to also present and overall jack of all trades number and imaging that they think and consider that number to be some highly scientific result of theirs.
Edited on Apr 26, 2011 at 07:33 PM · View previous versions
|
so Canon maybe hasn't seen enough pressure on them as of yet to bother going through the pain of design change or simply hasn't been able to come up with a design to be able to do so yet. There has also been some speculation that it might be harder to make the SONY design work well for video, since all the first nikons with that design lacked video while the other ones had it. It will be interesting to see what the next gen does.