p.1 #1 · Why do people put so much trust into DXOMark's ratings when...
You look at the lens ratings?
Something look a bit off here? (Some key-players highlighted...)
So if the lens resolution numbers can be so wrong... How can you look at their camera reviews with any more faith? If anything, lens resolution should be easier to test then an imaging sensor.
Just trying to make people realize that their testing is not "truth" in the real world. Seriously, these quantified results are hogwash, and that is put lightly. You think the camera sensor results are any better?
p.1 #2 · Why do people put so much trust into DXOMark's ratings when...
Because the lens tests may be done by different people and they are much trickier to do well and you can be bad at one thing and good at another. And lenses differ more (although 7D sensors did differ quite a bit copy to copy as per vertical gain banding). The TINIEST focusing error can totally alter results. Even using 10x liveview I found you really need at least six tries and sometimes more for reach sample to start getting a reliable results. That takes forever, they may only do best of one or two or three for all we know. It also depends how far the target is. Maybe they test near MFD. SOme great lenses go bad there. Maybe they refocus for edges maybe they don't so field curvature can play a part too. Etc.
And particularly because, unlike their lens tests, their sensor tests match so well to what so many other people have measured. I mean I got like to the 1/10th of a stop same ISO100 DR numbers they did for 5D2 and 5D3.
p.1 #5 · Why do people put so much trust into DXOMark's ratings when...
There are many things that DXOMark gets wrong (e.g., they play down differences in high ISO performance while any tiny difference in low ISO performance is severely heightened, they have their own definition of ISO that is useful for sensor comparison but has no relevance in real world usage etc etc), but they do get some things right. In this regard, they are heavily biased. IMO, to compare sensors, the best test is to take 2 cameras of interest, shoot them side-by-side under controlled conditions and compare. To me, that is real world usage.
Photozone often complains about de-centered lenses and show how they seriously affect measurements. Does DXOMark even know what de-centering in lenses mean?
p.1 #6 · Why do people put so much trust into DXOMark's ratings when...
I look at DXOMark as a good talking point but at the end of the day will it determine your switching to a different system, will DXOMark magically increase your gear budget and what about unique focal lengths that have no direct equivalents?
p.1 #7 · Why do people put so much trust into DXOMark's ratings when...
Those lens results are so far off the mark it makes my head spin. Any defense of the 300f2.8 II being bettered by a 24-105 automatically makes me think the rest of their data is out the door.
I don't even care if these data are truly representative of performance at optimum aperture (which I doubt), the conclusion is so grossly wrong in the real world as to be useless.
p.1 #8 · Why do people put so much trust into DXOMark's ratings when...
I think the consensus on FM is that the sensor tests seems much closer to real world experience and the lens tests are no where close. If you look at the references to DXOmark on FM it will always be in regards to sensors.
p.1 #11 · Why do people put so much trust into DXOMark's ratings when...
John Shultz wrote:
Those lens results are so far off the mark it makes my head spin. Any defense of the 300f2.8 II being bettered by a 24-105 automatically makes me think the rest of their data is out the door.
I don't even care if these data are truly representative of performance at optimum aperture (which I doubt), the conclusion is so grossly wrong in the real world as to be useless.
+1000.
Any company which rates the 24-105 f/4L as optically superior to the 300mm f/2.8L is absolutely right!
The IQ at 24mm for the 300mm prime lens is abysmal...
Seriously, they have lost ALL credibility with me if they come to THOSE types of conclusions!
p.1 #12 · Why do people put so much trust into DXOMark's ratings when...
DXO has only one (BIG) problem : their completely ridiculous rating system for lenses AND sensor.
They may make some good individual tests that can be taken seriously if viewed individually, but anyone believing that a d5100 as 1 point more than a 5d mkII and ranked higher than a medium format camera is completly rejecting 100 years of physics research regarding photon noise (which is the predominant noise in an image may I reming you). I mean a MF collects more than 5 times more photons than an aps-c, so unless the d5100 has a QE of 50% vs 10% for the ME
Their rating system is just badly conceived and the problem is that this is the No1 criteria they advertised.
So if the lens resolution numbers can be so wrong... How can you look at their camera reviews with any more faith? If anything, lens resolution should be easier to test then an imaging sensor.
Just trying to make people realize that their testing is not "truth" in the real world. Seriously, these quantified results are hogwash, and that is put lightly. You think the camera sensor results are any better?
So you know I'm not just whimpering because some Canon gear ended up with a lower rating than some Nikon gear, I've been saying for quite a while that you really have to look closely at the basis for the DXO "ratings." A lot of people glom onto the fact that some thing gets a "higher number" than some other thing and therefore "ranks higher" and presume that, because numbers are involved, that this must be evidence-based and incontrovertible and unarguable.
Not quite.
For a long time - in cases where Canon stuff ranks high and in cases where it ranks low and in cases having nothing to do with Canon - it is has been apparent that whatever objective stuff these values represent, they has very little if anything to connect them to actual photographic results.
Again, I have no ax to grind relative to brand or model of camera. As I've been pointing our recently, there are multiple brands making multiple models for really great gear right now and it is hard to go wrong if you make a need-appropriate choice from among them, probably continuing to use whatever brand you currently use.
p.1 #14 · Why do people put so much trust into DXOMark's ratings when...
skibum5 wrote:
Because the lens tests may be done by different people and they are much trickier to do well and you can be bad at one thing and good at another. And lenses differ more (although 7D sensors did differ quite a bit copy to copy as per vertical gain banding). The TINIEST focusing error can totally alter results. Even using 10x liveview I found you really need at least six tries and sometimes more for reach sample to start getting a reliable results. That takes forever, they may only do best of one or two or three for all we know. It also depends how far the target is. Maybe they test near MFD. SOme great lenses go bad there. Maybe they refocus for edges maybe they don't so field curvature can play a part too. Etc.
And particularly because, unlike their lens tests, their sensor tests match so well to what so many other people have measured. I mean I got like to the 1/10th of a stop same ISO100 DR numbers they did for 5D2 and 5D3. ...Show more →
Yes - the larger the magnifcation the tougher the testing because of the 1/mm shake issue and even more importantly - between the tolerance of the lens mount and the tolerance of the camera mount, the lens has to be micro-adjusted, to do a fair test. And keep in mind that 24-105 vs 70-200, is comparing the sharpest spot on each lens. And further, you can't test a lens without a camera body, so comparing Nikon lens to Canon lens, you have to take the best vs best and if the best canon is better than the best Nikon, the Nikon lens will be underrated. Further the best lens on a ff is not the best on a crop, so you have to take this into account.
So comparing 24-105 to 70-200 - 70-200 would be more effected by micro misalignments, lens mount misalignments.
However, in defense of Dxomark, they are testing where other raters are just providing subjective remarks. I find their data useful. But I verify visuall with thedigitalpicture iso charts.
p.1 #15 · Why do people put so much trust into DXOMark's ratings when...
For the cameras I've owned, their rating system seems to be spot on. As others have stated, it's a starting point for comparison, nothing more. The only people who actually take them seriously are people who want something to talk about on message boards.
p.1 #16 · Why do people put so much trust into DXOMark's ratings when...
skibum5 wrote:
Because the lens tests may be done by different people and they are much trickier to do well and you can be bad at one thing and good at another. And lenses differ more (although 7D sensors did differ quite a bit copy to copy as per vertical gain banding). The TINIEST focusing error can totally alter results. Even using 10x liveview I found you really need at least six tries and sometimes more for reach sample to start getting a reliable results. That takes forever, they may only do best of one or two or three for all we know. It also depends how far the target is. Maybe they test near MFD. SOme great lenses go bad there. Maybe they refocus for edges maybe they don't so field curvature can play a part too. Etc.
And particularly because, unlike their lens tests, their sensor tests match so well to what so many other people have measured. I mean I got like to the 1/10th of a stop same ISO100 DR numbers they did for 5D2 and 5D3. ...Show more →
RAW images don't come out of nowhere without lenses, unless they are lens-cap-on tests.
In particular, the DXOMark sensor tests seem to require focusing on various color targets. Won't lack of accurate focus affect the results? E.g., if a target is out of focus, its image on the camera sensor will be smeared, won't that affect the mean and standard deviation that is computed of values in the RAW file?
p.1 #17 · Why do people put so much trust into DXOMark's ratings when...
Arun Gupta wrote:
RAW images don't come out of nowhere without lenses, unless they are lens-cap-on tests.
In particular, the DXOMark sensor tests seem to require focusing on various color targets. Won't lack of accurate focus affect the results? E.g., if a target is out of focus, its image on the camera sensor will be smeared, won't that affect the mean and standard deviation that is computed of values in the RAW file?
You would need to be using a catastrophically bad lens for it to start affecting mean and SD values for colour patches, unless the patches were miniscule.
p.1 #19 · Why do people put so much trust into DXOMark's ratings when...
Correct or wrong
1. The Red, Green, Blue primaries on the camera sensor don't correspond to the R,G,B primaries of the sRGB color space. Or the R,G,B primaries of the Adobe RGB color space.
2. The transformation from camera sensor primaries to sRGB primaries introduces some noise (I'm unclear as to the mechanism, perhaps primarily from having to subtract quantities in transformation).
3. The transformation from camera RGB to Adobe RGB is a different transformation than for camera RGB to sRGB.
4. Nevertheless, the noise introduced by either transformation is pretty much the same. So the color space in which one does measurements does not affect color sensitivity measurements.
.