Home · Register · Join Upload & Sell

Moderated by: Fred Miranda
Username  

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | Canon Forum | Join Upload & Sell

1
       2       end
  

Archive 2012 · Do you think DxOMark is accurate?

  
 
dsr1
Offline
• •
Upload & Sell: Off
p.1 #1 · p.1 #1 · Do you think DxOMark is accurate?


I've been checking out DxOMark before deciding on lenses and these new Sigma II lenses are out doing a lot of Canon L lenses. The 70-200 2.8 L II IS is one and the 17-50 2.8 IS is another. I've always used their results and wonder what others think.

http://www.dxomark.com/index.php/Lenses/Camera-Lens-Database

Tell me what you think of DxOMark's testing.

DonR



Feb 02, 2012 at 01:13 PM
Jeff Nolten
Offline
• • • •
Upload & Sell: Off
p.1 #2 · p.1 #2 · Do you think DxOMark is accurate?


For lenses, I prefer the digital picture .com and the reviews and scores here on FM. I find the DxO lens testing difficult to interpret but I admit I haven't spent much time at it.

For sensors, I find their results correspond pretty well with my experience with the cameras I've had, so I trust them for cameras I don't have. But, you have to understand what they are testing and specifically what they mean by screen vs print to validly interpret their results.



Feb 02, 2012 at 01:22 PM
skibum5
Offline
• • • • • •
Upload & Sell: Off
p.1 #3 · p.1 #3 · Do you think DxOMark is accurate?


dsr1 wrote:
I've been checking out DxOMark before deciding on lenses and these new Sigma II lenses are out doing a lot of Canon L lenses. The 70-200 2.8 L II IS is one and the 17-50 2.8 IS is another. I've always used their results and wonder what others think.

http://www.dxomark.com/index.php/Lenses/Camera-Lens-Database

Tell me what you think of DxOMark's testing.

DonR



For lenses I prefer photozone.de most and DxO least. Photozone.de matches what I've seen more often than TDP which also has a lot of stuff that just hasn't matched my careful tests, although it still seems to be a lot more reasonable than DxO.

DxO has a lot of weird stuff such as:

at 200mm and wide open:
70-200 2.8 IS > 70-200 2.8 non-IS > 70-200 2.8 IS II

at 300mm:
70-300 IS non-L > 70-300 IS L (i think they may have even had the 300 f/4 non-IS L in last place there)

16-35L having sharper edge performance at f/2.8 than f/5.6 or something like that

and so on

I don't get it.

Their specific camera sensor data plots (not overall sensor ratings) seem to generally be pretty reasonable though.




Feb 02, 2012 at 02:32 PM
digitalbug30d
Offline
• • • •
Upload & Sell: Off
p.1 #4 · p.1 #4 · Do you think DxOMark is accurate?


charts and graphs turn me off...


Feb 02, 2012 at 05:23 PM
kwalsh
Offline
• • • •
Upload & Sell: Off
p.1 #5 · p.1 #5 · Do you think DxOMark is accurate?


I don't think many people consider the DxO lens tests to be very representative of reality when it comes to using a lens. Photozone, SLRGear and LensTip all do a better job but each of these relatively close focus test chart methodologies have short comings. Check out the recent very lengthy thread in the Alt. forum on lens test sites. Also note this reference:

http://toothwalker.org/optics/lenstest.html

A nice overview of the various issues with such tests.

Ken



Feb 02, 2012 at 05:35 PM
RobDickinson
Offline
• • • •
Upload & Sell: Off
p.1 #6 · p.1 #6 · Do you think DxOMark is accurate?


DXO for lenses seems to be utterly out of whack. Not sure what they base their scoring on but some comparison so some lenses are just nuts (say canons 50's)


Feb 02, 2012 at 06:28 PM
Pixel Perfect
Offline
• • • • • • •
Upload & Sell: On
p.1 #7 · p.1 #7 · Do you think DxOMark is accurate?


I prefer to rely on real life images and user feedback. I've never checked DxO or digitalpicture or photozone for lens results. I think half the time they are rubbish and bear no relation to reality.


Feb 02, 2012 at 07:32 PM
timbop
Offline
• • • • •
Upload & Sell: Off
p.1 #8 · p.1 #8 · Do you think DxOMark is accurate?


dsr1 wrote:
I've been checking out DxOMark before deciding on lenses and these new Sigma II lenses are out doing a lot of Canon L lenses. The 70-200 2.8 L II IS is one and the 17-50 2.8 IS is another. I've always used their results and wonder what others think.

http://www.dxomark.com/index.php/Lenses/Camera-Lens-Database

Tell me what you think of DxOMark's testing.

DonR

crap



Feb 02, 2012 at 07:35 PM
dsr1
Offline
• •
Upload & Sell: Off
p.1 #9 · p.1 #9 · Do you think DxOMark is accurate?


Thanks for the replys everybody.

When I was deciding between Canon & Nikon I Compared lens on Lenstips, PhotoZone, SLR Gear and DxOMark and was surprised that they agreed almost every time. Each had their own grading system but when you put each class of lens together they were almost 100% agreed on the placement of brands from 1-4 on the IQ, build quality and overall quality of each lens. Of courrse there's always the difference from copy to copy but it at least kept me away from some and pointed out other lenses to considers.

I jusst wondered what everyone else thought.



Feb 02, 2012 at 08:07 PM
skibum5
Offline
• • • • • •
Upload & Sell: Off
p.1 #10 · p.1 #10 · Do you think DxOMark is accurate?


dsr1 wrote:
Thanks for the replys everybody.

When I was deciding between Canon & Nikon I Compared lens on Lenstips, PhotoZone, SLR Gear and DxOMark and was surprised that they agreed almost every time. Each had their own grading system but when you put each class of lens together they were almost 100% agreed on the placement of brands from 1-4 on the IQ, build quality and overall quality of each lens. Of courrse there's always the difference from copy to copy but it at least kept me away from some and pointed out other lenses to considers.

I jusst wondered what everyone
...Show more

i've actually found lots of disagreement between them

lens testing is tricky, i found that it takes at LEAST six manual focus 10x zoom liveview attempts, if not a dozen, to begin to get something valid and you need to use stable indoor lighting, that is a real pain and takes a lonnnnng time and I wonder how many sites do that

also some sites focus once in the center while others refocus for edge and corner, the former means they also need to be utterly parallel which is very tricky to do, insanely tricky and field curvature changes by distance, in some real world photography the tests leaving FC in play might be a better truth teller but in other cases not at all

some sites appear to use test chart only a few feet away, others try for 20-25' away

copy to copy variation can exist too

that said if you look at many sites and weight those you agree with more and also look at reviews and some 100% crop samples and tons of user comments you can often get a sense of things

for my tests I prepared images and eye-balled it instead of MTF since those programs can be touchy and what do you pick, which MTF type? I just try eye-ball which overall gives best micro-contrast impression when testing for sharpness (and i've only been truly careful a few times since it takes too much time)



Feb 02, 2012 at 08:13 PM
Mike K
Offline
• • • •
Upload & Sell: Off
p.1 #11 · p.1 #11 · Do you think DxOMark is accurate?


skibum5 wrote:
lens testing is tricky,
also some sites focus once in the center while others refocus for edge and corner, the former means they also need to be utterly parallel which is very tricky to do, insanely tricky and field curvature changes by distance, in some real world photography the tests leaving FC in play might be a better truth teller but in other cases not at all

some sites appear to use test chart only a few feet away, others try for 20-25' away

copy to copy variation can exist too


Agreed that lens testing is extremely tough! As a scientist by training, I can see that there are far too many independent variables to precisely control, that may make one lens measure better than others. Usually no attempt is made to optimize or measure these myriad of variables, or their interaction. The distance dependent field curvature mentioned above, is one such example. Lenstip and TDP use charts at close to minimum focal distance for some of their testing, and many complain that this creates results which do not reflect actual use. Its very difficult to compare different, incompatible brands like Canon and Nikon, since different camera bodies have different sensors, and one cannot separate the lens from the capture and processing of the data. So arbitrary bodies and test conditions are chosen to test different lenses just like arbitrary lenes are used to test different bodies. Its actually a very small design space of probable users situations.

At best, the testers choose one condition (or range) for each variable and can compare various lenses on that same body. The tests are limited, so the interpretation of the data has to also be very limited in scope. There are many ways of measuring performance, so it is meaningless to compare the absolute results of one testing lab to another. There are simply too many combinations of variables to be comprehensive and exacting. While this does not mean lens testing useless, it does mean that one should take any data set with a grain of salt, the scope of the testing is very narrow.

On the copy to copy variation, read this
http://www.lensrentals.com/blog/2011/10/notes-on-lens-and-camera-variation
Statistics and error bars are rarely presented in the realm of lens testing.
Mike K



Feb 02, 2012 at 09:12 PM
dsr1
Offline
• •
Upload & Sell: Off
p.1 #12 · p.1 #12 · Do you think DxOMark is accurate?


I know test are tricky but I don't consider it meaningless to consider each labs results when you're making your decision on which lens to purchase.

Of course they all have different cryteria. That's why you're susposed to read their cryteria and see what value each lab assigns to each test. When I compare lab results, I list each set of lenses on the left. All the short standard f2.8 zooms, then the 15-85 and comparable lenses, then the 70-200 F2.8 IS lenses and then the multipurpose zooms.

I divide the right side of sheet into equal spaces for each labs test results. It's just numbers assigned to how well each lab thinks the lens did. Doesn't mean it's the end all accurate judgement of a lens but at least there's method to their madness and you know that method if you read what everything means and how they tested.

You look at their results, read what others think of the lens and make your own mind. It's just another tool to help make an informed decision. Some use it and some don't, no biggie.

DonR



Feb 03, 2012 at 12:58 AM
gspiridakis
Offline
• •
Upload & Sell: Off
p.1 #13 · p.1 #13 · Do you think DxOMark is accurate?


charts and graphs turn me off...
+1

What about the camera's sensor comparison they do? Do you think is accurate?
I believe that when they compare bodies between one manufacturer they are representative, otherwise the comparison between Canon and Nikon bodies seem to be quite off reality on my view.
Their rating shows a remarkable precedence to Nikon over Canon bodies, and this supriced me!
George



Feb 03, 2012 at 06:30 AM
Scott Stoness
Offline
• • • • • •
Upload & Sell: On
p.1 #14 · p.1 #14 · Do you think DxOMark is accurate?


I like Dxomark:
- They test all lens in a consistent manner
- They have useful categories (resoultion vs distortion etc)

I don't like their weighting that rolls it all into one measure. I rank distortion, resolution differently.

They also rank sensors - I find this very useful because they have objective / quantified comparisons of ISO that are not available elsewhere that I have found

I also like the thedigitalpicture because you can see the difference and then try to reconcile it to the math (Dxomark) which makes for better decisions.

I also like photozone but Dxomark has an easier interface where you can line up lens in a table. Photozone requires lots more work.

Everywhere else I do not find as helpful (even here you really have to judge the advice you are provided because people get emotional about their conclusions and base their views on others views rather than facts).

Other sources I use more for non obvious paradigm challenging questions, like may be you should consider a different camera instead of a different lens.

So Dxomark and the digit picture are my favorites because they provide data instead of opinions.

Take in point a comparsion of Rokino 14 2.8 and Canon 14 2.8. The digital picture clearly shows Canon better but people rank Roikino high based on mixing price and quality. I would rather rank this myself.



Feb 03, 2012 at 02:16 PM
skibum5
Offline
• • • • • •
Upload & Sell: Off
p.1 #15 · p.1 #15 · Do you think DxOMark is accurate?


gspiridakis wrote:
charts and graphs turn me off...
+1

What about the camera's sensor comparison they do? Do you think is accurate?
I believe that when they compare bodies between one manufacturer they are representative, otherwise the comparison between Canon and Nikon bodies seem to be quite off reality on my view.
Their rating shows a remarkable precedence to Nikon over Canon bodies, and this supriced me!
George


Their charts about sensor performance generally seem to be fairly reasonable for the most part. The overall sensor ratings are weird since they try to combine various aspects having nothing to do with each other using weightings that may not have any relevance to you while also giving sensor resolution zero weight.

As I said they have so many bizarre results in their lens tests that I think they don't take the necessary extreme care to make them useful at all. Either that or they have been unlucky enough to get more freak copies of lenses, by chance, than just about anyone ever.

TDP definitely seems more reasonable to me than DxO for lenses, that said a lot of TDP results haven't necessarily matched my own when I've taken careful time to do a very careful test. I think they test lenses kind of close in for one and for sure with their older tests they didn't focus carefully enough (even with new tests occasionally you will see a weird thing where a lens goes sharp f/2.8 to 3.2 and then much worse to 3.5 and then much much better to 4 implying they didn't use nearly enough focus trials to try to find best sample photo. And they also got a very odd rating for CA for one lens, i noticed, they had it having low CA on wide end and heavy on long end, and not a single user report or other testing site matched that finding. All the same it can be useful to take a look there all the same.

I'd always try to look at as many reviews as you can and not take any one as gospel.

SLRGear appears to have a very good procedure and yet they sometimes have weird reports like calling the OLD tamron 70-300 better than the new canon 70-300L and the 70-300 non-L from canon better as well. They didn't even use the same camera body to compare the three lenses and I wonder if they didn't forget that and just compare raw numbers from different versions of test program and different camera bodies!



Feb 03, 2012 at 03:06 PM
digitalbug30d
Offline
• • • •
Upload & Sell: Off
p.1 #16 · p.1 #16 · Do you think DxOMark is accurate?


Pixel Perfect wrote:
I prefer to rely on real life images and user feedback. I've never checked DxO or digitalpicture or photozone for lens results. I think half the time they are rubbish and bear no relation to reality.

yes...^^^^



Feb 04, 2012 at 03:01 AM
skibum5
Offline
• • • • • •
Upload & Sell: Off
p.1 #17 · p.1 #17 · Do you think DxOMark is accurate?


digitalbug30d wrote:
yes...^^^^


the problem with real life images is:

1. someone posts an 800x600 resize of dog poo on the sidewalk at 1/30th on a gray day with a 300 2.8 IS
2. someone posts an 800x600 resize of a glorious sunset in the dolomites on a tripod with mirror lock with the original 18-55 non-IS kit lens


So 18-55 non-IS is a much sharper and more dramatic lens than the 300 2.8 IS right?


or even someone posts a 100% crop the first taken with sun out and then the second with another lens with sun just a trace hidden by clouds and the AF missing a trace this time and the lower contrast from that makes the microcontrast on the bark on the tree look weak from the second lens so does the second lens actually produce images with less pop or not

granted sample images can still be useful, if you look at the right sort and take things into account you can still sometimes get a certain degree of information from them and sometimes it can be clear where they post a 100% crop from the edge that looks better than anything you have ever gotten from your lens at the edge ever, etc.

and, sure, general user opinions can be quite valuable too, if you take enough into account and weight them, etc.



Feb 04, 2012 at 01:17 PM
Jeff Nolten
Offline
• • • •
Upload & Sell: Off
p.1 #18 · p.1 #18 · Do you think DxOMark is accurate?


If I'm trying to make an expensive lens decision I'll try to use every resource available to me including review opinions. One lens measure I've used over the years has been the average ratings here on FM reviews. There is some science behind it. Experiments have shown that if you ask a lot of people how many jelly beans are in a jar, you'll get guesses all over the place, but if you take the average of a lot of guesses you get a fairly good estimate of the number of beans. The reviews and lens scores from individuals tend to be all over the place but the trend, I think, is a pretty close reflection of the lens' value. Of course, most users are taking lens cost into consideration and tend to weight that against build and optical quality. You have to wade through the reviews to get a sense of that. Like DxO, you have to have some sense of the methodology.


Feb 04, 2012 at 01:34 PM
Bruce Sawle
Offline
• • • •
Upload & Sell: On
p.1 #19 · p.1 #19 · Do you think DxOMark is accurate?


Lenstip and photozone all though there results sometimes contradict each other but overall there both good resources.


Feb 04, 2012 at 01:38 PM
skibum5
Offline
• • • • • •
Upload & Sell: Off
p.1 #20 · p.1 #20 · Do you think DxOMark is accurate?


Jeff Nolten wrote:
If I'm trying to make an expensive lens decision I'll try to use every resource available to me including review opinions. One lens measure I've used over the years has been the average ratings here on FM reviews. There is some science behind it. Experiments have shown that if you ask a lot of people how many jelly beans are in a jar, you'll get guesses all over the place, but if you take the average of a lot of guesses you get a fairly good estimate of the number of beans. The reviews and lens scores from individuals
...Show more

That reminds me of the time I won a nice TV by coming closest in a # of jellybeans in a tank contest. A little counting along edges and weighing adjustments and it came quite close to the reality.

Anyway yeah they do say there can often be quite a wisdom in crowd knowledge.




Feb 04, 2012 at 02:16 PM
1
       2       end




FM Forums | Canon Forum | Join Upload & Sell

1
       2       end
    
 

You are not logged in. Login or Register

Username       Or Reset password



This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.