I like DxO but that is not to say I put all my faith in there results. I think it is good to have different independent testing done on lenses. I like to look at multiple review sites and draw my own conclusion on how a lens is going to preform or not before deciding if it is something I want to invest in. TDP,lens tip and DxO are some of best IMO as it is not easy to sort through some of the canon cheerleader reviews. Most new lenses are better than the lenses they replace so it is easy for a reviewer to say it is better but I like to understand what a lens is really going to perform like strengths and weakness included.
Very interesting to see the feild curvature results at 35mm. I took a number of aerial photos in Iceland at 35mm and the corner sharpness was poor. I now realise this is feild curvature. I'll have to test this properly...
johnctharp wrote:
If you are looking at DxOMark results and you are NOT looking at Measurements -> Sharpness -> Field Maps to compare sharpness across the frame at various focal lengths and apertures, you are doing it wrong.
I agree, DXO is very extensive, and I love there charts, you can easily tell the behavior of a lens. Nevermind their overall scores, look at the measurements, that helps a lot more.
I think it's also pertinent to remember with 16mm lenses, DxO testing is done at a few feet shooting distance - I think about 4 or 5 feet. One of the main reasons I moved to an optical bench from Imatest was that I just didn't trust the results on lenses wider than 24mm. They all seemed rather similar even though what I saw in photographs showed them to be rather different. I always felt that testing system limitations started creeping in when we were working so close to the test charts.
For full disclosure, though, I seem to be in a very small minority (basically the group of me) in that opinion. Although I'd love to hear what Toothwalker might say concerning that. He's more knowledgeable in that area than I am.
RCicala wrote:
For full disclosure, though, I seem to be in a very small minority (basically the group of me) in that opinion. Although I'd love to hear what Toothwalker might say concerning that. He's more knowledgeable in that area than I am.
I've found that proponents of Zeiss lenses on this forum (as well as some Leica users) expound heavily on the qualities of lenses as they vary at different focal distances. I believe that not testing at different focal distances is one of the primary weaknesses of mainstream testing houses such as DxO, PZ, TDP, LT, etc.
alundeb wrote:
If you take the time to read how the score is calculated, you will see that the T-stop is included in the score. Meaning that a faster lens with the same sharpness and aberrations as a slower lens will get higher score. You may agree or disagree with the usefulness of this, but it does not take away a cent from the usefulness of their individual measurements or the credibility of the site.
RCicala wrote:
I think it's also pertinent to remember with 16mm lenses, DxO testing is done at a few feet shooting distance - I think about 4 or 5 feet. One of the main reasons I moved to an optical bench from Imatest was that I just didn't trust the results on lenses wider than 24mm. They all seemed rather similar even though what I saw in photographs showed them to be rather different. I always felt that testing system limitations started creeping in when we were working so close to the test charts.
For full disclosure, though, I seem to be in a very small minority (basically the group of me) in that opinion. Although I'd love to hear what Toothwalker might say concerning that. He's more knowledgeable in that area than I am. ...Show more →
I'd tend to agree. Some lenses perform noticeably different as they get anywhere near the MFD zone at all compared to say 15' and out.
I would second what Dan said here. I also, like Jim, have tested the lens extensively in test patterns {brick walls 15 feet away) as well as real landscape scenes, and the lens simply is a great improvement over all zoom ultrawide offerings from canon, esp so if you like shooting bellow f5.6, and you care about IS, but even so at certain focal lenghts and f11.
What I found further to be the case, is I can shoot the lens at f8, and within limitation of dof, end up with excellent resolution and contrast throughout the frame and focal lengths.
I am also in the camp to just not trust the results dxo is getting. for example, at 35mm, it shows the lens to do worse and worse in the corners as you stop down. this is simply not the case
Finally, both filed curvature, and near vs far focus performance plays an important role in resolution figures, which I have discovered when testing a lot of lenses. for example comparing two excellent lenses, 24tseII and canon 24-70II, you get mixed opinions of which one is actually sharper in the corners. wide open, the tse is phenominal in the near corners, and very good in the far. the canon is the opposite. even in a flat field test, if you are shooting a target 5 feet away, tse would win. object 50 yards away, zoom wins in the corners.
bottom line, the lens is great. If you dont need f2.8 it is the best canon ultrawide zoom lens, and I would say better than even my nikon 14-24 was when you calculate in flare, CA, range, ease of use of filter, etc. the lens is simply excellent.
skibum5 wrote:
Sometimes they compare f/1.4 to f/2 to f/2.8! Sometimes they compare a zoom at 50mm to another at 24mm!
They have some zooms getting worse and worse at the edges as you step down from wide open.
etc. etc.
I have trouble taking them seriously for lens reviews.
EVen here they rate the the 17-40L better at f/5.6 at 17mm than the 16-35 II or 16-35 f/4 IS (with the 16-35 II far, far behind the 17-40L) etc. etc.
Regarding the 16-35II getting worse in the corners stopped down, it is not unlikely for a single copy. I have seen similar behaviour in visual tests for some copies of both that lens and the 70-200 2.8 L IS II.
All test sites have some oddities. Just because the results for the canon 600 F4 L IS (mk I) are pretty soft at TDP, I don't distrust that site entirely. I don't trust it entirely either. We need mulitple sources and testing of multiple copies.
RCicala wrote:
I think it's also pertinent to remember with 16mm lenses, DxO testing is done at a few feet shooting distance - I think about 4 or 5 feet. One of the main reasons I moved to an optical bench from Imatest was that I just didn't trust the results on lenses wider than 24mm. They all seemed rather similar even though what I saw in photographs showed them to be rather different. I always felt that testing system limitations started creeping in when we were working so close to the test charts.
For full disclosure, though, I seem to be in a very small minority (basically the group of me) in that opinion. Although I'd love to hear what Toothwalker might say concerning that. He's more knowledgeable in that area than I am. ...Show more →
I don't think the minority is that small
Thank you for providing 50 lp/mm data in your test results. 30 lp/mm is not adequate for high resolution sensors anymore. is it possible to do 60 or even 80? personally I would prefer a 10, 20, 40, 80 presentation.
RCicala wrote:
I think it's also pertinent to remember with 16mm lenses, DxO testing is done at a few feet shooting distance - I think about 4 or 5 feet. One of the main reasons I moved to an optical bench from Imatest was that I just didn't trust the results on lenses wider than 24mm. They all seemed rather similar even though what I saw in photographs showed them to be rather different. I always felt that testing system limitations started creeping in when we were working so close to the test charts.
For full disclosure, though, I seem to be in a very small minority (basically the group of me) in that opinion. Although I'd love to hear what Toothwalker might say concerning that. He's more knowledgeable in that area than I am. ...Show more →
You are not alone in your opinions on this matter!
skibum5 wrote:
I have trouble taking them seriously for lens reviews.
+1
I am honestly shocked that anyone actually takes DXOMark results for lens testing seriously.
Weren't they the same folks who claimed the 70-200 f/2.8 Mk2 results were poorer than the predecessor... but back-tracked and changed their measurement results secretly after a huge outcry from the masses?
gdanmitchell wrote:
DXO has the veneer of objectivity and, as a result, gets a lot of attention. But the context is so complex and often of questionable relevance to real photography that the results often do not mean what the "values" assigned to lenses seem to suggest.
A simple test is to make some photographs with the lenses in question (likely the 16-35 f/2.8 L II, 17-40 f/4 L, and 16-35mm f/4 L IS) and look at photographic results. In terms of the real world effect of choosing among these lenses, this will be a lot clearer and straightforward than trying to sleuth out what the DXO numbers do and do not mean.
alundeb wrote:
If you make one error in you life, I will backtrack you forever and claim that you have zero credibility and be shocked if anyone takes you seriously.
But it wasn't just the 70-200 2.8 II being the worst of all Canon 70-200s at 200mm f/2.8
it was the 70-300 non-L being better than the 70-300L being better than the 300 f/4L at 300mm
and it was the 16-35 f/2.8 II having peak corner sharpness on FF at f/2.8
and it was the 24-70 f/4 IS performing much more poorly compared to some other lenses than it did on other tests or my own tests
and it is now the 17-40L being radically better at 17mm f/5.6 than the 16-35mm at 16mm f/5.6 and better than the 16-35 f/4 IS at 16mm f/5.6
and the 16-35 f/4 IS doing worse and worse at FF edges as you stop down at one focal length
and I seem to recall something that seemed questionable with their 50 1.4 test
To obtain meaningful results in the digital age would require a "state of the art", "super" high resolution test camera modified, or designed to accept "all" lenses.
It wasn't meant literally. I was just mirroring what someone was doing to DxO. It is so easy to find mistakes made by someone, and claim that they cannot be trusted. Everybody makes mistakes, and many improve further down the road. IMO the test from DxO have better consistency with other independent test lately.
.
