Just finished reading great article on lens variations from R. Ciala, owner of LensRentals. He looks at performance variations of various copies of the same lens, performance variations of a single lens on various cameras, there is also a nice chart comparing resolution of various copies of 100MP and Canon 100mm macro.
Very interesting indeed. There is one more parameter that would be interesting to see and that is how the variation looks over focusing distance. The lens elements move around when you change focus and the resolving power also varies by distance. The 100MP vs 100L is a good case in point - the latter resolves better at close focus (as shown by imatest results) while the former resolves better at infinity. There's bound to be variation between the individual copies as you go between MFD and infinity.
Another interesting thing would be to see how micro contrast varies over the individual copies. Imatest uses MTF50 as standard IIRC which is OK, but just a single point. It would be really interesting the full MTF charts (although it would require an MTF machine).
Anyway, kudos to Cicala and and Lensrentals for actually making a quantitative analysis.
Perhaps I missed this in the article on my quick scan, but do they mention the aperture at which those data plots were created?
Thank you!
Those were all wide open (f/2.8).
That's a good point about distance, Denoir: with the testing set up we have I can test at 3 different distances (at 100mm it would be abou 6 feet minimum, 12 feet, and 15 feet). The test mentioned above was at 12 feet.
However, the Imatest data is for 32 points over the field of view, and I graphed it as peak MTF50 (single point) vs weighted average MTF50 of the entire FOV. A full-field MTF analysis bench would be more accurate (and eliminate camera variability). That's in my dreams, but with the automated benches going for >$100,000 my dreams is where it will stay for quite a while.
The IQ variation in the test explains why sometimes people say lens A is better than lens B while my own experience is opposite. I was a bit surprised to see similar sample variation with the Zeiss - it appears that the standards are not better than Canon's.
I think another interesting data point would be lens variation in Leica M lenses. Do you have any experience or even test results.
Your tests are very valuable, thanks for posting them!
It's interesting that such a sizable number of copies of the 100L IS outperform the Zeiss MP at f/2.8. At that aperture, Photozone's test copy of the MP seems to outperform their copy of the 100L IS (in terms of resolution).
More interestingly, as the aperture is closed down a bit, the lead that the MP takes is not subtle at all, especially in the corners -- to the point that by f/4 or f/5.6, I'd expect that the resolution edge of the MP over the L-IS might out-weigh statistical differences between copies, barring outliers. Do Photozone test at a different distance?
Thanks. I also meant to ask: the Zeiss opens to f/2. Were the Zeiss samples tested wide-open, or at f/2.8?
kwalsh wrote:
I'd be surprised if the measurement error was much better than 10%.
I agree. If it were around the 10% mark (and not too much greater), the ~20% higher resolution numbers from the Zeiss, especially at corners at f/4 and f/5.6, are significant.
kwalsh wrote:
Yeah, but those can be just as misleading if there are systematic errors.
Ken
Photozone typically include a disclaimer about not comparing across mounts. Comparing only lenses that were tested with the same test equipment under largely similar conditions takes care of most of the issue of systematic errors. Which means that accuracy (lp/mm numbers) is probably less important to me than precision, in this regard.
The Zeiss were tested at f/2.0 compared to the Canon f/2.8 so there definitely was reason for the difference in my groupings. We test everything at it's widest aperture because our primary purpose is to detect bad lenses - they're much more obvious at wider apertures.
Since we test every copy of every lens repeatedly over time, I can comment on accuracy / reproducibility: As long as the same test camera is used (meaning serial number xxxxx, not just a 5DII or whatever) the data is repeatable +/- 2 to 4% when testing the lens on completely different days (which means the setup has been broken down and set up for that focal length again). Change camera copies though and the variation is greater than 10%.
The comment above about not comparing across brands and across camera is very pertinent. We us 5DII for Canon and D3x for Nikon. We wanted to used D3s for Nikon simply because of cost, but from a pure resolution standpoint the best f/2 and f/2.8 lenses were outresolving the D3s. (Since we're testing wide open, wide aperture lenses and of course lower quality lenses didn't.)
