There is of course sample variation for lenses of all brands, but the width of the distribution may vary between brands.
It would be interesting to see what the variation of the test method itself is. That is, the variation that results when the same lens is tested multiple times. Starting from scratch.
I don\'t have a large data sets for that - we test the lenses in rotation so in general a lens tested for the second and third time has been rented several times between tests, which adds the variable of wear-and-tear, either from use or shipping. Our primary goal is not really to look at the small copy-to-copy variability that exists between most lenses, it\'s to find the much larger difference in those few lenses that are bad, either out-of-the-box or from damage in use and get them out of stock. (The difference in these is dramatic and far removed from the pattern of minor copy-to-copy variation).
For what it\'s worth we have found (about 1,500 lenses with repeated tests, although I emphasize weeks or months and several rentals passed between tests) that as long as the same test camera is used a variation of more than + / - 2.5% (best shot of a 5 shot focusing bracket) is fairly normal. With very few exceptions change greater than that is MUCH greater than that and always negative, indicating some damage has happened to the lens.
If we change test cameras, though, the variation is at least double that - even if we change test cameras without making any other changes to the setup, simply changing the camera and running the same lenses again. Similarly, if we lock one lens in place and simply test a whole series of cameras, the variation is about 5% (with the exception of a few outliers - bad cameras happen at least as frequently as bad lenses).
This is one of the shortcomings, as you\'ve very accurately pointed out in your excellent posts, of the Imatest system compared to a bench testing system. On the other hand, it is also good information, knowing that Lens xxx, whatever the manufacturer, will vary about 5% changing from camera to camera.
All that being said, we\'re investing in a bench testing setup, more for speed of testing than anything else. (As Ruahrc\'s post demonstrated, not being able to test lenses after every rental can allow some bad copies to get out - our techs look at them optically after every rental on test charts but that\'s just not as accurate as MTF measurement. Imatest is too time consuming to test every lens every week.) Once it\'s here, the bench will allow us to present the data from large numbers of lenses tested at infinity (which Imatest can\'t do) and without the variable of camera attached. So hopefully we\'ll be able to present some new numbers in a couple of months.
