Lensrentals' Roger Cicala just posted a sample variation test using 70 copies of the Canon 24-70mm f/2.8L II.

Here is an excerpt:
"I think most people realize there is sample variation. But this provides a nice illustration showing just how random a test report comparing just one copy of each lens can be. Don’t get me wrong; those reports are totally worthwhile.
But we have to be really careful splitting hairs with camera lenses. With 70 copies tested, I’m comfortable the average (mean) resolution of the 24-70 Mk II is slightly higher than the 70-200 f/2.8 IS II at 70mm. But that difference is much less than the sample variation...."

Read the entire Canon 24-70 Mk II variation test | Post your review

Canon EF 24-70mm f/2.8L II at: B&H Photo | Amazon.com

Update: TDP just posted Part II his EF 24-70mm f/2.8L II review.

700 to 900? That's not good QC, is it?

Looks like a good result to me.

The range of sample variation in the 24-70 II at 70mm is pretty much the same as the range of sample variation in the 70-200 II. And not many people are complaining about "bad copies" of the 70-200 II.

Secondly, it looks like a "bad copy" of the 24-70 II is still better than the best copy of the original 24-70.

My takeaway from this would be that the lens is a winner, and that generally the complaining about "bad copies" from users can be ignored since the variation isn't any more or less than Canon's other well regarded zooms. To quote Roger Cicala from another thread regarding sample variation on the 24-70 II...


As to the earlier question, for zooms, this is about what most f/2.8 lenses look like. Primes have about half as much variation as a very general rule. Consumer zooms seem to vary less actually, but part of that may be that the absolute numbers are lower (a 10% variation from 500 seems smaller than a 10% variation from 800) and because the apertures are smaller. If I tested these lenses at f/4, the variation would be reduced by nearly half.

Wonderful to see real **data** on which to base decisions. That is so, so rare!

Thank you very much Roger. And to Fred for posting.

This is the type of analysis that should drive every important decision.

Best,
Michael

What Michael said!

That could be a new sideline for Lensrentals, testing and verifying the lens with some type of lensrentals certification. I'm sure people would pay a premium to know that a specific lens is at the top of it's game. Sorry to be a bit off topic, but just thought of that. Re: the topic, it is surprising how much variation there is with that high $ lens.

You have to feel for the person who shelled out $2300 for the new version and got only a little better performance for twice the price. Ouch.

Great to see some hard numbers. This is unfortunately what happens when lenses aren't tested after production. Even expensive lenses (or maybe particularly so for the expensive ones) are not immune to big sample variation--there's just so much that goes into modern AF lenses and so many things that could go wrong. Anything from simple AF miscalibration to element misalignment to possibly other more serious issues could cause the variation.

ggreene wrote:
You have to feel for the person who shelled out $2300 for the new version and got only a little better performance for twice the price. Ouch.

Maybe they had a poor-performing Mk I, and so it's still a signifiant upgrade.

ggreene wrote:
You have to feel for the person who shelled out $2300 for the new version and got only a little better performance for twice the price. Ouch.

Not really. First of all this is just at 70mm so it's not the full picture. Second, I've seen a Mark I at 70mm f/2.8 and I thought it was pretty good, so although the numbers aren't mind-blowingly good, in real life having 700 lppm as a bare minimum for your Mark II lottery sounds like great odds of getting a great lens to me.

Third, which is the takeaway for all of us, this same phenomenon affects all lenses to varying degrees. Let's not get too smug about our own lenses since the law of averages suggest that anybody with multiple lenses is likely to have experienced a bad or at least sub-par copy at least once. It's a shame that with these ever-rising prices they've still left it up to the consumer to figure out if theirs is a bad copy and put the burden on consumers to either microadjust it or send it in for repairs.

I swear my version I is at the bottom of that chart...

So what do you do if you bought a lens you feel is at the bottom of the quality index? Can you send it to Canon CPS and expect they will be able to adjust it properly? My initial tests told me the 24-70 was very sharp at 24mm but not as sharp at 70mm as my 70-200 Mark I, which is a very sharp lens by the way. I have to do some more shooting but just wondering if I have any options.

Yohan Pamudji wrote:
ggreene wrote:
You have to feel for the person who shelled out $2300 for the new version and got only a little better performance for twice the price. Ouch.

Not really. First of all this is just at 70mm so it's not the full picture. Second, I've seen a Mark I at 70mm f/2.8 and I thought it was pretty good, so although the numbers aren't mind-blowingly good, in real life having 700 lppm as a bare minimum for your Mark II lottery sounds like great odds of getting a great lens to me.

Third, which is the takeaway for all of us, this same phenomenon affects all lenses to varying degrees. Let's not get too smug about our own lenses since the law of averages suggest that anybody with multiple lenses is likely to have experienced a bad or at least sub-par copy at least once. It's a shame that with these ever-rising prices they've still left it up to the consumer to figure out if theirs is a bad copy and put the burden on consumers to either microadjust it or send it in for repairs.

Fourth, in normal shooting, especially handheld, you'll rarely if ever achieve the kind of IQ/sharpness that can be obtained in a well-controlled series of tripod/LiveView tests that Roger described. You'll probably also see more variation in "real world" results for multiple shots from the same photographer with the same lens, and between many photographers all using the same lens, than are evident in Roger's test results.

jcolwell wrote:
Fourth, in normal shooting, especially handheld, you'll rarely if ever achieve the kind of IQ/sharpness that can be obtained in a well-controlled series of tripod/LiveView tests that Roger described. You'll probably also see more variation in "real world" results for multiple shots from the same photographer with the same lens, and between many photographers all using the same lens, than are evident in Roger's test results.


Right. A lot of sharpness variation will be concealed by poor technique (raises hand), high ISO noise, etc., and when stopped down a lot of the variation will disappear as well as AF calibration differences are concealed by larger DOF. Overall I think a baseline of 700 lppm at the very worst is an extremely solidly performing lens sharpness-wise. As a point of comparison the venerable 35L is only between 500-600 lppm wide open, and I consider that lens quite good wide open. Number jockeys will have a conniption fit, but the rest of us are safe to happily continue using these lenses.

I want to add my opinion on the real world results varying a lot more than controlled lab test results, and I'm the one who lives in the lab.

But outside the lab, by 50-year-old hand tremor, aggravated by a caffeine intake rarely seen outside the computer programming industry, makes a bigger difference then any lens sample variation ever could. (Roger's exposure time rule is 1/ about 4 focal lengths)

On a tripod, with live view focusing and timed or cable release my photograph results tend to be like my lab results largely. In any other situation, nope, not at all.

Gotta love tests done "Roger" style...meaning taking hundreds of copies for an Imatest ride!
Now we got spoiled and want variation tests done for all lenses!

But when are getting that batch of 70 500 and 600 mk II lenses Roger? I want to make sure I'll get a good one!

We want to send our lenses to roger to test! ... I suppose that is a reason why people join CPS, don't the clean and checks check for sharpness of the lens and adjust them to be in-spec?

was the tests all wide opened?

Roger, thanks for that information.

One question it suggests for further investigation is how much of the variation observed is attributable to each lens individually (as a permanent feature), and how much to the spread of results you might get from a single lens if it were re-zoomed and re-focussed between each of multiple tests. I realize that that might turn into a test of the ability of the person doing the focusing to repeat the setting. Maybe it would be interesting, however, to let the lens do the focusing. Then you would get a "real-world-use" set of results that would add useful context to the lens-to-lens variation results you published (and I note the best-of-bracketed-results aspect of your method; I'm suggesting expanding that a bit, and for at least a few lenses publishing each result as a separate data point).

If on multiple tries any particular lens always grouped tightly, we would know that the question of getting a good copy is more important than it would be if each lens produced a spread of results similar to the one on your graph. Maybe your experience allows you to just answer the general question from your present knowledge?

Just one other question. The subjective quality measurement you use is based on an 8x10 print, and of course some of us print much larger. Would smaller subjective quality variation numbers become more evident in say, a 30x20 print? I'm basically wondering if getting a great copy is more important to a landscape photographer who prints big than to a wedding photographer, for instance.

Thanks again for the terrific work.