Home · Register · Join Upload & Sell

Moderated by: Fred Miranda

New fredmiranda.com Mobile Site

New Feature: SMS Notification alert

New Feature: Buy & Sell Watchlist

FM Forums | Leica & Alternative Gear | Join Upload & Sell

2 end

Archive 2013 · Guidelines for comparative lens testing?

genji
Offline
• • • •
Upload & Sell: Off

p.1 #1 · p.1 #1 · Guidelines for comparative lens testing?

When I just counted, there were more than 5000 posts on the various A7/A7R threads, most of which are about the performance of various lenses on these cameras. Despite all the links to various lens tests on the web, there is still very little consensus as to which lenses -- apart from the native FE glass -- will perform well/acceptably/adequately/etc. The problem is compounded by the fact that most of the tests have been less than rigorous and also that what is "acceptable" for one observer is "atrocious" for another.

A number of FMers have expressed the view that the only way to address this issue is to do one's own tests on one's own lenses and this approach -- as implemented by Ron Scheffler and Phillip Reeve (for example) -- has yielded valuable, objective data.

I have a bunch of different lenses and I expect to receive my A7R within the next few days. It occurred to me that it might be a good idea to ask a few questions about how best to structure a comparative test.

* If not infinity, what is the minimum subject distance that will provide useful data for landscape shooters?

* Is 1.5m (~5ft) a good choice for testing a lens at close distance? Or should the focus distance be a fixed multiple of the focal length?

* Should the lens be refocused at each tested aperture to show it at its best? Alternatively, should the focused distance remain constant to reveal any possible focus shift or field curvature? Or would it be best to focus two stops down from wide open?

* Does it matter whether the tests are done in sunlight or open shade?

I'm sure there are lots of other variables to think about so please regard my questions as a means of eliciting other factors that should be taken into consideration...

Nov 23, 2013 at 02:45 AM

JohnJ
Offline
• • • •
Upload & Sell: Off

p.1 #2 · p.1 #2 · Guidelines for comparative lens testing?

I think the most important aspect of testing is 1/ deciding what you want to test and 2/ absolute consistency.

Don't expect to shoot a brick wall or test patterns and know everything about a lens. You'll probably have to shoot a series of tests, preferably comparisons between similar lenses, to learn your lens.

I've written a basic how-to here http://photocornucopia.com/1044.html . It's a start and covers the basic methodology that I think works and is achievable.

For viewing/comparing results I like this kind of thing: http://photocornucopia.com/1055.html

Nov 23, 2013 at 05:00 AM

genji
Offline
• • • •
Upload & Sell: Off

p.1 #3 · p.1 #3 · Guidelines for comparative lens testing?

JohnJ wrote:
I think the most important aspect of testing is 1/ deciding what you want to test and 2/ absolute consistency.

Don't expect to shoot a brick wall or test patterns and know everything about a lens. You'll probably have to shoot a series of tests, preferably comparisons between similar lenses, to learn your lens.

I've written a basic how-to here http://photocornucopia.com/1044.html . It's a start and covers the basic methodology that I think works and is achievable.

For viewing/comparing results I like this kind of thing: http://photocornucopia.com/1055.html

John, many thanks, this is precisely the kind of information I was hoping for!

Nov 23, 2013 at 11:48 PM

Samuli Vahonen
Offline
• • • •
Upload & Sell: Off

p.1 #4 · p.1 #4 · Guidelines for comparative lens testing?

genji wrote:
* If not infinity, what is the minimum subject distance that will provide useful data for landscape shooters?
In the past 50x focal length was considered infinity from lens aberration correction point of view.

genji wrote:
* Is 1.5m (~5ft) a good choice for testing a lens at close distance? Or should the focus distance be a fixed multiple of the focal length?
No, for wide angle lens 1.5m is already VERY far away (see comment above). I don't know what is correct method but I would vote subject size based method, which also allows you to compare for example 50mm and 55mm lens properly.

genji wrote:
* Should the lens be refocused at each tested aperture to show it at its best? Alternatively, should the focused distance remain constant to reveal any possible focus shift or field curvature? Or would it be best to focus two stops down from wide open?
Opinion: Results to be shown with correct focus at each aperture and separately explain focus shift and field curvature. At least I shoot with optimal focus even some other photographer would not be able to compensate focus shift - so I personally don't see point comparing incorrectly focused images to correctly focused images due to photographers fail to compensate. With modern cameras and live view there is no excuse for fail to compensate.

Typically you don't need to refocus for each aperture, usually focus shift is gone two stops down (which you first have to study, all lenses/distances are not the same). So for example if you have f/1.4 lens, which has focus shift until f/2.8, you focus f/1.4 shot, you focus f/2 shot and then you focus f/2.8 shot and shoot the rest (f/4, f/5.6 etc.) with the same focus.

Also when you covering focus shift topic you need to understand that it's focus distance dependant, e.g. lens may have focus shift at 1m but doesn't have any at infinity.

genji wrote:
I'm sure there are lots of other variables to think about so please regard my questions as a means of eliciting other factors that should be taken into consideration...
I find 95% of lens test to be failures. Sure they tell something, but mostly testers are fixated to boring stuff like sharpness. It would be enough one tester to test sharpness so 100 other could concentrate to interesting stuff like bokeh, LoCa in actual shooting, color and contrast behaviour at different apertures, veiling flare in practical backlight scenarios etc. But instead majority of them fixate to sharpness, and most testers test do even that wrong (e.g. use only one distance, ruin images by not compensating focus shift, shoot target from angle and don't get all corners sharp etc. etc.).

Samuli

Nov 24, 2013 at 02:34 AM

sebboh
Offline
• • • • • •
Upload & Sell: Off

p.1 #5 · p.1 #5 · Guidelines for comparative lens testing?

with regard to focus shift, i would say it's best to refocus at every new aperture till you reach f/4.

also, i would recommend taking separate shots at each aperture one with the center in best focus and one with the corner in best focus.

Nov 24, 2013 at 01:42 PM

Lee Saxon
Offline
• • • •
Upload & Sell: Off

p.1 #6 · p.1 #6 · Guidelines for comparative lens testing?

Is testing at infinity likely best if we're trying to maximize color shift / smearing?

Nov 24, 2013 at 02:18 PM

JohnJ
Offline
• • • •
Upload & Sell: Off

p.1 #7 · p.1 #7 · Guidelines for comparative lens testing?

Lee Saxon wrote:
Is testing at infinity likely best if we're trying to maximize color shift / smearing?

Yes, because the lens will typically be closest to the sensor when focused at infinity. But isn't this the point of testing, so why not try it and see if there really is a difference?

Nov 24, 2013 at 02:37 PM

jcolwell
Online
• • • • • • •
Upload & Sell: On

p.1 #8 · p.1 #8 · Guidelines for comparative lens testing?

Here's a link to the PDML (Pentax Discuss Mailing List) test procedure that got me started,

http://www.takinami.com/yoshihiko/photo/lens_test/pdml-procedure.html

I still sometimes use the USAF 1951 test target and resolution calculations, but more often I use 'real' situations. Including some walls that have bricks in them.

Here's some links to lens comparisons I've posted at FM.

TSE 17mm: acceptable performance? https://www.fredmiranda.com/forum/topic/1229794
ZE18/3.5 - TS-E 17/4L https://www.fredmiranda.com/forum/topic/1213602/ 1#11566782
Replacement for 17-40L [links to all below] https://www.fredmiranda.com/forum/topic/1205010/ 1#11486857
Which swa for my 5dii - oly21, voigt 20 or zeiss 21? https://www.fredmiranda.com/forum/topic/1205970/ 0#11493516

85mm bokeh & detail images] https://www.fredmiranda.com/forum/topic/952008/0&year=2010#9001461
[best 85 Alt] https://www.fredmiranda.com/forum/topic/987267/0
[Rokinon 85/1.4 (sample pics)] https://www.fredmiranda.com/forum/topic/941350
Extender 1.4x Mk I vs. Mk III https://www.fredmiranda.com/forum/topic/1002803
Extender 2x Mk II vs. 2x Mk III https://www.fredmiranda.com/forum/topic/1002804

[X-Pro vs DSLR, XF 14 vs Oly 21, XF 35 vs CZ 35-70], https://www.fredmiranda.com/forum/topic/1219778/2#11619090
X-Pro 1 XF 18mm f/2 R lens tests https://www.fredmiranda.com/forum/topic/1100128
XF 55-200 OIS examples https://www.fredmiranda.com/forum/topic/1247748/0#11867987

My technique, which is described in at least a few of the links, has evolved as camera and lens technologies have evolved. For example, I sometimes do handheld testing with image stabilization. This simply shows me what I can achieve in this situation, not what the camera and lens are ultimately capable of achieving; for which you need tripod, MLU, remote release or shutter delay, multiple images, yadda, yadda, yadda. OTOH, if I find a significant difference in myhandholding IQ between two sets of gear, then I have learned something worthwhile.

A more recent development in lens testing is associated with autofocus microadjustments (AFMA), which I do on my Canon DSLR bodies,

MA Adjust Settings and Record Keeping https://www.fredmiranda.com/forum/topic/1197146/0#11415313
Micro. Adjustment for auto focusing https://www.fredmiranda.com/forum/topic/1196841/0#11412580

P.S. I agree with JJ's "most important aspects" stated in the first reply to the thread.

Nov 24, 2013 at 03:22 PM

Phillip Reeve
Offline
• • • •
Upload & Sell: Off

p.1 #9 · p.1 #9 · Guidelines for comparative lens testing?

Great thread!

If you are not careful enough with your testing all you measure are your own errors, not those of the lens.

I would like to get some feedback on my testing procedure (In the past I have not always followed it strictly):

I focus at the image center.

For f/1.4 , f/2 and f/2.8 I refocus 2 or 3 times and select the sharpest one in LR.

To check for field curvature I focus at the corners at a wider aperture and later compare it to the centrally focused image. If there is a significant difference I will report it.

My new infinity test target is my university, it has many details and straight lines and the subject distance is over 50 meters.

for closeups I use a rather small target which is 40cm x 60cm which is sub-optimal for WA lenses but I don't want to build something much bigger.

When if have selected the best images for each aperture I load them into PS and let an action do the work, all i have to do is to place the crop marks.

when placing the crop mark I allow myself to move about 200 px away from the ideal position if i can get better details at this setting.

How the results look:

howtotest by reevedata, on Flickr

My specific Questions

How much information about my testing procedure (like tripod, selfrelease, manual focus, oss, were i focused etc. should be written on the image?

Is the overview image necessary?

I am happy about every constructive criticism.

Nov 24, 2013 at 05:53 PM

Lee Saxon
Offline
• • • •
Upload & Sell: Off

p.1 #10 · p.1 #10 · Guidelines for comparative lens testing?

Phillip, definitely keep the overview image. I've seen a few "crop compilations" lately that omit it and it's pretty disorienting (especially one that was meant to assess color shift / smearing but only included areas affected by those problems! What's color shift and what's incorrect white balance? I needed the center of your image to know, dude!)

I'm confused by the 50x focal length (or 55x in Jim's link).

35mm x 50/55 = 5'9"/6'4", nowhere near infinity focus

Nov 24, 2013 at 05:56 PM

rscheffler
Offline
• • • • • •
Upload & Sell: Off

p.1 #11 · p.1 #11 · Guidelines for comparative lens testing?

Yeah, not sure about 50x focal length for infinity, but that is the distance recommended by Canon for micro focus adjustment.

Phillip, I think it's helpful to include all of that extra information as it will immediately address most questions/doubts a viewer will have about test procedures. The overview image is also extremely helpful.

As for your example scene, I think it's suitable for wide angle tests, but not far enough away and perhaps with too much depth for 50mm and longer lenses. For example, where will your center focus point be? On the retaining wall, the trees, the bicycle rack? That said, a 3D scene can be good because it can show how areas in front and behind the point of focus will be rendered, the effects of field curvature, etc.

Nov 24, 2013 at 07:04 PM

genji
Offline
• • • •
Upload & Sell: Off

p.1 #12 · p.1 #12 · Guidelines for comparative lens testing?

This thread is coming together exactly as I'd hoped, with very practical suggestions and observations from everyone who has contributed. Thank you! I like Phillip's presentational method a lot since it combines a wealth of useful information in an easy to comprehend form. (Phillip, have you read Edward Tufte's The Visual Display of Quantative Information?) I hope you don't mind if I attempt to emulate your visual presentation (as I also attempt to put into practice the advice contained in this thread).

Nov 24, 2013 at 09:25 PM

Phillip Reeve
Offline
• • • •
Upload & Sell: Off

p.1 #13 · p.1 #13 · Guidelines for comparative lens testing?

genji wrote:
This thread is coming together exactly as I'd hoped, with very practical suggestions and observations from everyone who has contributed. Thank you! I like Phillip's presentational method a lot since it combines a wealth of useful information in an easy to comprehend form. (Phillip, have you read Edward Tufte's The Visual Display of Quantative Information?) I hope you don't mind if I attempt to emulate your visual presentation (as I also attempt to put into practice the advice contained in this thread).
Feel free to copy everything, my method is inspired by other people's tests as well.

Nov 25, 2013 at 01:52 AM

theSuede
Offline
• • • •
Upload & Sell: Off

p.1 #14 · p.1 #14 · Guidelines for comparative lens testing?

100x focal length can definitely be considered infinity for most purposes except the most demanding ones - that also put extreme demands on test setup and execution. 50xFL is when focus goes past "midpoint" into near-infinity on a unit-focusing system without floats. Internal or rear-focusing systems are often more complex, but not to the point where you really need to care unless you're going really into it analytically.

"Close focus" depends on your definition of "close"...

but yes, that's also a magnification ratio if you want consistency from an optical PoV. So maybe 20-30x FL. Shift that to image relative magnification for small(er) formats if you want to keep the comparisons geared towards a pragmatic/practical photography point of view - so multiply by "crop ratio"...

Refocusing is necessary with quite a lot of lenses, especially if you're live-view focusing to get the best average corner-midfield-center focus. Especially for the close-range shots, where spherical aberrations are often quite a lot stronger. There's complex interaction between field curvature and spherical aberration. But in general, concentrating on center field, you'll need to move focus very slightly towards you as you stop down the first two stops from wide open.

Lighting conditions are critical - but only if you plan on making more tests another day. For comparisons between "only" two lenses, and you can keep testing to a few minutes of setup - it doesn't matter as long as they're identical within the set.
(but stuff like purple fringes and so on are more easy to see in UV/IR rich light, i.e sunshine or badly filtered flash

)

Nov 25, 2013 at 07:08 AM

Phillip Reeve
Offline
• • • •
Upload & Sell: Off

p.1 #15 · p.1 #15 · Guidelines for comparative lens testing?

theSuede wrote:
Refocusing is necessary with quite a lot of lenses, especially if you're live-view focusing to get the best average corner-midfield-center focus. Especially for the close-range shots, where spherical aberrations are often quite a lot stronger. There's complex interaction between field curvature and spherical aberration. But in general, concentrating on center field, you'll need to move focus very slightly towards you as you stop
thanks for your input, it is always good to hear someone with great theoretical (and practical) knowledge.

Is refocussing necessary even with rather slow lenses which start at f/2.8 or f/3.5?
theSuede wrote:
Lighting conditions are critical - but only if you plan on making more tests another day. For comparisons between "only" two lenses, and you can keep testing to a few minutes of setup - it doesn't matter as long as they're identical within the set.
(but stuff like purple fringes and so on are more easy to see in UV/IR rich light, i.e sunshine or badly filtered flash

)

So if I do a comparison like the one above, which was taken around sunset and with some artifical light and compare it to the images I took some minutes ago while there was some light rain, du you think the results would differ?

I think I should really test one lens in several different ways do learn how the conditions influence the results.

Nov 25, 2013 at 07:13 AM

Phillip Reeve
Offline
• • • •
Upload & Sell: Off

p.1 #16 · p.1 #16 · Guidelines for comparative lens testing?

Okay my latest test:

Minolta_MD_35mm_f28_SonyA7_infinity by reevedata, on Flickr

any idea how to improve it?

Nov 25, 2013 at 03:07 PM

jcolwell
Online
• • • • • • •
Upload & Sell: On

p.1 #17 · p.1 #17 · Guidelines for comparative lens testing?

Phillip Reeve wrote:
Is refocussing necessary even with rather slow lenses which start at f/2.8 or f/3.5?

IMO, yes. Who's to say that the first attempt is the best? I often get different results for "exactly the same" shooting parameters. "Best of three" is what I use. Sometimes all three images are about the same. Sometimes two are 'good'. If all three show different apparent focus/sharpness/clarity, then I discard and to it again, usually on a different day.

Phillip Reeve wrote:
So if I do a comparison like the one above, which was taken around sunset and with some artifical light and compare it to the images I took some minutes ago while there was some light rain, du you think the results would differ?

Yes. The relative position of the sun can have a huge effect on contrast (or lack of), and light rain will make everything at a distance look a bit soft. Plus, wet subjects reflect light diffrently than dry ones.

Phillip Reeve wrote:
I think I should really test one lens in several different ways do learn how the conditions influence the results.

I agree.

Nov 25, 2013 at 03:28 PM

theSuede
Offline
• • • •
Upload & Sell: Off

p.1 #18 · p.1 #18 · Guidelines for comparative lens testing?

Phillip Reeve wrote:
thanks for your input, it is always good to hear someone with great theoretical (and practical) knowledge.

Is refocussing necessary even with rather slow lenses which start at f/2.8 or f/3.5?

So if I do a comparison like the one above, which was taken around sunset and with some artificial light and compare it to the images I took some minutes ago while there was some light rain, do you think the results would differ?

I think I should really test one lens in several different ways do learn how the conditions influence the results.

Thank you, you also received many good pointers from others. Remember that there's many ways, many angles on "lens testing".

Refocusing is often necessary in the first step of stop-down even on F4.0 lenses. Especially if you want a weighted corner performance too - curvature of field and astigmatism/coma can change dramatically in the first stop. I find this more important at distance, for close-range shots if corner>corner performance is important you almost have to use a dedicated macro. As you get closer to 1:1 reproduction everything changes quite drastically, and the need for a meticulous setup grows almost exponentially.

Yes, slight differences in atmosphere conditions and surface moisture can change appearance of details (and detail contrast) by a lot. One should be careful with interpreting ANY comparison where the situations have changed by a larger amount than the differences you're looking for would induce.

I often see people "comparing" images taken under totally different circumstances, in totally different conditions, on very different scenes with each other. It's basically a void comparison, the result will be almost whatever you want it to be. Of course - if one lens is close to perfect, and the other is basically a broken bottle, then you're going to see the difference. The problem is that you can't know how big the difference is - was the good lens shot with perfect technique, with perfect focus - and the bad lens with horrible technique and slightly off focus? That could make a small difference into something huge, visually.
Now expand that series into several similarly flawed comparisons, compare the comparisons - and you get chaos at best, personal preference and bias if you're lucky, and all-out shouting wars if you're less lucky.

Nov 25, 2013 at 06:27 PM

genji
Offline
• • • •
Upload & Sell: Off

p.1 #19 · p.1 #19 · Guidelines for comparative lens testing?

Two more quick questions:

* If I'm comparing three lenses, for example two RF 35mm lenses (manual focus) and the FE 35/2.8 (autofocus) should I manually focus the FE 35 too? (My intention is to use the FE 35 as a reference lens.)
* Does white balance have an influence? If I'm shooting a test series in bright sunlight, should I set the WB to Sunlight or is Auto WB acceptable?

Nov 26, 2013 at 01:23 AM

sebboh
Offline
• • • • • •
Upload & Sell: Off

p.1 #20 · p.1 #20 · Guidelines for comparative lens testing?

genji wrote:
Two more quick questions:

* If I'm comparing three lenses, for example two RF 35mm lenses (manual focus) and the FE 35/2.8 (autofocus) should I manually focus the FE 35 too? (My intention is to use the FE 35 as a reference lens.)
* Does white balance have an influence? If I'm shooting a test series in bright sunlight, should I set the WB to Sunlight or is Auto WB acceptable?

manual focus all the lenses.

white balance matters. don't use auto WB. if you want to compare for final output set white balance off the same point in each image (different lenses will have different color transmission properties). if you want to see the differences in color transmission between the lenses, set WB to the same value across shots with all lenses.

Nov 26, 2013 at 02:09 AM

1 2 end
FM Forums \| Leica & Alternative Gear \| Join Upload & Sell 1 2 end

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.