p.3 #2 · How accurate is www.the-digital-picture.com?
PetKal wrote:
The TDP reviews are generally quite thorough and comprehensive although much of what they write is a rehash of the Canon specs/features and physical descriptions, and they are weak on practical lens assessment.
A noteworthy exception to that is when they documented a weakness in the MkII supertelephoto lenses......to my knowledge they were the first to do that, and are still amongst only a few folks who have done so.
Of all the review sites I do pay some attention to what P-zone have to say.
However, my most valued lens/camera info resource is in fact a couple of senior FM members and very active photographers who have no review web sites to draw traffic to, nor they sell gear, seminars and such, but they have used a goodly variety of gears for quality photography. I like to compare my notes with theirs....Show more →
www.petkal.com appears to still be available for you and your senior contributors to start publishing your notes.
p.3 #3 · How accurate is www.the-digital-picture.com?
taylorman22 wrote:
The title says it all. I've been reviewing a lot of lenses and was just curious how the sharpness samples on the site compare with the real world.
His lens tests are based on individual samples. Canon lenses tend to have some copy-to-copy variation. Furthermore, his ISO tests are based on one corner (out of 4). It is known that some Canon lenses suffer from decentering. So, IMO you have to take this with a grain of salt.
p.3 #4 · How accurate is www.the-digital-picture.com?
I value the Digital Picture's lens comparison tool. For example it is a simple way to compare the new 35 f2 to my old 35 f2 and to my main zoom at 35 mm. Generally his resolution tool results compare well with my experience with my lenses. But as Peter said I also rely on the comments made in this forum by individuals I respect and also on the reviews and overall score from the lens reviews here on FM. I like the stars to align before I commit to purchase.
p.3 #5 · How accurate is www.the-digital-picture.com?
It's always fun to read a thread and find out we all generally agree.
Personally, I try to paint with a broad brush quickly when something new comes out, because I know a lot of people are deciding about preorders, etc.
Then I look at TDP, SLRGear, Photozone, Lenstip, and Diglloyd. I think each brings something a bit different to the table and each gives a much more thorough look than I ever could (or would - my ADD tires of a lens after a day or so).
Finally, I find the comments on FM as the lens comes into general circulation are the final word. I can't say how many times the people on the forum have found something none of the review sites have ever found. Which is what happens when people go out in the real world and take 2,000 shots. As compared to 500 shots made in the lab.
One aside I would like to mention - we work with TDP, SLRGear, and Diglloyd concerning copies. If Lloyd, Dave, or Brian have questions about the lens they're testing we try to get them a tested comparison copy so they can confirm if their copy is good or bad. In some cases we send them a copy we've tested so they know they're starting out with at least a decent one. I'd do the same for the others, too, I admire their work, but there's no way to ship out of the U. S. without paying duties and taxes.
This is something that has started in the last year and I think will end up being a very good thing. I could never spend the kind of time and painstaking effort with a lens that those guys do, but at least I can help make sure they have a representative copy to work with.
p.3 #7 · How accurate is www.the-digital-picture.com?
The people criticising the ISO 12233 tests need to provide more information. The implication seems to be that it is his methodology that is incorrect, because otherwise you are criticising the actual test itself in which case it doesn't matter what site it's posted on.
The only relevant point I've seen so far is that only one corner is shown which will not illustrate any decentering problems. I realise copy-to-copy variation exists but it is not the-digital-picture's responsibility to ensure he's using "a good copy" of a lens for his tests. The fact there's bad copies is a problem with the lens manufacturer, not the tester.
p.3 #8 · How accurate is www.the-digital-picture.com?
Harry.C wrote:
The people criticising the ISO 12233 tests need to provide more information. The implication seems to be that it is his methodology that is incorrect, because otherwise you are criticising the actual test itself in which case it doesn't matter what site it's posted on.
The only relevant point I've seen so far is that only one corner is shown which will not illustrate any decentering problems. I realise copy-to-copy variation exists but it is not the-digital-picture's responsibility to ensure he's using "a good copy" of a lens for his tests. The fact there's bad copies is a problem with the lens manufacturer, not the tester. ...Show more →
as I said the earlier tests on that site used very few trials per shot, many of which even relied on AF and since even the tiniest, tiniest, tiniest list shift can mix the results around for this sort of chart test....
for some reason basically every single tamron lens he tests does way worse than any of my copies (and sigma and such always seem to do poorly even a few lenses of theirs that tend to get praised), I don't understand how over years and years every single tamron, and even to some extent third party lens in general, always seems to be a substandard copy on his site (one recent case, his tamron 70-300 VC suddenly goes to utter trash at 300mm and get easily beaten by 70-200 f/4 IS + 1.4x TC III and yet i compared three tamron 70-300 vc and 2 70-200 f/4 IS+ 1.4x TC and not in one combo did I ever see anything remotely like that going on)
a few times a lens would suddenly get way blurrier and then get sharper again as the fstop varied, granted with so many tests it's easy enough to mix things up here and there, i can't even fathom testing all that he does, that many lenses at so many different f-stops and ranges and bodies, it would drive me mad, it takes forever, hours just to do a single lens if you use enough trials to make it more likely to be valid, I don't know how he even does it, pretty remarkable.
i don't think he refocuses for edge performance so in certain types of real world shooting you may end up with results that feel a lot different either because the FC is different at landscape distances or in scenes with objects all over some will fall into the crisp zone right out to the edges, OTOH in other cases this may tell the truer tale (not refocusing also makes the test even ever all the more utterly sensitive, and a lens with just modest FC of a certain type might put up nasty results and yet for anything but stuff like flat field painting reproduction do very well, but again the refocusing method can questionable for some scenarios too)
p.3 #9 · How accurate is www.the-digital-picture.com?
I'm not criticizing Bryan's methods with regard to the ISO crops...in fact, I admire the fact that he goes through so much effort to make sure his results are as accurate as they can be. Perhaps it is copy variation or just the nature of the tests...I don't know.
I'm just saying that I've owned multiple copies of certain lenses and they looked much better in my own 'resolution' tests. Also...sometimes other sites like PZ will show a lens performing better than what you see in the crops at TDP.
The 85/1.8 is one that comes to mind...the copy he tested looks soft wide open (looks more like a 50/1.4 wide open). Even stopped down to F/2.8 the corners are still soft. Both of the copies I have owned have been very sharp right from F/1.8.
The copy PZ tested looks quite good right from F/1.8 which is more inline with my experience with that particular lens.
I've run into other examples but that's just one that comes to mind...
p.3 #10 · How accurate is www.the-digital-picture.com?
ChrisRD wrote:
I'm not criticizing Bryan's methods with regard to the ISO crops...in fact, I admire the fact that he goes through so much effort to make sure his results are as accurate as they can be. Perhaps it is copy variation or just the nature of the tests...I don't know.
You've listed two possibilities to explain the difference you are seeing:
1.) copy variation
2.) nature of the tests
The ISO 12233 test is not some arbitrary test developed by the owner of that site, and unless you are arguing the owner is performing the tests erroneously then I think we can rule out that option.
As I mentioned before, the fact copy-to-copy variation exists is something that should be seen as a negative towards the lens manufacturer, not the poor soul who is trying to provide a standard for which to compare lenses against each other. Trying to account for copy-to-copy variation would mean exponentially more work for the site owner and also the ability to obtain 30+ of each lens.
p.3 #11 · How accurate is www.the-digital-picture.com?
IMHO TDP has a slight tendency towards (sympathy for) Canon products without saying they are biased. Often it says "I would like to complete a full review of…." where I would like to see an alternative product reviewed.
Like a lot of other review sites they seem to tell us: BUY BUY BUY! and of course provide the fitting link where you find your best price. IMO this moves them close to the industry and sometimes further away from the end user needs.
Then I go there because they provide good information about canon products and some other. I like the graphics and I find them accurate in their findings. All together it is a nice resource on Canon
p.3 #12 · How accurate is www.the-digital-picture.com?
Hulot wrote:
IMHO TDP has a slight tendency towards (sympathy for) Canon products without saying they are biased. Often it says "I would like to complete a full review of…." where I would like to see an alternative product reviewed.
Like a lot of other review sites they seem to tell us: BUY BUY BUY! and of course provide the fitting link where you find your best price. IMO this moves them close to the industry and sometimes further away from the end user needs.
Then I go there because they provide good information about canon products and some other. I like the graphics and I find them accurate in their findings. All together it is a nice resource on Canon...Show more →
p.3 #13 · How accurate is www.the-digital-picture.com?
I would say that the majority of we readers of this Canon-mount SLR forum have a tendency towards sympathy for Canon products since we've ended up buying them. TDP is created by a Canon user so he must like Canon products. That doesn't mean he isn't objective when he runs his ISO 12233 tests. When I first started using his site, he only tested Canon products. Now he includes other manufacturer's products and there is no reason he would treat them differently when testing.
When reading his written reviews I assume I'm reading the opinions of an experienced photographer who is familiar with the entire Canon line which he uses in his profession. I'm sure he emphasizes the positive aspects of each product but there are plenty of critics on the forums to balance that out.
p.3 #14 · How accurate is www.the-digital-picture.com?
Harry.C wrote:
The people criticising the ISO 12233 tests need to provide more information. The implication seems to be that it is his methodology that is incorrect, because otherwise you are criticising the actual test itself in which case it doesn't matter what site it's posted on.
The only relevant point I've seen so far is that only one corner is shown which will not illustrate any decentering problems. I realise copy-to-copy variation exists but it is not the-digital-picture's responsibility to ensure he's using "a good copy" of a lens for his tests. The fact there's bad copies is a problem with the lens manufacturer, not the tester. ...Show more →
It's impossible to provide more info when we don't he how he is doing the tests. Do you know if he uses AF, MF or Live view on the test charts? Do you know if he refocus every time in the corners, edges or center? Do you know if he do the test 1 or 10 times? Of course it matter what site the test is on, when people do it in a different way on different sites.
(I like his website)
p.3 #15 · How accurate is www.the-digital-picture.com?
Harry.C wrote:
The ISO 12233 test is not some arbitrary test developed by the owner of that site, and unless you are arguing the owner is performing the tests erroneously then I think we can rule out that option.
When referring to the 'nature of the tests' as a possible variable I was thinking more along the lines of the difficulties inherent to photographing a flat target with a lens that is not really designed for photographing flat targets in order to determine how good it will be at photographing targets that are not flat.
I'm not a lens designer nor do I have much technical knowledge about lens design so I have no idea how much this may be a factor (if at all).
Harry.C wrote:
As I mentioned before, the fact copy-to-copy variation exists is something that should be seen as a negative towards the lens manufacturer, not the poor soul who is trying to provide a standard for which to compare lenses against each other. Trying to account for copy-to-copy variation would mean exponentially more work for the site owner and also the ability to obtain 30+ of each lens.
I don't think anyone was trying to blame Bryan for copy variation...he's actually made an effort to test multiple lenses for several of his test runs.
I think the point some of us where trying to make was that you have to take the ISO crops with a grain of salt...probably largely because of copy variation and because it's not practical to test a large number of lenses in this fashion.
When referring to the 'nature of the tests' as a possible variable I was thinking more along the lines of the difficulties inherent to photographing a flat target with a lens that is not really designed for photographing flat targets in order to determine how good it will be at photographing targets that are not flat.
I'm not a lens designer nor do I have much technical knowledge about lens design so I have no idea how much this may be a factor (if at all).
I don't think anyone was trying to blame Bryan for copy variation...he's actually made an effort to test multiple lenses for several of his test runs.
I think the point some of us where trying to make was that you have to take the ISO crops with a grain of salt...probably largely because of copy variation and because it's not practical to test a large number of lenses in this fashion....Show more →
"However, I'm not totally satisfied with the Canon EF-S 17-55mm f/2.8 IS USM Lens results - the 17-55 under-performed in the comparison tool images in my opinion. And the cheap Canon EF-S 18-55mm f/3.5-5.6 IS USM Lens performed better at the wider focal lengths at the relatively short distance than I would expect to see from this lens - it over-performed in the comparison tool. "
interesting thing he brings up in that the 18-55 IS actually really is rather crazy sharp, it's easy to think oh a cheap kit, and while it may not deliver the most stunning large-scale contrast and colors, it really is pretty sharp, honestly it seemed sharper than my 24-105. And the fabled 17-55 IS, if you look at some other reports and tests, while it is very sharp, it actually has often tested just a bit under many of the other top lenses (although the 17-55 has the lowest CA on the wide side for sure)
so, oddly enough, the one pair of results he questions, is the one I suspect may actually be dead on
p.3 #17 · How accurate is www.the-digital-picture.com?
Harry.C wrote:
The people criticising the ISO 12233 tests need to provide more information. The implication seems to be that it is his methodology that is incorrect, because otherwise you are criticising the actual test itself in which case it doesn't matter what site it's posted on.
The only relevant point I've seen so far is that only one corner is shown which will not illustrate any decentering problems. I realise copy-to-copy variation exists but it is not the-digital-picture's responsibility to ensure he's using "a good copy" of a lens for his tests. The fact there's bad copies is a problem with the lens manufacturer, not the tester. ...Show more →
I agree with this post. My view is that generally the iso tests agree with my own intuitive sense whenever it is a lens I am familiar with. While there is lens to lens variation, I think the bigger issue is that people get too wrapped up in measurements, whereas the ISO tests are actual results.
Aside from sample to sample I have not heard any valid critiques.
p.3 #18 · How accurate is www.the-digital-picture.com?
skibum5 wrote:
"However, I'm not totally satisfied with the Canon EF-S 17-55mm f/2.8 IS USM Lens results - the 17-55 under-performed in the comparison tool images in my opinion. And the cheap Canon EF-S 18-55mm f/3.5-5.6 IS USM Lens performed better at the wider focal lengths at the relatively short distance than I would expect to see from this lens - it over-performed in the comparison tool. "
interesting thing he brings up in that the 18-55 IS actually really is rather crazy sharp, it's easy to think oh a cheap kit, and while it may not deliver the most stunning large-scale contrast and colors, it really is pretty sharp, honestly it seemed sharper than my 24-105. And the fabled 17-55 IS, if you look at some other reports and tests, while it is very sharp, it actually has often tested just a bit under many of the other top lenses (although the 17-55 has the lowest CA on the wide side for sure)
so, oddly enough, the one pair of results he questions, is the one I suspect may actually be dead on ...Show more →
The iso test does show that 17-55 does perform better at f4 than the 18-55 at f4. You are likely comparing 17-55 at f2.8 vs 18-55 at f4 which is not a valid comparison. Most really fast lens do worse wide open, but its nice to have the f2.8. So the iso charts do reflect my understanding.
Thats what I like about the iso test. A lot of what you hear on forum's is just a repeat of what people heard from someone else without verifying it.
For example, what you will find using the iso test is the new 24-70 v2 at f4 is better than the 24-105 at f4 until about 35mm but after that they are basically equal with 24-105 obviously being way being way better from 70-105. And this gap disappears to almost nothing at f8 at 28mm, with is typically where you would shoot landscape. But if you listen to feedback on forums this is all summarized down to 24-70 being way better. A more nuanced view is that its tougher to build a wider zoom but a wider zoom is more useful. So the big difference is f2.8 vs f4, reach, and weight.
p.3 #19 · How accurate is www.the-digital-picture.com?
Scott Stoness wrote:
The iso test does show that 17-55 does perform better at f4 than the 18-55 at f4. You are likely comparing 17-55 at f2.8 vs 18-55 at f4 which is not a valid comparison. Most really fast lens do worse wide open, but its nice to have the f2.8. So the iso charts do reflect my understanding.
Thats what I like about the iso test. A lot of what you hear on forum's is just a repeat of what people heard from someone else without verifying it.
actually I wasn't comparing anything myself, I was just quoting what TDP themselves said about those two lenses and their test of them!
For example, what you will find using the iso test is the new 24-70 v2 at f4 is better than the 24-105 at f4 until about 35mm but after that they are basically equal with 24-105 obviously being way being way better from 70-105. And this gap disappears to almost nothing at f8 at 28mm. But if you listen to feedback on forums this is all summarized down to 24-70 being way better. A more nuanced view is that its tougher to build a wider zoom but a wider zoom is more useful. So the big difference is f2.8 vs f4, reach, and weight. ...Show more →
And yet you find few people saying that their 24-105 matches their 70-200/300L 70mm, f/4, center frame and tons of people saying their 24-70 II matches or beats their 70-200/300L center frame, often when they are at f/4 and the 24-70 is at f/2.8!
And look at the photozone numbers of the lens rental numbers at 70mm....
p.3 #20 · How accurate is www.the-digital-picture.com?
skibum5 wrote:
actually I wasn't comparing anything myself, I was just quoting what TDP themselves said about those two lenses and their test of them!
And yet you find few people saying that their 24-105 matches their 70-200/300L 70mm, f/4, center frame and tons of people saying their 24-70 II matches or beats their 70-200/300L center frame, often when they are at f/4 and the 24-70 is at f/2.8!
And look at the photozone numbers of the lens rental numbers at 70mm....
Thats my point to skibum - the iso charts don't show this to be true. At 50mm and f4 the 24-105 is quite good as 24-70v2 at 50f4 and 70-200v2 at 100f4. So either the differences are not noticable to the human eye or something is wrong with the digitalpicture. I think that it is more likely that the differences are not noticable and the mtf varies considerably across the zoom. Typically zoom lens are sharpest away from their extremes and away from their wide open points. And typically longer range of zooms are not as sharp as shorter range of zooms. These nuances are lost in dicussion which boils 24-70v2 and 70-200v2 down to the best lens ever and 24-105 as an average performer.
Thedigitalpicture is the only site that has easy visual comparisons between fstops and mm settings for centre and edges.
