Jordan -- Thanks so much for the comparison shots. Regardless of any criticisms, these are very interesting and valuable in presenting positive aspects of M43. The overall impression is that it provides outstanding IQ in a small, and relatively cheap package. I'll continue to consider getting one for the reasons of convenience and carry, but it is still on the back burner for the time being.
I also appreciate very much denoir's analysis, and mostly agree with his conclusions. I have owned the 17-40 and currently own the 16-35II. I agree that both are quite weak in the corners, especially the 17-40 (note the color fringing on the sample you provided). My 16-35II is better, but not perfect there. But you are right -- there is a certain point we all say, "This is more than adequete for my purposes" and move on. I find the 16-35II more useful to me than the ZE 21/2.8, even though the Zeiss lens is optically superior (I own both). Whenever possible, I use the Zeiss, but if I am packing for an assignment with unknown parameters, it stays home. I'd like to mention that the 17-40 was exceptionally sharp in the broad central image area, especially for a WA zoom -- the 16-35 is in the same league there, IMO.
I do see a lot more noise from the M43 throughout the detail of the rocks/pool. I think the 1Ds2 could have done a lot better there with slightly finessed PP sharpening, but the inference is clear: the M43 is noisier by far.
I'm very impressed with your comparisons, and hope you will consider doing some exact head-to-head shots at the same location, at the same time, to best show the attributes of each format.
Thanks again for putting forth this valuable effort.
denoir wrote:
...DxO is well... of debatable usefulness. They should get credit for trying to make proper test. The problem is that they summarize the data in a too simplistic way which often leads to rather ridiculous results....
I'm interested in more info on this, Luka. Can you go into detail a little bit about how exactly the graphs for DxO Mark's sensor results are too simplistic? Thanks!
douglasf13 wrote:
I'm interested in more info on this, Luka. Can you go into detail a little bit about how exactly the graphs for DxO Mark's sensor results are too simplistic? Thanks!
I'll be very lazy and just quote an example complaint from an article on LL:
-------------------- Unfortunately DxO Labs has complicated matters. Their DxO Mark camera tests have become widely read by photographers around the world, and are quoted as gospel by many. Regrettably there are concerns that I, and others have expressed about some aspects of DXO's tests, and I have been in communication with them over this in the past.
The only aspect that is relevant to this discussion is with regard to dynamic range. The standard definition is, as mentioned above, how many F stops above and below middle gray can be recorded while delivering full texture and detail. The DXO definition, according to their web site, is the range between zero signal to noise and full saturation of the sensor.
This approach is not inherently flawed, it's just that it does not take into account the linear nature of sensors. It is therefore not a particularly relevant way of measuring DR to a photographer, as opposed to an engineer....Show more →
-----------------------------
Now add to that what DxO labs actually say to the user that doesn't dive into their documentation that states their definition:
------------ The Landscape Score is defined as the maximum Dynamic Range of the camera sensor and its unit is an exposure value (EV).
A value of 12 EV is excellent with differences below 0.5 EV usually not noticeable.
This score is named "Landscape" as contrasts are typically very high in landscape photography and it's therefore critical to have wide dyanamic range for such use.
-------------
This type of dumbed down explanation is not exactly helpful as their measurement is not what photographers usually refer to as dynamic range. And then the P&S users conclude "Yey, my camera got a high 'Landscape score" while the digital medium format aristocracy go bananas because their beloved $$$$$ equipment is according to DxO outperformed by cameras in the 'peasant class'. You see the problem.
There is no simple objective answer to this important question. Probably every image quality expert would have a somewhat different personal preference for a benchmark like this. But my impression is that the benchmark is pretty useful: I analyzed the model and the data, but didn’t find any serious flaws. Furthermore, results like Figure 2 appear to be pretty consistent with traditional hands-on reviews: models that were stronger [weaker] than state-of-the-art when they were introduced (such as the Canon 40D [50D]) show up as expected in the DxOMark data. And, again, having a pretty solid metric by an independent party is better than endless discussions about what an ideal metric might look like.
Better in an absolute sense, sure, but it suffers from exactly the same problem as DxO does. His second sentence in the quote you selected is the key:
Probably every image quality expert would have a somewhat different personal preference for a benchmark like this.
The guy that wrote it is an engineer and approaches it from an engineering point of view. Nothing wrong with that, but the problem is that the benchmarks are primarily being read by photographers and not by engineers and image quality experts. The problem is that DxO uses nomenclature that has a different accepted meaning among photographers and then they follow it up by dumbing down the descriptions they give on the site.
Another problem is their scoring system which often leads to ridiculous results - such as rating the Canon 85/1.8 higher than the Zeiss 100/2 MP. In fact they rate the 85 as being the best lens they've ever tested.
Thanks, Luka. I believe that you're saying that it's not the DxO sensor tests themselves that are in question, but rather how we interpret the measurements.
Douglas, yes and no. The interpretations are a major source of misunderstanding, but I think it's DxO's fault. They are using nomenclature that is misleading to photographers. And they make things worse by the oversimplified explanations to the measurements they post together with the results.
As for the tests themselves, I honestly don't know. Their more detailed technical explanations of their methodology seem plausible. At the same time their lens ranking and tests are absurd beyond belief. If you looked at a car review site where they had a ranking by performance and you discovered that they've placed a mid range KIA sedan well before all the Bugattis, Ferraris etc.. I mean, what would you think? Could you ever take such a site seriously?
So I don't know. On one hand they seem serious but on the other hand they've posted such ridiculous stuff that it's hard to believe that they are serious. Even if it's not the tests but the scores they assign to the tests, they should have have the common sense to realize that something is wrong when they rank the Canon 50/1.8 (canon's cheapest prime) above lenses such as the Canon 50L, Zeiss 50 MP or when they list the 21 Distagon as the lowest resolving lens that they've tested on a 5DII. If they can't even see how silly that is, then it is impossible to have any confidence in their tests.
I've never really paid any attention to their actual ranking system, but when you click through to look at their actual measurements, I don't really see anything far out of line, even in the lens rankings. I agree that how DxO accumulates the data into some kind of ranking is silly, but the measurements themselves seem ok.
douglasf13 wrote:
I've never really paid any attention to their actual ranking system, but when you click through to look at their actual measurements, I don't really see anything far out of line, even in the lens rankings. I agree that how DxO accumulates the data into some kind of ranking is silly, but the measurements themselves seem ok.
I've taken a look now at the MTF measurements they've done and I'm afraid it looks like they're as silly as the rankings are. Example Canon 50/1.4 @ f/2.8 vs Zeiss 50/2 MP @ f/2.8:
najibs wrote:
If you think Micro 4/3 is great and compact, try a Leica M8 or M9. It will hti you hard in the wallet, but what an experience it is...
I tried it and was not impressed. IQ was great but everything else e.g. ergonomics, strange MF and VF that needs to get used to (it's the normal RF method but it was uncomfortable for me and was much slower than MF in DSLR) and weight (it's small in size but heavy - I didn't like it) mean that it was out of my dream list.
CKrueger wrote:
I don't see M43 as sub-par. Yes the IQ from a FF DSLR could be better. I AM giving up IQ for convenience. But I'm giving up a little IQ for a LOT of convenience.
Well said. I fully agree.
cputeq wrote:
Seems some of you guys are missing the forest for the trees : why argue merits of IQ when the OP specifically has acknowledged ultimate IQ isn't the goal of m43? There are tradeoffs in everything, and I think most people realize when they pick up m43 as a system it's not for absolute IQ.
Yeah, but IMO these conclusions are more absurd than DXO's tests and rankings. If we define convenience by how easy it is to operate the camera, take the photo, and later process, then all mirrorless designs lose and lose big. The mirrorless bodies lack almost all of the convenience features for control and shooting and the images captured need a bit more processing time per image on average than most APS images and a lot more than images from a FF sensor!
So where's the convenience factor? What, just limited to what or how much one can fit in their camera bags? Really? Give me a break. And even that is really only applicable if you limit yourself to the kit lenses. As soon as you put some nice alt grass on the system you're right back to the old sizes and weights. And not only that but now the ergonomics, handling, and balance are all shot to hell! For example put that famous Zeiss 100MP on your GH2 (admittedly the best for this over say the GF3 - lol) and shoot all day with it. Then do the same thing with any of (5D2, 1D3, K5, D700, D3, A850, A900). You come away thinking (actually knowing!) that the bigger bodies were more convenient or you'll have to redefine the meaning of the word convenience in order to justify it otherwise.
So, IMHO m4/3 convenience is limited to how the strap feels on one's shoulder and that's a pretty lame-arse excuse for giving up IQ. Now, if one were to say adaptability in place of convenience then I could certainly buy that. I shoot with M4/3 specifically for that reason. In my particular case I'm sacrificing all of the convenience and IQ that a FF or good APS system would offer just for adaptability - as my goal is to mount many different kinds of lenses.
In conclusion, if one is really willing to sacrifice IQ for shoulder-strap-convenience then M4/3 isn't really the best choice - qualitatively and qualitatively! It's probably something more like this:
I've been reading people say that the M4/3 cameras are "more convenient" for over a year now and I've twisted my head every which way in order to try and see how they were coming to these conclusions but in the end I find it to be little more than mass-dillusion. Either that or word-redefinition.
It's kind of funny that the only solution that offers a compact body and a FF sensor is trapped in an expensive anachronistic manual focus range finder body.
Don't take me wrong, I love the M9 and the way it feels to use. However, rangefinders were replaced by TTL cameras for a good reason as were manual focus lenses for auto focus lenses. There's no getting away from that the M9 is an oddity in today's world where convenient fully automatic TTL cameras are the norm.
Yep, agree. I haven't even considered that option tho. Mostly because of the price - which I also find perversely odd. Not the price but rather that I can 20-spot to 50-spot my way into about $20,000 worth of lenses but not fork over even $1k for a good body. That's another advantage to the panny m4/3 - for me at least, I've always been able to find a steady supply of them for well under $500. The GH1 is currently selling new here for like $250. I would never (EVER!) have considered the GH1 nor the GH2 at MSRP. OMG, how absurd, $1500 to $2000 for that? It's a really bad joke!
With regards to the DxOMark sensor data I think denior has it right about being unsuitable for the target audience at most every level.
The best data they have are their full SNR curves. Great if you are an engineer and can interpret them, not useful to most folks. You can't compare these on their site, so it can be a hassle to view differences between sensors. Still not perfect either, they don't distinguish between random noise and pattern noise because it is hard to define a metric for that.
Next are their more distilled curves, 18% SNR and DR, that you can compare between cameras. These are named and described in a way that doesn't match what many photographers think about. In many cases at high ISO what people object to is shadow noise, but their "noise" curve (18% SNR) has nothing to do with the shadows, really the DR curve is a better stand in for shadow noise. The DR point they select is arbitrary (0dB) and not photographically relevant and this results in that curve often deviating from photographic reality. This is why the 5DII scores as low as it does on DR, they are sampling the SNR curve too low and as a result the 5DII read noise overly penalizes the camera compared to other sensors who in fact have similar SNR at a real photographic shadow level but a better 0dB point (remember, past read noise the curve drops faster).
Finally there are their scores, which are difficult to fathom with names that are a bit silly. The "landscape" score is based on their DR measurement at base ISO, but of course frequent sensor problems in landscape are sky noise in the midtones so base ISO 18% SNR is often more relevant. The "sports" score is an ISO based off the 18% SNR curve, when in fact since shadow noise is the more frequent high ISO complaint they'd be better served by the DR curve!
Anyway, neat data to play with and seems to be taken with reasonable consistency. Based on the frequent number of posts on this forum and others where photographers have made completely erroneous conclusions from the various DxOMark scores and curves I'd have to say the evidence is they aren't doing a very good job of presenting the data to the community even if they are doing a good job measuring sensors.
denoir wrote:
Your second sentence is pretty much an indirect explanation of you first sentence.
An MTF chart shows you certain very important aspects of the rendering style of a lens. If you care about the final look of the image knowing the MTF of a lens will help you predict what that look will be like. It's of course not the only aspect of the rendering style of a lens and certainly not the only thing that determines the final look. The aspects of a lens that an MTF chart shows do however have a significant impact on the final image.
I understand that the charts measure something real. I'm just saying that it's not useful when you could just as easily examine actual photographs. I suppose if you can't get your hands on sample images (or a lens to test yourself) an MTF would be useful to tell you just how smeared the corners might be, but in the Internet Age where reviews and sample photos are a Google search away why bother poring over charts to learn only one aspect of a lens' performance?
This is why I don't find MTF useful. It's not that I don't care about lens performance. I just don't think it's a useful measurement, like horsepower per liter or 3DMark. It's better to examine the actual output of the product... like 0-60 time, application frame rate, or--in the case of lenses--a photograph.
Bifurcator wrote:
Yeah, but IMO these conclusions are more absurd than DXO's tests and rankings. If we define convenience by how easy it is to operate the camera, take the photo, and later process, then all mirrorless designs lose and lose big. The mirrorless bodies lack almost all of the convenience features for control and shooting and the images captured need a bit more processing time per image on average than most APS images and a lot more than images from a FF sensor!
As someone who has shot extensively with FF (5D, 5D2), APS-H (1DMk2N), APS-C (too many to list), and Four Thirds/Micro Four Thirds (again, too many), I disagree that smaller sensors require more processing. Some cameras require more attention in post processing than others, but sensor size is not a factor.
If I wanted to shoot straight ouf of the camera, I'd happily choose my old E-410 over my 5D. Images straight out of the 5D are like a lump of clay: featureless, but easily molded. On the other hand, images from my E-410 (and most of my other Oly cameras) look fairly finished straight out of the camera. The only time my E-410 would require MORE processing is if I blew the exposure at high ISO... the E-410 tended to fall apart if I had to pus/pull the exposure much and had shot at ISO800+.
Notice I'm not saying the E-410 made BETTER pictures than the 5D. Just that the E-410 images came out a lot closer to presentable than the 5D's images, even if I shot 5D JPEGs and played with the JPEG settings.
As for convenience features, you'd have to be specific about what SLRs have that mirrorless bodies lack. About the only thing I can think of is an OVF?
Bifurcator wrote:
Yep, agree. I haven't even considered that option tho. Mostly because of the price - which I also find perversely odd. Not the price but rather that I can 20-spot to 50-spot my way into about $20,000 worth of lenses but not fork over even $1k for a good body. That's another advantage to the panny m4/3 - for me at least, I've always been able to find a steady supply of them for well under $500. The GH1 is currently selling new here for like $250. I would never (EVER!) have considered the GH1 nor the GH2 at MSRP. OMG, how absurd, $1500 to $2000 for that? It's a really bad joke! ...Show more →
I'd like to mention that the 17-40 was exceptionally sharp in the broad central image area, especially for a WA zoom -- the 16-35 is in the same league there, IMO.
You see the problem. 
