Upload & Sell: Off
I spent quite a bit of time with this recently as a new 1dmkiii owner, maybe others that have used the systems longer can add better long term additional info.
It seems that the MA depends on a lot of things, bright vs low light, the distance and focal length the lens is shot at. Just those three factors alone change the results of the ideal MA # from my testing, usually not by much but it does vary. That's without even changing methods of the analysis testing itself.
I did the "shotgun" approach and tried to average the results to get the best function over the widest range of use. Of course if I was a portrait photographer or all landscape I would bias the testing to what I normally shot, but i shoot a bit of everything.
I did the angled ruler thing, and adjusted for where I wanted my ideal DOF, recorded the results, in both low/high light, and at different focal lengths for zoom lenses. Both handheld and on a tripod.
I did the flat target at various distances, lighting and focal lengths, again handheld and on a tripod. Using both the bracketing method and also using live view to focus and shoot the target then using the AF and comparing the images and the distances on the lens reported.
Then I went outside and just shot things, again doing the live view to get the best manual focus image I could get and then shooting the same image with the AF, both handheld and on a tripod.
Honestly for my camera I found there was more variation most times in the lighting conditions than in distances or zooming over a lens. Focus would shift on the same target that was dimly lit or brightly lit. I know I've read that only bright light targets and tripods should be used but to me you have to test the system how you use the camera, and we don't always shoot on a tripod or in super bright light.
In the end most of the numbers across the testing were pretty close in the case of the same lens, for example my 24-105 ranged from +3 to +7 in all conditions, my 100-400 from -1 to +2, and my 50 1.8 from +2 to +4 So in most cases I just split the difference and called it good. All those lenses were previously calibrated by canon to a 30D. So it seems even canon's calibration has say 3-4 MA points of give in it.
The only odd one was my sigma 12-24 that was still pretty consistent but ended up being +14.
Perhaps one of the best things it shows is how well the lens is calibrated, for example one would expect the resultant MA #'s to be close if the lens has a good calibration across various distances/focal lengths. Obviously they are not going to be perfect but if there was a wide swing it might indicate the lens itself is badly calibrated and obviously the MA cannot compensate for that since you only get one value per lens.
So in the end perhaps what it does best is give the owner a more precise method of tunning the lens (assuming the lens is well calibrated) to the individual body. So in the past say with my old 30D you had two variables. Your body might range from say +/- 3 and the lens in acceptable calibration might be +/-3. Now you can at least adjust the body to be as good as it can be for the lens, but you can still have lens variation, and it can't compensate for a poorly calibrated lens, especially one that is not consistent in it's poor calibration.
In the end it's better than not having anything calibrated as in the past, but probably not as good as sending in your camera and all lenses to have them calibrated to each other. I think it can however show a poor lens that needs calibration when if doing the MA testing it's way out from the group such as my sigma is, or the ideal MA number varies a lot from test to test. Obviously they will never be perfect we know that lenses especially zoom lenses have sweet spots in both focal length and distance.