Home · Register · Search · View Winners · Software · Hosting · Software · Join Upload & Sell

Moderated by: Fred Miranda
Username   Password

  New fredmiranda.com Mobile Site
  New Feature: SMS Notification alert
  New Feature: Buy & Sell Watchlist
  

FM Forums | Canon Forum | Join Upload & Sell

  

Archive 2012 · How does 5D3 track subjects? (Object recognition)
  
 
trale
Offline

Upload & Sell: Off
p.1 #1 · p.1 #1 · How does 5D3 track subjects? (Object recognition)


Can anyone explain, at a highly technical level, how the 5D Mark III (or 1DX) track moving subjects in AI-servo?

I know how phase-defect AF works, so I'm not asking about that. I'm asking what processes the camera uses to determine that an object is the same object moments later (after it moves), and thus intelligently maintains focus on that object.

The 5D3 has many tracking parameters in its AF menu, where you can set the tolerances for tracking sensitivity, acceleration / deceleration tracking, AF-pt auto switching for us to fine tune the current shooting situation.

Underlying all this must be a sophisticated algorithm to determine and isolate objects from other elements in the frame (including other objects). I'm guessing that it uses a combination of the following:
- Color information (hue, saturation)
- Outline of object (from what is currently in-focus in the frame)
- contrast in the scene
- Luminosity of various parts of the scene
- Distance of the thing that is currently in-focus
- Maybe pre-defined common shapes for comparison purposes?

Assuming (some) of the above are correct, it'll be nice to know how exactly those data are analyzed to determine an object. And once the 5D3 has determined that something is an object to be tracked, how then does it go about predicting/tracking its movement? Assuming a difficult object is constantly changing shape, direction, velocity, and the portion of the frame the object fills - for example a line-backer charging at the camera while juking to avoiding the opposing team's players.

If anyone has some good resources on this topic, it'll be greatly appreciated. The practical benefits of having such in depth knowledge is that then I can better predict how my camera will handle a certain real scene, optimize my AF tracking settings, and ultimately get more keepers.



Sep 26, 2012 at 12:02 AM
RobDickinson
Offline
• • • •
Upload & Sell: Off
p.1 #2 · p.1 #2 · How does 5D3 track subjects? (Object recognition)


AFIk the 5d3 doesnt use any colour/luminosity info from the metering system because its too simple vs 1dx.


Sep 26, 2012 at 12:19 AM
Access
Offline
• • •
Upload & Sell: Off
p.1 #3 · p.1 #3 · How does 5D3 track subjects? (Object recognition)


Think about it again for a sec, realistically it can't use things like color information because it isn't in live view mode -- the sensor is blocked by the mirror and all.

Basically think of it turning on all the focus points simultaneously and working from there. Setting up the different parameters according to your sport or activity is one thing you can do, and this can help. But mainly it's trying to follow the target as much as possible on your own, rather than relying on the camera to do it for you. The more you watch your sport, the better you can predict this, or the better you know what is going to happen next in a play (sometimes). For instance, in your example, do your best to keep that linebacker in the center of the frame as he jukes to the right and left.

Once you get good at this for your given sport, you can use zone focus and just keep the focus zone over the target's face or upper body. That's what I've found works best, don't use more focal points than you need to; sometimes if I am using a telephoto lens I will do the keep your other eye open trick to keep a wider view of the action.



Sep 26, 2012 at 12:57 AM
RobDickinson
Offline
• • • •
Upload & Sell: Off
p.1 #4 · p.1 #4 · How does 5D3 track subjects? (Object recognition)


Access wrote:
Think about it again for a sec, realistically it can't use things like color information because it isn't in live view mode -- the sensor is blocked by the mirror and all.


1Dx has the metering system (which sees in colour) plumbed into the af tracking system (as did the 7d I think, and as a lot of nikons do).



Sep 26, 2012 at 01:01 AM
Pixel Perfect
Offline
• • • • • •
Upload & Sell: On
p.1 #5 · p.1 #5 · How does 5D3 track subjects? (Object recognition)


The camera can measure a direction and speed of the object under the AF point and uses predictive algorithms to estimate where the subject will be slightly ahead in time based on it's current measurements and focuses at this updated point in advance taking into account shutter lag. If the subjects alters course and/or speed it will need to update it's information for an improved estimate. This is why they have spent so much time improving how quickly the AF can react to sudden changes in the subject speed and trajectory and the 1D X can also make use of colour data, but apparently only when automatic AF point selection is enabled.

Ohter factors have to be taken into account such as contrast, lighting, lens used.


Here is alink to an old Chuck Westfall article about AI servo works from the days of the EOS 1, but the principles are exactly the same on the new DSLRs

https://groups.google.com/forum/?fromgroups=#!msg/rec.photo.equipment.35mm/pT2PYfobRCM/tiasWvrPCB0J



Sep 26, 2012 at 02:10 AM
JimboCin
Online
• • •
Upload & Sell: Off
p.1 #6 · p.1 #6 · How does 5D3 track subjects? (Object recognition)


I will tell you what I think I know, or what I understand at least about how AI Servo works in the Canon 5D Mk III. I am sure that others will post their corrections to my understanding - and we all will be able to learn from that.

I believe this is substantially different from the more advanced tracking that the Canon 1DX does, which I won't be discussing here.

The best source of knowledge on the subject that I know of is the posting from Chuck Westfall that Pixel Perfect gave us. The understanding that I have that I pose below is somewhat different from what I think I understand from Chuck's post - but I don't know how to reconcile my understanding with what Chuck presents.

Unfortunately what makes its way to us, the end-users, is not very technical in nature - so we just have a cursory understanding of the details. I bet somewhere there are patents that do a pretty good job of actually explaining the technical details. I don't have the energy to to a patent search on the subject, but perhaps someone else has the patents.

On to my understanding of AI Servo in the 5D Mk III.

First, some terms I will use to describe my understanding.

Subject movement can be in the X, Y and/or Z direction. X-direction subject movement is movement that is side to side in the viewfinder. Y-direction is movement that is up or down in the viewfinder. Z-direction is subject movement closer to or farther away from the camera - depth movement.

In AI Servo tracking in the Canon 5D Mk III (and in most other Canon cameras) the first step in the focusing process is for the photographer to select the focus point that the camera is to use. For the sake of discussion, lets say the photographer selects the center focus point to be used. Use of that focus point sets the X and Y position for the focus point. The camera (the focus point) has no ability and does not track the subject in the X or Y directions - it can't. It only sees the subject under the selected focus point - in this case the center focus point. This sets the X and Y directions.

There is no tracking of the subject in the X or Y directions in AI Servo (and I won't get into AF Point Expansion or Zone Focus, but they don't actually track the subject either). I think you might be confusing AF point auto switching sensitivity (which allows the photographer to select the sensitivity at which focus is changed from the selected focus point to one of the surrounding focus points) with true subject X-Y tracking - which it is not. If this is the confusion - we can discuss this in more detail.

AI Servo tracks the subject that is under the selected focus point in the Z-direction - the distance between the subject and the camera. It does this by continuously using the phase-detect AF circuits. It takes multiple distance readings, and then based on those multiple distance readings (the last three distance readings, based on Chuck's information) and the camera's knowledge of how long it takes between the time the photographer presses the shutter button and when the image is taken (its actually a little more complicated than that, but simplistically I will leave it at that for now) the camera calculates the predicted focus DISTANCE for the time at which the picture will be taken. The camera commands the lens to focus at that distance, and when the camera takes the image the lens is focused at the proper distance.

I know Canon (and others) use advanced algorithms to make these calculations, but the basics of this are simple. Lets consider a simple situation where we have two distance measurements from the AF system. With the AF system in AI Servo, we have the first distance point and the exact time that the distance measurement was taken at. Then we have a second distance measurement, along with the exact time that the distance measurement was taken at. We also know the exact time (or almost the exact time) that the image will be captured at. This is the time at which the photographer presses the shutter, plus the shutter lag (again, this is an overly-simplistic explanation). From all this data (time and distance for point one, time and distance for point two, and the calculated time for when the image will be taken) it is rather simple math to calculate the predicted distance from the camera to the subject at the instant the photo will be taken. In reality the calculations are a lot more complex, because quite a number of other variables are taken into account, and Canon uses the three (or maybe more now) most recent distances rather than just the two I used my simplified explanation.

I hope that makes some sense, and I am sure others will help me form a better understanding of how all of this really occurs.

I believe my understanding and explanation is not totally in sync with the Chuck Westfall post that Whayne posted. Perhaps Whayne or others can help me better understand what Chuck is saying versus my understanding.

My guess is that the OP actually wanted to know about subject tracking in the X and Y directions, something that the Canon 5D Mk III does not do (at least as I understand it). I believe that the Canon 1DX (and various Nikon cameras) actually have the ability to do this, but I am unsure of specifically how they do this. Here again, obtaining patents on the subject would be most helpful. Perhaps others can help us both to understand how this subject tracking in the Canon 1DX and various Nikon cameras is accomplished.

Jim



Edited on Sep 26, 2012 at 03:57 AM · View previous versions



Sep 26, 2012 at 02:45 AM
JimboCin
Online
• • •
Upload & Sell: Off
p.1 #7 · p.1 #7 · How does 5D3 track subjects? (Object recognition)


I found the following from Mr. Tomokazu Yoshida, Canon Camera Development Center at Image Communication Products Operations.

"We use enormous amounts of historical data to provide fast, comfortable focusing on subjects that move irregularly.

Predictive AI Servo AF is a useful function for shooting moving subjects. This function looks at multiple records of past focusing data to predict where the subject will be in the next instant. This is necessary because, with phase-detection AF, there is a time lag between when the shutter is released and when the exposure is made because the main mirror and sub-mirror must be raised.

It is possible that the subject moves in the instant the photo is taken when it is impossible to gauge the focus distance. Because the depth of field on digital SLRs is shallower than on compact digitals, the amount the subject may move during this time lag cannot be ignored. The problem then is how to provide fast, comfortable focusing while predicting the subject’s movement during the time lag given all the types of possible shooting conditions, all the types of possible subjects, and all the types of motion. This is the hardest part of predicting motion, but Canon has an algorithm that optimizes focusing on many kinds of subjects using data that we have accumulated over many years. This is both one of the features of EOS SLRs and one of our assets."




Sep 26, 2012 at 03:31 AM
Access
Offline
• • •
Upload & Sell: Off
p.1 #8 · p.1 #8 · How does 5D3 track subjects? (Object recognition)


Okay now that I actually have the camera in front of me, along with the manual:

page 71: "AI servo AF for Moving Subjects
AF mode is suited for moving subjects when the focusing distance keeps changing. While you hold down the shutter button, the subject will be focused continuously.
The exposure is set the moment the picture is taken.
When the AF area selection mode is sest to 61-point automatic select (p. 72).
the camera first uses the manually-selected AF point to focus. During autofocusing, if the subject moves away from the manually-selected AF point, focus tracking continues as long as the subject is covered by the Area AF frame."

The rest of the chapter is a decent read, but good information. Page 85 talks about how to set the parameters for your preferred sport, from Tennis to Rhythmic Gymnastics.

page 88: "Case 5: For erratic subjects moving quickly in any direction.
Even as the subject moves dramatically up, down, left, or right, the AF point will switch automatically to focus-track the subject.

There's a lot of other good info in there, I can't quote it all. RTFM.

Also if it's constantly extrapolating like that, try to avoid shooting unnecessarily, and unless it is your specific intention, try to avoid shooting with or after sudden direction changes.

The only other advice I can give is to study your sport inside and out, watch a few games or performances without photographing so you get a general feeling for what is going to happen next. This comes with all things, but try to put yourself in the head of the athlete, know what he's going to do next so you can follow it and capture that defining moment like it's second nature. If you are dedicated enough, learn to play your sport (if possible, even if only on the novice or rec level) to learn the basics, spend time with other players, just get to know them and the culture, traditions, physical nature, physics and intricacies of the sport. Many sports have concepts like signaling (some type of body language specific to that sport, typically used to communicate with friendly teammates) and reading (being able to interpret what an adversary's next move will be in order to outsmart or outplay him) and understanding these different concepts as well as keeping your ears open can help you get into the mindset of the athletes, capture more defining moments, basically shoot the sport better.

Edited on Sep 26, 2012 at 04:01 AM · View previous versions



Sep 26, 2012 at 03:36 AM
JimboCin
Online
• • •
Upload & Sell: Off
p.1 #9 · p.1 #9 · How does 5D3 track subjects? (Object recognition)


An interesting discussion by Rudy Winston of Canon - here.


Sep 26, 2012 at 03:49 AM
JimboCin
Online
• • •
Upload & Sell: Off
p.1 #10 · p.1 #10 · How does 5D3 track subjects? (Object recognition)


Rudy Winston has another great discussion here.

It presents things differently than what I had understood, but it is too late and I want to read and better understand this later.



Sep 26, 2012 at 04:09 AM
 

Search in Used Dept. 



trale
Offline

Upload & Sell: Off
p.1 #11 · p.1 #11 · How does 5D3 track subjects? (Object recognition)


Wow, thanks guys, especially you Jim. That's a lot of information I'm going to have to digest, but that's what I asked for.

I can see now that I made some erroneous assumptions in my OP. In a way I let some CDAF methodology creep into a PDAF system.

It amazes me even more now to realize how little raw data the 5D3's AF system is actually working with - only what's available under the focus points (or more specifically the tiny parts of a scene that intersects with the phase-detect AF sensors). In my OP, I mistakenly assume that the 5D3 had the entire scene's data to work with - that probably only applies to cameras with only live view / CDAF.

Actually, the 5D3's AF system does have some help (besides what's seen by the AF sensors) for subject tracking... more on that later.

Clearly it's not possible to get a 100% complete and accurate understanding of the 5D3/1DX's AF system at the algorithmic level (that's Canon's protected proprietary code). The computer science nerd part of me would love to read through such algorithms. But even without going to that level, there's lots to learn about it.

And to make my own contribution to this thread, here's something I came across:
http://www.arihazeghiphotography.com/AH_CW_interview/

Here Chuck Westfall explains the differences between the 5D3 and 1DX AF systems,


AH: Are there any differences in hardware or software between the EOS 5D MKIII and EOS 1DX AF modules?

CW: The AF sensor and AI Servo III tracking algorithms are identical for both cameras. However, the EOS 5D Mark III uses a 63-zone Dual Layer iFCL sensor for exposure metering whereas the EOS-1D X uses an approximately 100,000 pixel RGB sensor powered by a DIGIC 4 processor for exposure metering. As a result, the EOS-1D X supports EOS iTR AF (Intelligent Tracking and Recognition) that can apply face and color detection data from the 100,000 pixel sensor for improved subject tracking when the camera is set for automatic focusing point selection. In addition, the EOS-1D X achieves a higher lens motor drive speed with select L-series USM telephoto lenses than the 5D Mark III because of the 1D X’s more powerful battery pack.


Here's a link to details about that iFCL sensor:
http://cpn.canon-europe.com/content/education/infobank/exposure_settings/iFCL_metering.do

And about the RGB sensor in the 1DX:
http://learn.usa.canon.com/resources/articles/2011/1dx_rgb_meter_article.shtml

So If I'm reading that correctly, the 1DX is basically using a something like a small 250x390 full-color shrunk version of the current scene to help with tracking, and this is where conventional object recognition ideas come into play.



Sep 26, 2012 at 05:51 PM
JimboCin
Online
• • •
Upload & Sell: Off
p.1 #12 · p.1 #12 · How does 5D3 track subjects? (Object recognition)


trale:

I did not post anything about the 1DX because your question was in reference to the 5D Mk III. As you have found out, the 1DX has capabilities that the 5D Mk III does not have in the area of subject tracking.

Another interesting post from Rudy Winston can be found here.

Look about 2/3 of the way down, under "Automatic AF point selection – use the entire 61-point AF area"- particularly the section "AI Servo AF mode."

Here Rudy says something similar to what Chuck said in the post you included, with perhaps a little more added:

"One important difference between the EOS 5D Mark III and the EOS-1D X: while both share the same 61-point AF system and essentially the same AF optics and supporting electronics, only the EOS-1D X has the 100,000 pixel RGB metering system, and ability to activate "EOS iTR" – Intelligent Tracking and Recognition.

This system uses not only AF information, but information about an initial subject's color, size and location (from the RGB metering sensor) to help guide the AF system to continually update AF points when Automatic AF point selection is combined with AI Servo AF.

With the EOS 5D Mark III, this automatic switching of points to follow a moving subject is done strictly using info from the AF system, not the metering system as well."

Reading all of this information makes me realize that I was certainly wrong about the 5D Mk III's ability to do subject tracking. It certainly has that ability. Re-reading Rudy's post makes it sound like it may not work that phenomenally well. Maybe I will give it a shot and see how it works for me in my sports shooting.

Jim

PS:I wish I could have Rudy on a hot line to ask all these questions of him. He really does have a great practical knowledge of how all of this works.



Sep 26, 2012 at 11:55 PM
Access
Offline
• • •
Upload & Sell: Off
p.1 #13 · p.1 #13 · How does 5D3 track subjects? (Object recognition)


JimboCin wrote:
Reading all of this information makes me realize that I was certainly wrong about the 5D Mk III's ability to do subject tracking. It certainly has that ability. Re-reading Rudy's post makes it sound like it may not work that phenomenally well. Maybe I will give it a shot and see how it works for me in my sports shooting.

Honestly I've only used it for about 15 minutes in difficult situations, but it worked pretty well when I did. For instance at the beach it could easily track my friends who were in the water with the waves breaking around them, this was just in the default mode. Once it 'locks on' to a subject, it seems to hold it pretty tight. I don't remember it ever losing track or doing something stupid like trying to follow a wave or anything like that.

For general motion, I've had very good luck with just setting zone focus and following with the camera.

Also I've never used a 1Dx or any 1-series, or even a 7D, so I can't really compare. When I was shooting sports before, I used a 20D and this is a world above that.



Sep 27, 2012 at 12:30 AM
JimboCin
Online
• • •
Upload & Sell: Off
p.1 #14 · p.1 #14 · How does 5D3 track subjects? (Object recognition)


Access: Thanks for your comments!

I too had the 20D. I currently also have the 7D (in addition to the 5D Mk III). Reading the 7D manual, it works the same here as the 5D Mk III except for the number of AF points.

I shoot sports extensively. I rarely use zone focus. I almost always use either single point or AF point expansion, as I want to precisely control at least the initial focus point.

There is a real dirth of information about focus tracking on the 5D Mk III. In Charlotte Lowrie's book "Canon EOS 5D Mark III Field Guide" she alludes that in the AI Servo with 61 point selected that the camera will focus on the closest object under any one of the 61 AF points - not exactly what I would call focus tracking (and I have no idea if Charlotte is right about this or not).

In Douglass Klostermann's book "Canon 5D Mark III Experience" Douglas has this to say about using 61 point with AI Servo "It is best to avoid this mode except for action situations where the motion is too quick or erratic and you would not be able to make use of Zone AF." Although Douglas does go on and seems to indicate that the 5D Mk III actually tracks the subject in this mode.

Douglas does not explain why he believes it is "best to avoid this mode.."

I definitely will have to play with this and see what I learn and how I like it.

Jim



Sep 27, 2012 at 01:08 AM
Access
Offline
• • •
Upload & Sell: Off
p.1 #15 · p.1 #15 · How does 5D3 track subjects? (Object recognition)


JimboCin wrote:
I definitely will have to play with this and see what I learn and how I like it.
Jim

I would recommend that over reading too much, it seems like there is a lot of misinformation out there.



Sep 27, 2012 at 01:15 AM
JimboCin
Online
• • •
Upload & Sell: Off
p.1 #16 · p.1 #16 · How does 5D3 track subjects? (Object recognition)


Access:

What specifically are you referring to as misinformation, and how are your experiences different?

Thanks!

Jim



Sep 27, 2012 at 01:25 AM
Access
Offline
• • •
Upload & Sell: Off
p.1 #17 · p.1 #17 · How does 5D3 track subjects? (Object recognition)


The quotes from the books in your above post, for instance.
At least Rudy Winston knows what he's talking about.

But then personally I've never been much of a book person to begin with, they're on the same level as workshops or 'how to' DVDs in my mind.



Sep 27, 2012 at 02:06 AM
trale
Offline

Upload & Sell: Off
p.1 #18 · p.1 #18 · How does 5D3 track subjects? (Object recognition)


Something just came to my mind concerning the iFCL metering sensor of the 5D3

http://cpn.canon-europe.com/content/education/infobank/exposure_settings/iFCL_metering.do

Does this only apply when "Evaluative metering" is used (as opposed to Spot, Partial, and Average metering)?

In other words, if I'm using Spot, Partial, or Average metering mode, is the iFCL sensor is completely irrelevant?

And another thing - it would seem to make sense that Evaluative metering mode is more complex than the others, hence, would using Evaluative metering potentially slow down my burst rate (say under difficult low light conditions)?

On the other hand, would using Evaluative metering (and taking advantage of the iFCL sensor) result in more in-focus shots when I'm shooting fast, erratic subjects?

Does anyone have any real world experience regarding this?



Sep 27, 2012 at 10:39 PM
Access
Offline
• • •
Upload & Sell: Off
p.1 #19 · p.1 #19 · How does 5D3 track subjects? (Object recognition)


I wouldn't 'read too much into things'... I think it applies all the time regardless.

Options like spot, partial, or average metering is just what part of the data (and with what ratio or weight) from that light sensor is used to determine the automatic exposure parameters like shutter speed, ISO (if in auto ISO mode), TTL flash, etc. Nowhere does it actually say that part of the sensor is shut down, or that you have to be in a certain metering mode to take advantage of the sensor data. In simple technical terms, part of the data is disregarded. The same applies to if you set your camera to manual exposure. If it's necessary for the other features to work, the light sensor isn't actually shut down, at least there's no reason to think it would be. It's just ignored for purposes of exposure.

As for metering slowing down the burst rate, I remember something about the 7D burst rate slowing down when it was in dark conditions(?). Don't think anyone has reported the 5d3 doing that, though. In all honestly, there's no reason it should have to based on the complexity of the calculations, microcontrollers can make those calculations in an instant, and the only result is the setting of the automatic exposure. If you push the shutter, the camera should prioritize that over trying to keep the exposure up to date. With iFCL, it's probably the DIGIC (ASIC) doing everything so it's not like it would slow down just because the task is complex.



Sep 28, 2012 at 12:13 AM
trale
Offline

Upload & Sell: Off
p.1 #20 · p.1 #20 · How does 5D3 track subjects? (Object recognition)


Yeah... I guess it'd be a waste to have a sensor that's inactive/useless for 3 of the 4 metering modes.

And I have a 7D so I'm familiar with the slow downs in low light.

Come to think of it, the time it takes to physically drive a lens to the correct focus position probably overshadows any time the metering system might take.



Sep 28, 2012 at 07:27 AM





FM Forums | Canon Forum | Join Upload & Sell

    
 

You are not logged in. Login or Register

Username   Password    Reset password