VFX Back to Basics Series: 3. What is tracking and Match Moving?

This is part of a series on the basic elements of Visual Effects. Each post will talk about a certain element which is one of the basic bricks used for building VFX shots.

In this third post I will talk about tracking and match moving.

some of the tracking markers which I use on set. Eric Alba designed these markers.

We can divide tracking into two categories. There is 2D tracking mainly used for adjusting live action in the comp and 3D tracking for adding 3D generated imagery to a shot. The last category is also known as match moving. I'll start with 2D tracking as it is the base for the 3D kind.

Very often compositors need to follow certain objects or shapes in a shot. A good example is a billboard with some commercial message which the client likes to have replaced by something of his own. If the camera doesn't move then it is very easy to place the new image over the billboard and we can call it a job well done. Most of the time though directors do like to move the camera around to make those money shots. A solution would be to move the new image manually frame by frame to keep up with the movement. This is very time consuming and not too pleasant for the VFX artist. Instead we use a tracker and let the computer do the tedious work for us at a much higher speed. Another common use for tracking is to stabilize the backplate. Sometimes the movement of the camera is unwanted and instead of trackers following the image, the image tries to follow the stationary points. This is slightly oversimplified but brings the point across. Have a look at the image below and I'll explain how a tracker works.

A typical 2D tracking point with the tracking center and two boxes for the search pattern.

The main components are a tracking point, right in the middle, and two boxes around this point. The central tracking point is the coordinate of the tracker on your image. The smaller box is is the region which the computer tries to match when comparing frames and the bigger box is the search area in which the computer will make its search when trying to find the match for the smaller box. Choosing the location of your tracker is rather important as the computer needs a pattern which it can recognize. Placing the tracker on a solid color for example is bad as there are several positions on the image which will generate a match. Use corners of objects or high contrast pixels which can be provided by carefully placed tracking markers (see image at the start of the article).

Example of a bad tracking point. The pattern in the sand is not contrasty enough and the tracking point will probably start sliding when the computer tries to find a solution.

Once the computer has figured out the motion path of the marker it is possible to attach an image to that marker. The image will now follow the motion of the backplate. If the movement is only a translation then a single tracker is enough. If the backplate also rotates you need a second marker which combined with the first marker will find you the proper rotation. Stabilizing a backplate which was shot handheld is a good example where you need at least two points. If there is also a scale or perspective change then it is best to use four points. Adding more points usually gives better results.

Let's move to the 3D realm again and I'll show you how we can use the above technique to our advantage to extract three dimensional information out of a two dimensional image.

The world around us is a three dimensional space but when we shoot film or pictures that world is converted to a flat image. In my previous post I was talking about how convenient it is to use 3D modeling for Visual Effects but how can we place those objects in the scene if our source is only a two dimensional image? Luckily some smart mathematicians came up with Photogrammetry. This fancy word means that we can deduct the measurements of the objects in the photo when we know at least some measurements on a couple of objects in the scene that we are photographing. With these measurements we can rebuild the whole scene in a 3D space.

While filming with a moving camera there is an added benefit. When seeing the same objects from two different points of view and two frames from a shot with moving camera provide exactly that, we can use the power of triangulation. This concept is also called parallax. The more depth there is in the scene the better this will work as objects on different distances will appear to move different distances. An extreme example of parallax is watching the moon while sitting in a moving train. The trees close to the track whiz by but the moon doesn't seem to move at all.

A typical automatic track with lots of tracking points chosen by the computer.

3D tracking is done by the help of specialized software. Matchmover, Boujou, PFTrack and even NukeX can do the job quite well. These programs can track a shot automatically but if the result are not satisfying then it is possible to add tracking points manually. Same rule applies here too, more tracking points give a more accurate result as all the little mistakes get averaged out. The tracking points do have a quality value added to them which is compared to the average value. When they exceed a certain error margin they will be dropped from the results. It is also possible to remove an automatic tracking point manually if you think it gives bad results.

The ultimate match move is to track actors and their movements such as facial expressions. By putting a grid on the actors face it is possible to follow the facial features and then use them as a base for a CGI face which will replace the original face of the actor.

This concludes the third part of the VFX Back to Basics series.
Make sure to subscribe (at the top) or follow me on Twitter (check the link on the right) if you want to stay informed on the release of new posts.

In this series:
1. What is rotoscoping?
2. What is 3D modeling?
4. What is 3D animation?
5. What is shading and texturing?
6. What is lighting and rendering?
7. What is matte painting?
8. What is compositing?

Useful links:
Eric Alba's website.