Arvutigraafika projekt - Kursused - Arvutiteaduse instituut

Motion Tracking in AE

Mehdi Hatamian

Link to Final Result

What is motion tracking?

Put simply, motion tracking is the process of tracking the movement of an object in the footage.

How does the procedure work?

The process will be shown by Adobe After Effects. Firstly, we collect track data from a selected point of footage. Secondly, apply these data to an object or element to match the movement of footage.

Some examples: We can create an effect or a composition that will follow a motion in a selected piece of footage or when you want to map a text to an object within your footage. You can even replace some information with another in your footage such as computer screen information with one that you are interested in or when you want to add a 2D or 3D object in your footage.

What we use motion tracking for?

By utilizing tracked data we can stabilize a motion
Add elements such as text or solid layer to a composition
Insert 3D objects into 2D footage
Apply effects or color grading
Replace screen on a TV, computer and etc.

Type of motion tracking:

Single-Point Tracking
Two-Point Tracking
Corner Pin Tracking
Planar Tracking
Spline Tracking
3D Camera Tracking

Some simple definitions:

What is a composition? In Adobe After Effects, to create a project we need a composition. A composition is a container where all possible layers will be put there.

What is a solid layer? A solid layer is a layer in the size and color of your choice. Usually, we create a solid layer in a composition for a background of an object or to create graphic elements.

Foreseen difficulties:

The difficulties of this project are mainly about finding appropriate footage to mostly cover all types of motion tracking and the power of hardware that it might be a challenge. Accordingly, if I am going to record a video with my camera then the quality will be challenging too. Additionally, in some cases, the power of hardware might be challenging because rendering is very important when it comes to motion tracking.

Milestone 1 (05.03)

The first step is to cover my knowledge to master the topic in detail. Thus, I will do some research about motion tracking.
The second step is to find an appropriate video for the first type of motion tracking that is single-point tracking.
I will focus on single-point tracking and I will cover this topic at the beginning of this project.
The goals of the first week is to add the followings in a composition for motion tracking:
- Adding Text
- Adding a composition in the form of nested composition(an object which is animated)
- Adding a shape to cover the tracking object which is a vehicle

Milestone 2 (19.03)

Going to work on Two-point tracking and maybe corner pin tracking
By two-point tracking, we are able to apply scale and rotation to our element which is going to follow a specific element in the footage
Searching free stocks to find appropriate footage
acquire knowledge in the above field

The final result is:

https://courses.cs.ut.ee/MTAT.03.328/2021_spring/uploads/Main/Two-point%20tracking.pdf

Milestone 3 (09.04)

Get to know what is options for motion tracking
Get familiar with Mocha
Using Mocha for tracking (Investigating Planar Tracking)

What are the options for motion tracking? These options are mainly for improving the tracking process

How we can use the channel?

RGB
- it will Find differences in color value
Luminance
- It will track something with different brightness surrounding
Saturation
- It is related to high concentration of color regarding the variation of the same color for example when there is a red color spectrum in a specific element in our footage.

Other options are:

Enhance before Match
- It will temporarily sharpen an image to track better.
Track field
- It will double the frame rate temporarily of the composition to analyze it better.
Adapt feature on every frame
- Adobe After Effects will adapt the tracked feature to each frame and Adaption is based on the previous frame, not the beginning frame

Drop-down menu

Stop
- If the confidence is below a given threshold then the process will stop.
Continue Tracking
- This option will ignore the threshold
Extrapolate Motion
- This will estimate the position of the feature region and it will delete attach points for low confidence frames

For planar tracking, I searched youtube videos for Mocha but what is Mocha?

It is a separate application that introduced by Imagineer System in 2010 and has bundled with Adobe After Effects nearly 10 years

Mocha is an advanced tracking tool that has different UI and workflow and is mainly for planar tracking and works perfectly as a masking tool. Additionally, this plugin will read footage directly from After Effects.

What are the uses of Mocha?

Add objects into footage
Remove object from footage
Mask and track shapes
Replace screen contents Or any other object such as replacing the sky and etc.
Insert 3D elements into the scene

What is the main approach? The main task for tracking is done by selecting around an object by spline that is similar to the pen tool in other applications. However, it is also possible to change handles and creating keyframes in all frames, and selecting is mainly by:

X-Splin
Rectangular
elliptical

What Mocha can track?

Different motions that mocha can track are:

Translation
Scale
Rotation
Shear
Perspective

Milestone 4 (23.04)

Showing an example by using mocha in a footage
Isolating an element from footage (Rotoscoping)
Removing an object In the footage (Optional)
Presenting different example to show what is the ability of Mocha for VFX in movies (optional)

For this milestone, I decided to prepare an editing video by Mocha and After Effects. I used planar tracking and rotoscoping. I have recorded the video by myself. The tablet and videogame have been added to the footage by tracking in Mocha.

Moreover, I have used rotoscoping technique by Adobe After Effects. Rotoscoping provides you the ability to isolate an element from your footage. I have used Roto Brush Tool in Adobe After Effects.

First, we need to import footage into After Effects then lunch Mocha within the software. Then we use mainly the Spline tool for selecting around of our object.

Because planar tracking is related to the corner pin, then we use the planar surface tool.

The planar surface can not have any keyframe, while the spline tool itself can have a keyframe, and we can modify it. The planar surface represents the tracking data. It is not physically the tracking data but represents tracking data, and This gives a place for the searching area of tracking. It will generate the corner pins. This surface is completely independent of the tracking data.

While holding a book just as a tablet, then we should track the book to cover it with a fake tablet.

Select around the book and then check the perspective option for tracking.

Track the book and adjust frames where tracking is out of the element.

For sending back data to the After Effects, we can use export data to copy them into the clipboard.

Then by using a solid layer, we can paste all those data into a solid layer. Additionally, we should adjust the solid layer to fit the size of the book and even a little bit bigger.

Final result:

Milestone 5 (07.05)

Learn about mesh tracker in Mocha
Showing the power of mesh tracking in a footage

What is mesh tracking?

Mesh tracking is another powerful feature by Mocha. Imagine you want to add an element to the footage and place it on a shirt of somebody who is walking or running. In this case, we cannot use the planar surface alone regarding warping and deforming surfaces.

Mesh tracking is helpful for warping surfaces.

This feature is called Power Mesh in Mocha. In another case, imagine somebody is smiling, and lips are moving around in different warp ways. However, in this case, flat tracking or planar tracking ability inside Mocha is not enough.

Again we use the spline tool to select around the face and lips to detect all movements in this area. If we use perspective for tracking, we can get some distortion, especially when the chin moves in the form of a little skew. However, we cannot get curves or the distortion of the lips close to her/his mouth. To achieve this, we can use Mesh to generate a mesh, especially around the mouth.

As you can see in the below images, the area will be divided into smaller parts to detect distortions or warps.

Mesh tool has some features:

Vertices on spline:

By this option, we will get a broader mesh shape inside the selected area.

Adaptive contrast:

It will help to find more details in the selected area.

Uniform mesh option:

It will generate a uniform mesh, and we can use it, for example, for painting or something that we need to be more accurate.

Auto smoothness:

It lets Mocha guess how much motion to track within the plane. We can call it rigidity factor, and 100 percent shows very rigid, and 1 is non-rigid.

Warp spline:

It makes Mesh distort the spline as we are tracking, and we do not have to keyframe spline.

The way the mesh tracker works is that first of all, it does planar, or surface tracking, then tries to subdivide the plane and work out the individual motion inside that planar surface.

We can get the motion of the plane and also getting distortions, and as the mouth starts to love around, it is going to follow the distortion of the lips around and also distort spline along with that Mesh. We can get the spline and mesh adjusting to the distortion of the face.

In this example, I have chosen good footage to cover the topic. Firstly, I have detected the area for tracking by the ability of Power Mesh.

Then I have tried to add an element just like a tattoo.

and adjusting the tattoo by scaling it down and choosing the appropriate color mode that is multiplying and this is the final result:

and please watch the final result in full-screen mode:

Milestone 6 (21.05)

Search and learn 3D Camera Tracking
Show an example of how it works in a footage

 ----------------------

3D camera tracking

3D camera tracking will be generated by Adobe After Effects and Mocha is also able to do camera tracking by Camera Solve Module. Additionally, this tracking is highly dependable on high-resolution footage and the amount of shaky camera. There are three main settings:

Pan tilt zoom shots: when the camera is on a tripod that is not moving
Small parallax shots: where the camera is moving distant from the subject and there is little parallax in the scene
Large parallax shots: Where the camera is moving close to the objects being filmed to create a large amount of parallax

The camera solve module is used to take 2d planar tracking information and convert it to 3d space.

However, in this milestone, all tracking is done by the ability of Adobe After Effects itself. This solution is useful for inserting 3d text, 3d objects, and particle effects.

By selecting 3d camera tracker from the animation menu then we let After Effects analyze our footage.

 All frames will be analyzed.

After analyzing little crosses (track points) just such as tiny icons will appear on the screen as shown below.

By hovering the mouse over this crosses a plane will appear

If we need some crosses that are so tiny then we can adjust the size by 3d camera tracker options.

This can be done by change the value of track point size and if we want to change a specific track point size then we can select it and change the target size

The graphic should be aligned up with the target we choose.

Set ground plane and origin: The graphic which we want to import is going to have a position setting of 0,0,0, for x,y,z, and in fact the position will be set to 0,0,0. In fact, it will properly orient graphics in the scene by defining a ground plane.

The final result is shown below.

However, as you can see the final result is not bad but not perfect. Because of the low resolution of the footage, the text is also a little bit shaky and is not properly tracked. It indicates that the resolution is important and how hardware capability is important for rendering such track points.

Optional

For this section, I have prepared an interesting task.

Removing an element in your footage

How the remove module works and what it does and doesn’t.

It uses a very clever algorithm to examine planar data behind the object that we are trying to replace. In fact, Mocha will use background pixels to replace the foreground pixels. To generate a removal object first we need to select around the shape by spline tool (Roto Shape)

Select a frame where the car is least blurry and if the car is approaching from a distance then let it arrive closer to the camera and get bigger. You can adjust Roto Shape by adding keyframes for shadows because the Mocha algorithm is able to detect it. From the motion panel, we only need translation, scale, and rotation. When tracking of the vehicle is done then we need to track background. Simply by using the spline tool select around the road to track it.