Computer Vision: Motion and Optical Flow

 A property of a scene that I want to recover is motion. Given a sequence of images, we would like to estimate motion to see how things are moving with respect to the camera. An example is a camera surveillance system, or tracking the motion of facial sequences. Measuring the motion of a camera itself is important. We also need to track the motion for autonomous cars, etc. 

Both cell phones and computer mouse have a tracking system which is based on estimating motion, also tracking facial features. 

Let's first formalize what to measure.

Suppose there is a scene point, and it is given by image location. Given a time direction 𝛿t, how does the diagram move? This 3D displacement vector gets mapped with a 2D vector in the image space. A collection of 2D image space vectors is called the motion field images. These 2D image motion vectors are related to 3D vectors via the laws of perspective projection. Motion field is what we would like to measure.

The optical is the direction of the motion of a brightness pattern around a pixel, and the length of an arrow in the flow diagram is the magnitude of the motion. 

Suppose there is a sphere that is spinning. If we take a series of images within time, all of them will look alike because the sphere is completely uniform. But clearly an object is moving, so there is a motion field. However, if a light source is moving, there is optical flow, but no motion field. Therefore, optical flow is not the same thing as motion flow. In general, we do assume that the optical field is the same as the motion field. 

We start by formally defining optical flow. Let's take a particular pixel at location (x,y) and now the new coordinates of the pixel are (x + 𝛿x, y + 𝛿y) and the optical flow of the pixel is (𝛿x/𝛿t, 𝛿y/𝛿t) (displacement is (𝛿x, 𝛿y)) and the optical flow field is a set of optical flow vectors, one for every picel in the image. To do that, first define constraints on the optical flow vectors. We first assume that the brightness of an image pixel remains constant over time. I(x + 𝛿x. y + 𝛿y, t + 𝛿t) = I(x,y,t) which is the relationship of the intensity of a pixel before and after motion.

The second assumption is that the two images are taken in rapid succession meaning that time step 𝛿t is small, and the displacement (𝛿x, 𝛿y) is also relatively small. How is this useful? We can think about image intensities as a 3D function, dependent on 2 spatial coordinates.  I(x + 𝛿x. y + 𝛿y, t + 𝛿t) = I(x, y, t) + (𝛿I/𝛿x)𝛿x + (𝛿I/𝛿y)𝛿y + (𝛿I/𝛿z)𝛿z, which is a Taylor First-Order Approximation. The value of a function of a point can be approximated by the value at the neighboring point added with the derivative of said function. 

For the Taylor First-Order Approximation, we need to assume that the function is continuous. In theory we want to assume high enough resolution for continuous functions. This will only work if the images are smooth, without strong step edges. This is why these methods have problems around object boundaries. Ix is the horizontal image gradient and Iy is the vertical gradient.

We want to get Ix 𝛿x/𝛿t + Iy 𝛿y/𝛿t + It = 0. U and v are the optical elements and Ix and It are the flow gradients/vectors per pixel. 

There are constraints for images, but we need additional constraints needed in order to solve images. There is one constraint that still fulfills modern optical flow methods that is covered here. 

The Lucas Kanade Solution means that the optical flow is the same for all neighboring pixels in a small window, assuming that Ix(k,l)u + Iy(k,l)v + It(k,l) = 0, so now we have n^2 equations, and 2 unknowns can stack these equations into a series of equations and subsequently solve this system through a least squares solution. We made the assumption that the motion vectors for all the pixels in a small window will be the same, which is why motion vectors are similar for pixels that are neighboring. 

There is a change of intensity along the x and y dimension, as well as a change of intensity in the time direction. Think about an edge that has an intensity gradient, and moves around the edge direction. These intensities will be cancelled out as a result.  

Now to recover an optical flow vector, just do this matrix: 

 For this system to be solve reliability, det(ATA) != 0 and ATA must be invertible, and well-conditions. All the eigenvalues should be comparable and large at the same time.

A matrix with similarr intensities have the gradients near zero, which would make the eigenvalue equal to zero because the image gradient values summation will equate to zero, and it will be extremely hard to compute because there is no real change in tensities. 

If the pixel moves along an edge, the eigenvalues are drastically different and we can only recover the motion that is perpendicular to and edge. Now if there are many gradients and full rank, we can reliably compute optical flow. 

We want to take a difference of the sum of pixels after certain amounts of averaging, We do this for both the horizontal and vertical derivatives. 

We take the difference between k,l and k + 1, l and we perform some averaging and then subsequently take the difference over 2 of the pixels. We make sure the 2 images are captured. Similary, we want to find Ix(k,l,t), Iy(k,l,t), and It(k, l, t). If the camera moved rapidly, the amount of motion may be large. 

 We keep doing course to fine estimation until we reach the original resolution. We start with low resolution version of the original image, and assume that the motion between 2 images is very small, then get the recovered motion, and go one scale up, and warp the image by using the motion. The amount of motion between image is small, so we can use a linear equation to estimate motion, and add 2 motions together for warped, and repeat this until we get to final state of the image.

OF represents the recovered motion, and then go scales up and warp images by using a certain motion vector. The amount of motion between warped image and then use linear equation to estimate motion, and repeat this process until we get to the final stage, then we add all these vectors together to compute the final flow for these images. This coarse-to-fine approach can actually recover the motion quite accurately. This method looks extremely computationally intensive, heavily depending on the number of levels that are there inside of the hierarchy. There are chips whose only purpose is to compute fast optical flow, especially in applications involving motion. 

For each template window T in image I1, find the corresponding match in image I2. We can use the sum of absolute difference of values, etc. This is simple to implement, but can be computationally very expensive. Template matching is slow, though when the search window S is large. 

On the other hand, the Lucas Kanande approach is for interactive applications that are speed critical.

We assume the appearance of the pixel is the same for 2 images. If something drastically changes over time, these approaches are not going to be valid. We do assume the brightness or the appearance of a pixel in its neighborhood will appear more or less the same. 

For example, tracking the motion of the hands using Lucas Kanade. Optical flow is used to determine the immediate frames when producing slow motion effects.