Computer Vision: Camera Calibration and Simple Stereo

   This is a method that humans use to recover geometries. How do we figure out how far things are from us? One way is to use the size variable. We figure out how far an object is using a method called parallax, or stereo. If something is really close, we can look at a finger alternatively with 2 eyes, and the image will seem to shift. The closer an image is, the farther it will shift. This is called "binocular stereo". This is one of the primary cues in relation to recovering geometry. We want to use computer vision algorithms to recover scene geometry. 

Here's the basic intuition. Suppose there are 2 cameras from 2 different viewpoints. Camera 1 has a particular view, and camera 2 has an alternative views. If we know these corresponding pixels, we can recover the 3D location of the scene point passing through camera center, and see where 2 rays intersect. 

We start with a 3D coordinate system with its coordinates given by a certain vector, and the X and Y axis is parallel to image plane, and the Z axis is perpendicular to the image frame. There is a camera coordinate frame and world coordinate frame.

 The image plane coordinates are [Xi Yi], the camera coordinates are [Xc Yc Zc] and the world coordinates are [Xw Yw Zw]. It takes a coordinate transformation to go from world to camera coordinates and perspective projection to go from camera to image plane coordinates. This mapping can be described in terms of the pinhole projection, or perspective projection.

What we have is a pinhole camera, and we know that xi/f = xc/zc, and yi/f = yc/zc, where f is the focal length of the camera. We need to convert continuous value coordinates to discrete incides, so we convert continuous coordinates with dimensions of the pixels. Most cameras have a square image plane, meaning the height and widths are essentially the same. If pixel (ox, oy) is the principle point where the optical axis pierces the sensor then u = mxf(xc/zc) + ox and v = myf (yc/zc) + oy. 

We are adding an offset to the coordinates where the offset is the location of the principle point. We take these equations and focus on them for a little bit. We combine terms as fx and fy, as the camera focal length in the x and y directions. (fx, fy, ox, oy) are the intrinsic parameters of the camera. 

All the derivation, we place the origin of the camera image point at a principle point, but sometimes we want the origin to be in the top left corner of an image, and to do this shift, we need to add a specific offset ox and oy. 

Ox and Oy are defined in the image plane's coordinate 2D system. 

First we did a proepective projection, then we convert continuous image coordinates to discrete image coordinates, then we added an offset ox/oy to shift the center of the pixel to the top left of the image. These equations are non-linear. Is there a way to write these into linear equations? We want a linear relationship between the 2D and 3D coordinates. We can write these into linear equations using homogeneous coordinates, as a transformation of a matrix multiplication, to convert nonlinear equations into linear equations. 

The homogenous representation of a 2D point is a 3D point, where u = ~u/w and v = ~v/w

and the following i a homogeneous representation of (u,v)

The left matrix is called the upper right triangle matrix (first 3 columns) and the entire matrix is called the intrinsic matrix (depending on the local parameter).

World coordinate (coordinate transformation) -> camera coordinate's (perspective projection) -> image coordinates. This is how we translate 3D coordinates into 2D image coordinates. 

First we need to translate the 2 frame so they have the same origin, then we have to rotate the fram eso that the axes are aligned. The orientation/Rotation matrix is orthonormal. The sum of squares of eacn of the rows is equal to 1. r = [r11 r12 r13; r21 r22 r23; r31 r32 r33] where the row are direction of xc, yc, and zc in world coordinate frames. 

The rows are the directions of the world axes in the camera frame. These are orthonormal vectors where uTv = 0 and uTu = 1. Ornonormal matrix is a square matrix whoe rose or columns are orthonormal. RTR = RRT = I. R^-1 = RT. It'll be great if we can write the whole thing as a matrix vector multiplication using homogeneous coordinates, with 4D coordinates, we can write both rotation and translation into a single combine matrix. 

It turns out in homogenous coordinates we can write both rotation and translation into a single combined matrix. This is called the extrinsic matrix, in terms of rotation and translation. 

This is the world to camera, and camera to pixel, respectively. 

We combine things together into a single matrix P, or a projective matrix. So now we want to know: What information does the extrinsic matrix encode?

WE translate the rotation matrix, and Xc = R(Xw - Cw) = RXw - RCw = RXw + t.  Mint * Mext * ~Xw = P ~Xw, which afterwards we can completely predict what a certain camera will do geometrically. The projection matrix will be a 4x4 matrix as

p11 p12 p13 p14

p21 p22 p23 p24

p31 p32 p33 p34

p41 p42 p43 p44

These matrices completely characterize the camera. 

There are many applications in computer vision and robotics where we will find the 3D geometry of a particular object, and we need to figure out projection matrix for camera. Which is called calibrating the camera. 

Suppose we have an object for some known geometry with a checkerboarded pattern and we known the dimensions of each of the squares.

Step 1: Capture an image of a known object with a known geometry.

Step 2: Identify the correspondence points between 3D scene points and captured image. This provides some constraints on the projection matrix.

Step 3. For each corresponding point i in the scene and image, we want to have the projection matrix equation, and we expand the matrix as linear equations and dehomogenize. 

We can rearrange the terms and stack all the equations, very similar into how we did the homography. 

 

And solve the simple equation Ap = 0.

We get the same result if we multiplied a scaled matrix to a homogeneous matrix. We want to set Ap as close to 0 as possible such that ||p||^2 = 1. We can set up the constraint and finally solve for the projection matrix p. All rays will be mapped to the image pixel given by the projection matrix. The inverse of the projection matrix defines the ray that maps the 2D pixels to a 3D point. This is how you calibrate a camera. We are given a number that is a product of 2 numbers, what are the 2 numbers? This is a mathematically impossible problem, so we cannot really do it in general. We cannot uniquely factor a matrix into a product of 2 matrices. 

Given that K is an upper Right Triangle Matrix and R is an Orthonormal matrix, it is possible to uniquely decouple k and R from their product using RQ factorization. 

And we can use camera calibration procedure which we can use to decompose into extrinsic parameters. Now we can extract an intrinsic matrix K and rotation matrixx R. So we can recover T, in which we can get the location of the camera center. 

We use camera calibration to estimate the projection matrix decomposed into the extrinsic matrix and the rotation parameters. We can then determine the location of the camera center, or the Pinhole. 

Given by these equations: 

There are lenses with the same pinhole model, but there may be radial distortion or tangential distortions, and we need to account for this during calibration. Above C (center point) is how to extract out the individual intrinsic properties of a matrix. 

An image pixel just defines a ray, and 3D point can lie anywhere in a ray, which is why it is hard to determine a 3D scene coordinate from a 2D coordinate, since there is one parameter z which is the depth of the scene point. However, we can go over this hurdle by determining the true depth. This process of shooting rays and intersecting them is called triangulation. 

ul - ur is the disparity, indicating how much an image shifted from an initial point. The close the object is to the eyes, the more the shift of the object will be. 

Eyes will be far apart for images needing more accurate depth perception. 

This is an example stereo camera.

For every pixel in the left image, we need to identify the pixel in the right image that maps to the same point. For a point in the left image, the corresponding image will lie in the same row as the right image. Instead of a 2D search, we now only need to do a 1D search and compare candidate pixels to each other.  We can compute the sum of absolute differences, L2 Norm (sum of squared differences) or the normalized correlation value. There is a tradeoff where how large a window will be. Larger windows gets more robust matches but then we lose localization, where the depth will look overly smooth and patchy. 

The state of the art makes computationally expensive stuff. 

However, this won't work well in texture less machines, and don't use a laser scanner.