Computer Vision: Depth from Focus/Defocus

 Here, we try to step behind the image into the 3D scene and start looking at metric that recover the properties of the 3D world behind the 2D images. 

3D vision is a very important part of vision. You would want to know the 3D layout and surroundings. We look at methods that can cover the 3D motion of objects as well. 

Supposed a camera is focused at a plane. This is the plane of focus. Anything inside of the plane of focus is perfectly in focus, and we want to measure the amount of blur at every image location of every pixel.

There are methods to compute depth by analyzing the amount of focus or defocus in an image. Let's first review the image formation model.

Suppose there is a camera with a lens with a certain diameter, t and the camera is imaging the object at a certain distance o from the length. The lens focuses on a certain distance I from the other side of the image. F is the focal length and o is the object length. Then 1/f = 1/i + 1/o.

 The larger the aperature, the larger the length diameter, the larger the blur circle. The further the sensor is to where it should be, the larger the blur circle diameter as well.

Based on these idea, there are 2 depth recovery algorithms. There is a depth from focus algorithm. The farther the scene point is from point of focus, the larger the point, and these are used to determine 2 different algorithms. The end goal is to return a depth map, a 2D image of the value of each pixel, returning how far each image is, giving the 3D geometry and the 3D layout of the scene at hand.

The first algorithm is called Depth from focus.

Suppose we have a camera and an object with a certain distance o from the length. Suppose you were able to determine the sensor location so that the object is perfectly in focus. Then we can simply use the Gaussian length formula 1/o + 1/s = 1/f.

How do we place the sensor such that the object is in focus?

Sweep the sensor in the entire range, and take an image in every location. Take an image in s1, s2, s3, and place the sensor in many, many, MANY locations. Suppose we have the series of images and these images are taken just by moving the sensor locations. We want to assume that one of these locations is the perfect location. How would we use a series of images to determine the "perfect" edge location? 

Moving the sensor is more like changing the focus of the lens. 

The only difference in images is the amount of blur. All these images will be blurred to varying degrees except that one image that was taken with the sensor at a particular location. From looking at the sequence of images, we can identify the image where the object is perfectly in focus. Then we can simply take the corresponding sensor location and use it in the Gaussian length formula in order to estimate the object distance. This is the Gaussian length filtering. 

For each small patch in the image, determine what is focused. We have a sequence of images in order to focus on. 

We try to figure out the best images individually at each pixel. We then use the sensor location and apply to lens law. We repeat the process on each individual pixel.

This is how Depth from Focus (DFF) works. 

Again, 1/f = 1/s + 1/o

The main question is: How can we find the best-focused image? 

We can look at Fourier Transformations to turn the concepts of sharpness into an algorithm.

o = sf / (s - f).

Since defocus attenuates high frequencies, use a high-pass filter to measure the amount of sharpness in the filter. 

At each pixel location, define a small window. For each pixel in the window, compute the value of the modified Laplacian and add all of them together, which is called a focus measure. The higher the value, the sharper the image patch around this pixel. 

The x-axis is the sensor location, while the y-axis is the value of the focus measure. 

Depending on the image, the focuses can be different, and the most in focus image is the maxima location of the focus measure. The max is the sensor location, which can be estimated using the Gaussian length parameter. 

Depths can only have N values where N is the number of sensor locations, which will cause a discretized structure. 

Interpolate a function here!

The peak of the focus measure curve is a Gaussian-Like function. We use the Gaussian length law to recover the scene distances. 

If we can linearlize the problem, we would only need three samples to estimate s(line) since there are only 3 unknowns. This is called the Gaussian interpolation. 

A depth from focus (DFF) system is often used in microscopic systems. Move the opjects vs the sensor. As it moves, a series of images are captured, and we can use DFF to recover the architecture of an image. We can uncover silicon wafers and leaves stomata. DFF is a method to recover the 3D orientation of objects by taking a series of images. This may be okay for things are not moving.

Can we recover 3D geometry with a smaller number of images?

Yes! there is a method called depth from defocus.

We are trying to identify the image, where things are in focus, and try to directly give a series of depth values. 

An image could be a blurred circle of diameter B. b/D = (I - s / I). 

So o = sf/ (s - f - b(f/D))

We are taking the sensor distance, the aperture, and the blur diameter, if we can somehow compute the blur circle diameter, we can take an expression and compute an object distance, which is what makes a single image enough to get the 3D map representation. 

Defocus acts as a low pass filter for an image. The blur circle should be a circle with sharp edges.

We are trying to find the size of the Gaussian filter, and a Fourier transform will be able to tell how much frequency the content at each patch has. Taking 2 different images with 2 different focus setting can help to resolve lots of ambiguity. 

The blur images is the convolution between the sharp image and the blur kernel. This is impossible to solve without explanation. We want to take 2 images with different defocus settings, and the sharp image, which is common across the image, and 2 different blur kernels. 

2 blur kernels are related. It turns out the ratio of the BLUR circle diameters is the same as the ratio of the aperture sizes. 

We then get the following equations to get  σ1 and  σ2.

Instead, we can impose this as an optimization problem to write the error as a 2 variable function, and can be minimized using simple linear least squares. 

DFD requires scene depths for each image, and determine the 2 unknowns σ1 and f(x,y) and use this to then compute the size of the blur circle, which can then be used to compute the object image. 

So how do we capture 2 images.

1 way is to change aperatures, another way is to move the sensor towards/away from the length with respect to the length. This method still requires 2 images, and if the object moves between 2 images, there are going to be errors. 

One way to cover both images simultaneously is to use 2 different cameras.