Computer Vision: Computational Cameras

We're going to talk about this Thanksgiving week, keeping things light and overviewing the research in the area called computational camera.

This is a series of articles about computer vision. At the very beginning, the first known cameras is a large dark room with a small pinhole in the wall. Light would come in, pass through the pinhole, and come into the wall. This is literally called a camera obscura. We have come a long way since then.

There has been a remarkable evolution since the first camera. The underlying physical principle of the camera has remained the same, like the pinhole model, with a planar detector. Light comes in, passes through a pinhole, and behaves like a pinhole in terms of the physics, and light passes through the pinhole and projects an image to the detector.

You can now increase flexibility of camera models that you cannot achieve through conventional camera. To get an unconventional mapping, you need to get different types of optics.

There's computational cameras and lightview capture, image based lighting, holography (in scientific), defocus blur, motion blur, spectral channels (capture more 3 color channels), etc. For example, it  can be in remote sensing and medical imaging. There's then 3D capture, and various other techniques. This field is still relatively new, but a lot of different methods have been developed.

We can do object side coding, pupil plane coding, focal plane coding, and illumination coding. 

There was some type of relighting in James Cameron's avatar. Also virtual reality. This movie was way ahead of its time. 

And lighting cameras to see the characters from multiple viewpoints and multiple lighting levels, used to render these characters.

Object side coding is placing something between a camera and element. This can be catadioptric imagine, when a mirror is placed, augmenting the relative camera.

Let's say there's a camera. Mirrors create 2 virtual viewpoints in addition to the original viewpoint. From the virtual viewpoint, there will be 3 cameras imaging from 3 different viewpoints.

Then there's a locus of viewpoints from a circle around the original cameras. 

We get weird images not regularly interpreted by humans. Here is an example: 

We can use this to do 3D reconstructions. 

We can also use BRDF to picture the reflectance of a pin, for example, this is used in virtual reality apps. 

You need to capture all of the images, Only then you can get a proper 4D system.

Any light coming from the sample pin would reflect 4 times in the mirror before moving. The raw capture images have 4 concentric rings, as if there is a camera taking an image in a ring.

We can render this in a non-Lambertian form.

This is one class of coding methods. another thing is optical filtering, where I place some type of filter in front of the camera. Polarization is used to advance the quality of the photographs, etc. 

One application of polarization is to increase the contrast of images degraded by a scattering medium in the scene. Dehazing reduces the smoke or fog and increases the contrast of the images. The idea is that the reflected light have different polarization properties, and we might be able to get rid of the scatter. The scene point is heavily defined by the depth of the scattering. 

We can also estimate the depth of a scene image as well. 

The below diagram shows an image and its "range map".

There is a flutter shutter camera where the optical element is dynamic and its properties change over time. We are coding in time, where the shudder is opened or close in a fast time.

Motion blurring can mathematically be interpreted as convolution. The filter can be considered as a rectangle. The x axis indicates the time and the y camera indicates if the filter is on or off at time x. 

We can multiply the fourier transform of an image, with the fourier transform of a box feature. There are areas where the fourier transform equals 0, in which these components are zeroed out, or loss. This is why recovering the sharp image is challenging. 

We are switching the shutter on and off multiple times. There are certain components that are  going to 0. We want a Fourier transform not go to zero color. Now we can get a broadband function, that gets high spatial frequencies: 

These are just a few examples. I can get a hyperspectral camera and light film imaging, which acts as an array of individual cameras.

Pixels can be demosaiced to get red, green, and blue color. 

Unfortunately, some cameras have a limited dynamic range, or uses small low exposure. One way to deal with this is called exposure bracketing, which captures multiple images of different exposures, which we then put together into a single High Dynamic Range picture.  There is a problem of ghosting, and many problems due to motion

If we can take multiple exposure images at the same time, then there would be no problem with filters at all. 

White filter lets in a lot of light and the black filter lets in a bit of light. We place a spatially Varying exposure(SVE, 4 images) or a simple array.

We  can design a camera where the exposure time goes into many shorter slots as follows: 

Using this, we can get a video from a single image and subsequently  motion blurs to reconstruct videos.

There is a spatio-angular resolution tradeoff. If you look at a regular camera, each pixel collects lights coming from all the light rays, and the pixels sum the intensities of all of these light rays. To get more information, move the sansor back and placicng a microlens array between the main lens and the detector. It turns the single camera intoa  2D array of cameras.

With digital refocusing, you can focus on whatever you want.

We have  a microlens array, and each lens covers a 10x20 pixeel, almost looks like a defuser. 

With a lightview camera, we have  almost an entire collection of pinhole cameras. 

The 3rd class is pupil plane coding, where we place an element in the aperature of the lens itself, and we can represent this y coded aperture imaging, where we place a musk in the aperture of the camera. the coded aperture will block some of the light that is passing through. 

We can also remove defocus blurs, and as a result, recover teh depth map of a scene, all focuse recovert. As a byproduct, we also get the depth map of a scene.

Illumintion coding augments the regular camer with some kind of prograammable "light flash". this light source can be a projectos, which is mostly thought as a disply source. It's a coordinate source that projects a patter in the scene. The following diagram demonstrates this. 

This is called structured light, which is the basis for many modern 3D cameras. The first KINECT generation is based on this idea. 

The camera would compare the projected pattern and image to get the 3D scene Geometry.

Another instance can be either an LED or laser arrow, as a scene light source which is coded in time. 

The time of flight-based depth sensing is where the time array determines how far a light source is from a scene. This started making its way in consumer devices. 

Light scattering due to disturbances always screws the problem up. 

All these makes the vision systems a lot more challenging. We need to perform an adaptive approach to recover many fine details. 

This capability can be useful in outdoor 3d Imaging, especially under a strong power budget, including autonomous navigation. 

Even indoors, light will bounce multiple times, and this will always create a problem. Multiple deflections will distort things. 

Another example is a scene with a shower curtain. What happens is that light goes to a curtain, and only part of it comes back and most of it goes through. The goal, is to recover the shape of the curtain. This happens a lot in medical imagine. Micro phase shifting can get geometric details like the folds in a curtain.

From spectral images, we can get measurements of certain images. We can track seconds hand in watches and the minutes and the hour hands. One application is to look at celestial objects and space and satelllites.

This is also good at gesture recognition, a device that tracks very subtle gestures, using a 3d Motion histogram, whether its a swipe, stretch, using a 3D motion histogram.

We can use multi-path analysis to recover certain objects.

We could extract signals as to record the motion of a certain object, and we can do multiple objects. Eventually we can design cameras that look through opaque materials, or enhance monitoring or see what something is made of. It can be used in robot grasp planning, or robot yield inspection.

Finally, we can design vision systems where we can detect one photon in a time. This can be defined in systems and astronomy.

According to Werner Heisenberg, "what we observe is not nature itself, but nature exposed to our methods of questioning".