Computer Vision: Radiometry and Photometric Stereo

 

Photometric stereo  is a method for recovering the 3D shape or Geometry of objects. We are trying to reason about properties of the 3D world of the images. This is critical for robotics, user interfaces, augmented reality, and so on. There are methods for recovering 3D shape, and after that we will talk about another important property of real world objects: their motion. 

To do this, we need to understand how different elements of the 3D world work together to create a 3D image. How do objects reflect light to create an image? We need to think about how light gets reflected by objects, and finally, how this reflected light goes through the camera to make an image. Then we may be able to invert an image, if we fully understand how the light works.

Suppose we have a small surface patch illuminated by a light source. We define a surface normal ->n, a unit vector that is perpendicular to the patch at a point, defining a local orientation with this patch.

-> n is a 3D vector but a length of one, so the vector only has 2 degrees of freedom.

Image Intensity = f(Illumination, Surface Orientation, Surface Reflectance)

 

The higher the surface reflectance, the brighter the image will be. The image brightness also depends on the surface orientation, which encodes the 3D geometry. The image brightness is a function of all these factors.

We first look at the relationship between image brightness (Image Irradiance) and scene brightness (scene radiance). The natural question is, what should be the natural relationship between scene radiance and image irradiance? They should be directly proportional  and L should be larger than E, they should really be the same. E ≈ L (π/4) (d/f)^2. d/f is the direct f number which is the camera aperture size d. The image intensity is linearly proportional to the amount of light reflected by an object.

Say we have 2 surfaces, 1 is a white screen and the other is a mirror and let's say both are illuminated by similar light sources and imaged by similar cameras. Would the 2 surfaces appear the same.

A mirror reflects light into one directions, getting reflected on the mirror opposite direction. However, the white screen scatters the light making light reflected in multiple directions. The total amount of light is the same in all cases.

Surface reflecting depends both on viewing and illumination directions. The dependence is different for different materials. Different materials reflect light in different ways. 

BRDF stands for Bidirectional reflectance distribution function. We define direction vectors using 2 angles with θ and Φ. Theta is the zenith angle and phi is the azimuthal angle. E (θi, Φi) is the irradiance due to source in direction (θi, Φi).

BRDF completely describes the reflectance issue on a surface, and completely describes how a surface reflects light, and we can predict predict how a material will appear under any type of lighting geometry.

BRDF = f(θi, Φi, θr Φr) = L(θr, Φr)/E(θi, Φi)

The origin is the point of incident at the light ray, and the z-axis is the surface normal. Its definition is independent of what the light source really is. We use the BRDF to predict the outgoing reflections and sum all of them together to get the total reflecting light. All these angles are defined with respect to the surface normal, it is always less than 90 degrees. If I know this I can completely predict what the surface will do to any kind of incident light.

Its reciprocal which means switching the incoming and outgoing reflections still get the same value, it's the same as long as we reverse the direction of light. Isotropic is the reflection is about the surface normal. So it depends on the difference of the azimuth angles.

f(θi,θr, θi - θr). If completely says how the surface reflects light at any appearance. This is a good  time to make fin of my looks. We need to understand how the images are formed. The albedo can be 1 if all success. The brighter and surfaces can compare. As a result L = pdE/π. The brighter the light source the higher the irradiance, and vice versa. E = J cos(θi)/r^2 = J/r^2 (n x s)

S and n as a result are unit vectors and direction vectors. The radiance is directly proportional to the amount of incoming light. We get a important expression that get the Radiance L of a Lambertian surfaces as a function of the properties of the surface itself. 

The surface appears equally bright, but laminating from an angle produces less light reflected back on the surface.

The  brightness of a surface is independent of the viewing direction but heavily dependent on the angle of the lighting direction s makes with the surface normal n. 

Lambertian models don't emulate shiny surfaces. We use Lambertian model to develop 3D geometries. Photometric Stereo is the method for getting the 3D shape of objects from image intensity values. The image intensity is a function of the light source directions, and the surface properties, which is the 3D shape that I am interested in recovering.

Image intensity is F(source direction s, surface normal n, surface reflectance)

L is the image brightness. The only unknowns are the object properties and the surface normal. These are the things that we are trying to measure.

We have to assume we control the light source, and we need to compute the surface normal. We don't know surface properties, orientation etc. The one thing that is under our control is the lighting, and we can control the direction of the lighting. We can change the light source direction and we can generate many equations by simply changing the light source direction and the unknowns remain the same as we change the light surface direction. We hopefully can recover the three unknowns with this, and this is the idea behind photometric stereo.

Now let's consider a single pixel (x,y) and let's say the three intensities of the pixel are I1, I2, and I3. These are arbitrary values that are the same for all the images, and now we can just assume them to be one. There's 3 equations, 1 for each intensity. N are the unit vectors in 3 directions, and we want to write these equations more compactly.

I1 = (ρ/π) n x s1

I2 = (ρ/π) n x s2

I3 = (ρ/π) n x s3

We can then write this in matrix format.

where I = SN where N = (ρ/π) n

How do we determine Normal. N = (S)^-1 I where n = N/||N|| and albedo is (ρ/π) = ||N||.

Surface Normal gives us the 3D shape and albedo tells us how bright and object is. 

It happens when all the light source directions lie on a single plane when, and this does not work when S3x3 is not invertible.

We're going to be okay if we have 3 directions that don't lie in a plane.

Often people take more than 3 images to be robust, and we get better results by using more (K > 3) light sources, obtaining vectors of specific intensities.  

S matrix is not a square matrix, and because of that we cannot invert this matrix directly, so we take the least squares approach.

STI = STSN

The product would be a 3x3 matrix, which is the size of the matrix.

N = (STS)^-1 STI

 Input images are 5 of them taken in 5 different directions. A give pixel looks at the same scene point, and these are the recover surface normal vectors, and we get the albedos at each pixel. Each needle is the projection of a surface, which is a 3D vector onto the 2D plane, telling us the surface orientation on each point of the object. 

Gelsight is a novel system for measuring surface texture and shape. We place the sensor on a glass, we are able to measure the sensor by having 3 different light sources, which makes the camera capture a color images. Albedo is the proportion of light that is going onto the surface. Lambertian reflectance is the property that defines an ideal "matte" or diffusely reflecting surface. The apparent brightness of a Lambertian surface to an observer is the same regardless of the observer's angle of view.