Computer Vision - Guest Lecture

The lecturer is a faculty member of UCLA and  Computational image scientist that owns his own company.

We're going to talk about Polarized Imaging, have a quick snapshot of the images, and have a discussion when time allows.

When we combine physics with computation, we don't know much about physics sometimes, so it's immensely helpful to have a reminder with polarization. 

Polarization is the rotation of light in space. 

The E field lays in within, and we can have rotated plan of polarizations. Light incident on water means that light is propagating a certain way. We also draw hashes and dots on a light ray, indicating that the ray contains both horizontal and vertical polarization. 

The ray that hits the water has in-plane polarization, while the ray that leaves the water may only have out-of-plane polarization, and we call this s-plane polarization. If we know the combination of dots and hashes on the input, perhaps we can infer something on the scene interface and extract meaningful scene properties. 

If we want to get the geometry of a cup using polarization, take multiple pictures using different polarization rotation angles. We want to see what geometry we would extract from some picture. 

We blow up the 2 most dissimilar photos.

How would you tell if polarization has scene information? One person said that you would physically rotate the filter, and then show a difference image. 

And there is nothing that would give me high quality 3D shape. However, we need to add more sophisticated techniques in contract with just subtraction. We need to perform computational imaging using polarization.

Taking an image and making sure the sharp and smooth regions are sharp and smooth is very difficult.

 

Now we focus on vector notation ->n which we call the surface normal, saying the local slope for the surface which in turn is a proxy geometry.  

We can find an axis, and in the time of light I can directly get the range of the pixel, and for every pixel, determine the depth using the time of light. Another way to get the slope of the object is to get the surface normals. It turns out the tangent line of a curve is orthogonal to the normals, which is the surface normal rotated by 90 degrees. So we know the gradient of the curve, which can help us represent the curve accurately. We're able to take the surface normals and potentially integrate them. What we can do is to measure gradients and then integrate to obtain the shape of the object. We use polarization to estimate the normals.  

We're going to do this with a method called polarization. 

Let's say light hits the surface and we obtain a specular reflection. We want to solve for the zenith angle at every position. Unfortunately every line segment has tons of dots and height, and we want to make sure the angles relates to surface normal.

Fortunately, several electromagnetic equations tells us how these variables are related. The crux is to solve for theta. 

r⊥ = (cosθi - ncosθt) / (cosθi + ncosθt)

r|| = (cosθi - ncosθt) / (cosθt + ncosθi)

We don't get n by measuring the refractive index at each pixel. 

For previous, we hardcode n to 1.5 or we might be able to perform machine learning. 

Assuming we knew the refractive index, how would this data be on a real-world object? 

We collect images with different angles.

We end up with a sinusoid when we plot the polarization angle in comparison to the intensity of a pixel. Geometry has different surface normals on spheres and we plot different surface normals on a circle.

The brightness as a function of the polarizer angle is related to the cosine of the domain which is Φpol - phi, which is called the azimuth angle, which is exactly what we are trying to compute. We want to extract Φ from this.

One way to do this is to extract many polarization angles to estimate the phase. 

I(Φpol) = (Imax - Imin) / 2 + ((Imax - Imin) / 2 ) cos(2(Φpol - Φ))

Suppose we have solve for an angle Φ and Φ' is Φ + π. This creates ambiguity, and I don't know what the correct solution is. There is ambiguity because there is symmetry of the azimuth angle at around 180 degrees. However, there is an azimuthal ambiguity problem with 2^p solutions, since we have 2 solutions at every pixels. But the solutions make this problem extremely difficult. 

The electromagnetic analysis alone gives us an underconstrained problem. We need to know the refractive index, ambiguity in surface normal, but we get low SNR, and the surface conforms to existing Fresnel models. This is not to scale to real world devices. 

The ground truth has been simulated, and we can plot a horizontal slice through a depth map, and a slice indicates that there is a high frequency content. The time of flight is really good for cmm-mm problems, We would get a noisy depth map, and smooth it, but unfortunately high frequencies can be lost. 

From electromagnetic analysis, we get the shape of a curve, from the 180 degree π ambiguity, but it still have support on frequency domain. We combine the map and then blend it with the polarization data.

The time of flight is using a range by sending a ping of light and see how long it takes to reach an object and come back. We need an ultra fast camera to timestep the light, but it's hard to make faster and faster cameras. 

We have the depth map, but we get grooves. The zenith and azimuth angle are estimated from Fresnel equations. How do we approach the problem? How do we blend the depth map into surface normals? We need to blend them into the same domain!

To get the surface normals, we use a depth camera to differentiate a depth map, and obtain a surface normal and azimuthal angle. The depth map doesn't have the polarization shift, so we can blend things together, and do a gradient domain correction and then using a constrained azimuth, resulting in a better version of the polarization data. 

Now that I corrected gradient, I simply need to integrate the algorithm, and get the shape. We can use the depth map as the regularizer, and that if I were to integrate gradients, then the integral of the gradients should be close to the 3D shape. This regularization strategy is still a little bit naïve, and we want to find the integral of G and we set this gradient constraint with a lambda λ and let this λ value to be very high. We call this naïve because if enforce this constraint too hard, we will just spit out z. 

We want to integrate the region such that the low frequency gradient = Z. Let's do a low pass filter.

This is a physics based regularizer and we end up getting an ideal.

The one last thing we must do is to pose the integration problem as a graph problem signal. We can look at gradients as a graph. Very pixel is a coordinate and every edge of a graph is a gradient. We have something called the degree of polarization signal, and put a red flag when the threshold of polarization small, which refutes the use of the gradient as a reliable cue for integration. We can get a more clean integration without noise when we go and remove these noisy gradients.

There are alternative approaches to get the depth map of the coffee cup, using a kinect and industry laser scanner, or learning the shape from shading refinement, or base machine learning and try to learn 3D shape. Every 3D strength has its own strengths and weaknesses. 

We can can have a computer science understanding on how to prove light standings on unique bounding light transport.

One area we apply this to is to look at how different types of reflection (more complicated reflections) and we need to account for cases where real world objects are not mirrors or lamberian. We must study mixed reflection cases. 

The time of light has so few scene assumptions and will get a depth map otherwise. Self-driving cars use this time of light system. It gets good geometry of the scene but doesn't have the detail.

We blend depth and photometry cues in order to make the image more details, creating ridges for geometry as a texture copying variation.

Polarization works really well with specularity, and that's why it's a very useful edition to the sensor fusion stack.

However, there are limitations. The device requires a core step map. On top of that we need to take eah image with a polarized camera, and the technique that is presented is not real time. These 3 limitations prevent any industrial use of this technology.

Polarization can now be real time due to some device called the multipolar camera. 

Depth priors for machine learning gives us a reasonable depth. We have machine learning solutions to go and define a physical glock, and approximate things with neural networks. We can scale not-real time pictures to something that is more practical to the world.

The depth data is used to solve ambiguities in polarization. 

The next step is the additional applications for polarization. The classic application for polarization is glare removal. in order to see through the water, because the glare is typically polarized perpendicular and now we can block polarized glare.

Polarization enables us to remove scattered light and specularities as well. We can remove specularities with polarization first, then perform polarization stereo. 

3D glasses are polarized. LCD (liquid crystal displays) leverage polarization. We can control brightness using polarization cues,. The rotation of filters pass through lights, and the amount of light that passes through is related to the the cosine through the filter. We can do this over many pixels uniquely.

We assume the light that is coming in is unpolarized and has an equal distribution of angles. We want angles at all different distributions coming in, and we call these the unpolarized world assumption. Most natural lights are unpolarized. If we break this assumption, we can regularize out errors as a result.

Kinects are polarized since they use lasers. I would have to apply for calibration to remove nonpolarized light from the surface.

If I have a zoom lens of a camera, I am able to get the shape at any range. In terms of the measurement itself, this still holds if everything else is constant.

We need to account for how lighting changes the behavior at these ranges. We would obtain range technology unbounded by the range if I succeed. 

One assumption is that texture itself is lower frequency than the color frequency of the depth map and the material exchange is generally low frequency with embossing.