Computer Vision: Learning Illumination for Unconstrained Mobile Mixed Reality Summary
Introduction
The following article that is being read discusses researchers inventing a new method.
An example of HDR, or high dynamic range imaging.
- The input to this experiment is a low dynamic range image from a mobile camera with a limited field of view.
- The training data of this experiment are videos of spheres placed within the camera's field of view.
- The LDR ground truth sphere images are matched with image-based relighting. This inference runs at a normal device. A neural network is used.
- Realism is enhanced by training on white-balanced videos.
This is an augmented reality article, so what we are trying to do is we are trying to composite rendered objects into photographs and videos. Realism is dependent on lighting, etc.
An issue is that only around 6% of a panorama scene is actually observed by a camera.
It's hard to capture lighting without omnidirectional viewpoints.
Convolutional Neural networks have state-of-the-art techniques for a limited Field Of View for both Indoor and Outdoor Scenes.
The following comprises of the Indoor Scene of the Convolutional Neural Network.
1. Trained a Neural Network with LDR Panoramas.
2. Fine-Tuned the network with HDR Panoramas.
For outdoor scenes the researchers also
1. Fit a sky model with LDR Panoramas.
3 spheres are used to determine the ground truth illumination of a specific surface.
There are 3 surfaces
1. Mirror surface: This surface basically reflects high-frequency lighting, but fails to represent the intensity and color correctly.
2. Diffuse gray: This surface reflects low blurred frequency lighting but captures a complete record of the lighting.
To train the Convolutional neural networks, the researchers tried to adjust the parameters to minimize the difference between the the ground truth images, and the actual appearance rendered from lighting.
The steps are as follows:
1. Measure the reflectance field of each sphere.
2. Add an adversarial Loss term.
3. Collect 200,000 new terms
So these "spheres" used to determine ground truth are utilized finally to determine backgrounds.
We are trying to predict illumination from a single unconstrained image and this can be quite difficult.
The goal is to predict the HDR lighting with a limited field of view. Spheres with diverse amount of reflectance functions reveal different lighting cues, enable the recording of training data using a LDR video stream.
The inverse rendering problem is the joint recovery of geometry, reflectance, and lighting.
The researches wanted to estimate the lighting of unconstrained images with unknown Geometry and complex scenes.
Scene's geometries are estimated with Convolutional Neural Networks by finding specific errors, the real panorama, vs the predicted ones. They estimated the scene's:
1. Geometry.
2. Diffuse Albedo
3. In view light sources.
The researchers subsequently used a database to find a matching panorama. The Convolutional neural network helped the researchers to learn a mapping from the limited Field of View LDR to HDR.
New images can be formed as a linear combination of reflectance-based images. Image-based relighting was what was used.
Now that the introduction is finished, the method that the researchers conducted this experiment with is below.
Method
HDR Lighting is inferred from only LDR images with reference objects, provided they span a range of reflective functions that reveal different lighting cues. There are 3 spheres that are holiday ornaments with 82.7%, 64.4%, and 34.5% reflection rate.
Portrait HD video is collected, which increased the speed of the training data, since millions of images are subsequently captured.
A training network was fine-tuned.
The spheres are located by detecting circular boundaries through marker-based tracking, then the copped images of spheres are resampled. For the background images, the lower 20% of each frame is removed. The final training data consists of a set of cropped background images, one for each diverse reflective function. ,
Now, I want to talk about the network architecture.
The input is a resize LDR image, normalized to the range [-0.5, 0.5]. The architecture used for this problem is the encoder-decoder type.
There are 17 Mobile Net V2 layers, and the output feature maps with a Fully Connected Layer. This is an encoder-decoder layer. The Mobile Net V2 layer represents the encoder. A vector of size 256 is generated.
For the decoder, the vector is reshape and upscaled by a factor of 2, generating a 32x32 color image.
The researchers used bilinear upscaling with convolutions, and the reflectance functions are subsequently revealed.
So basically we're converting a 134 x 192 to a 32 x 32.
The image is denoted by a particular 4D reflectance field (θ, Φ, x, y), where x and y are the pixels and θ, Φ and the dot product of the reflectance field. This method was used to render spheres. The light source color is converted to a mapping and the subsequent ball is mapped. There is a computer controllable sphere are white LEDs, and reflectance fields are filmed this way, generating one-hot matrices.
The next topic that I want to talk about is the loss function. Image relighting loss is minimized and adversarial loss is added for optimization.
The loss function minimizes the reconstruction loss between the ground truth spheres and the prediction spheres. We can compute a linear combination. Some equations ensue now. And I will go over the equations one-by-one.
The first equation is where L represents the intensity of light and R is the individual pixels, and we are trying to compute a linear image I from the intensity, and reflectance fields.
Here it is:
Ground truth is Gt and it represents 8-bit encoded images, and Q is the pixel values.
The rendered sphere images has a different soft-clipping function.
The linear encodings are then gamma-encoded in order to match. The pixels in the equation are
. A binary mask is applied to each ball to get the following reflectance function. λ represents the optional weight for each reflectance function. 
The next thing that I want to discuss about is adversarial loss. The reason is because rendering shiny objects with realistic specular reflections requires high-frequency lighting. To render plausible clipped ball images and higher frequency illumination, since humans have difficulty reasoning light directions. This should be simpler since real-world lighting is based on statistics.
.
There is also a discriminatory network with the base Convolutional Neural Network as the Generator. The generator network attempts to tempt the discriminator network to try to appear as "real as possible.
The four corners of ground truth are sampled, and a background is interpolated. The ground truth is then softclipped. The following equations computes the adversarial loss network, and then interpolate a 32 x 32 background.
The respective equation describes the adversarial loss term.
The next thing I want to talk about implementation details.
TensorFlow was used for 16 epochs with many optimizers and learning rates 0.00015 for G. Lambda was set to 0/2. 0.6, and 0.2 for each of the filters, and the gamma was set to 2.2 and beta was set to 0.9 and 0.999. The batch size is 32 and batch normalization.
RELU6 was used as the rectifier.
There we many unseen and seen test videos, indoor and outdoor. The evaluated results are below.
The lighting is measured as compared to the ground truth. Show the average pixel loss for each unseen test dataset. The per-pixel angular error is also measured.
There is both reflective function loss and adversarial loss, and report the reflective function loss.
The mirror function leads to higher reflective error for others. Training with only the mirror ball fails to reveal the full dynamic range for lighting. Training with the other balls refuses to produce a realistic mirror ball, and training with the adversarial loss reveals a higher frequency illumination than expected.
The following shows the rendered inputs.
Diffuse(d) matte silver(s) mirror(m)
The next experiment, the researchers 3D printed identical bunnies using a certain model. The 2 bunnies are colored diffuse gray and matte silver, respectively. The bunnies are 49.9% and 34.5% reflective, respectively, and they are photographed in real spaces. The real bunny images are then compared vs other composites, and recorded the ground truth using a CANON camera. A linearization curve is subsequently fit. The network is retrained which is described below.
The spheres are used again to compare the predicted rendering with the ground truth, and the loss is measured.
Multiple reflective fields testing generally improves results significantly.
The loss for the method improved significantly, and runs on 12 to 20 frames per second on mobile devices.
The next part of the paper discusses the limits to this experiment
This is where the important stuff comes up.
A suggestion for improvement would be to replace the bottom directions from the lighting sample with the pixel values from the surface. More than one camera should also be utilized. Try to simulate different camera models. This is a High dynamic Range Imaging technique for mobile mixed reality with low dynamic range imagery using reference spheres , materials, and Convolutional Neural networks. This has helped in mobile mixed reality.
Thank you for reading.
Comments
Post a Comment