Computer Vision Neural Network Project: 11/14/2020 Summary

 I am now trying to install the anaconda environment now. I think that PyTorch is installed now, so I am going to read the rest of the proposal. 

The proposal is the same: to estimate lighting information on Augmented Reality devices. 

There are 2 parts:

The time needed to run through the network

and

The time needed to render an object. 

For the first part, a pretrained network is used. The second part is more important, where a render engine is used, to make real-time predictions possible. This allows less info to be extracted and reduces the complexity of a neural network.

We want to discard unconfident output, interpolate between frames, and compensate for the movement of users. 

Here is the dataset for the illumination prediction. 

Each image's resolution is 3844 x 7768 and the size of the entire dataset is 154GB. There is no label information for this dataset, so it is an unsupervised dataset. The 2 important considerations is to reduce the input size to make training viable or to label images with illumination source parameters. 

The steps are as follows.

The first step, is called Pre-Processing. 

There are 2 approaches in order to deal with the dataset size.

1. Reduce the resolution of the image to 400 x 900. This will convert the HDR images to LDR images. 

2. The next step is to create labels.  We fix the light source number to be equal to 3, and model each light source as an ellipse. 

3. For each image, we determine a threshold, which is the extreme value (the max pixel value) / 3, and it is exr/100 for .exr (raster image) files. For each image, a binary output image is resulting, with 1 indicating there is a light source and 0 indicating there isn't. 

4. Now there is going to be the algorithm that find the connected components of the light pixels. I believe that we can use Depth First Search to do that, and for different components as light sources, there is a mask for each on of them.

5. After which a rectangle is drawn surrounding each light source. These parameters are also defined with regards to an ellipse, and are used to label each light source. 

The algorithm in action. 

The next step is using a convolutional neural network, which is described in this article. It uses DenseNet 121 as a feature extractor and the output layer is the linear network. The loss function will be just the sum (Σ) of all the square loss functions. 

In preprocessing, extracting locations, colors and brightness is reliable. We also want to classify the many types of light sources during labelling. The area light separates them to directional lights and point lights.

The dataset will be transferred using a transformational spherical coordinate matrix, and the environment map is cropped using an unwarped image. 

So obviously the input image size has to be modified for better performance. Other networks that are subject to use are ResNet and MobileNet.

Maybe use the Cosine loss function instead of the original loss function. We want to interpolate results between frames, and rebuild labelled lights using the Blender software.

Now, I'm gonna start to train my convolutional neural network. 

TORCHVISION is a package that has data loaders for many common datasets, which helps in the writing of boilerplate code. 

Now, I will start training an image classifier, using the Spyder IDE in Python. 

However, there's a problem: I'm getting a "no module named 'torch'" error. A simple reinstallation solved this problem. 

Compose(

    ToTensor()

    Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))

)

This is the result when I try to compose a tensor. Now I want to install a dataset, and download the dataset, and subsequently put them into many classes.

Part 2 will be tomorrow morning.