AutoEncoders

 

AUTOENCODERS

There are many types of autoencoders. Standard AutoEncoders help to minimize the L2 reconstruction error used for images. The sparse autoencoders minimize L(x, g f x + Ω(z), where Ω is a sparsity penalty. It can be used for application where a small set of variables are expected to explain a process.  The denoising minimizes the L(x, g f ~x ) where ~x is a corrupted version of X. These denoising autoencoders learns mechanisms from removing noise from observations.

Contract Autoencoders Minimize L(x, g f x + Ω(z, x), where Ω(z, x) =

This forces the encoder to learn features that do not change much when slightly perturbing data. The stochastic encoders and decoders encode probabilities such as p(z|X) and p(X|z) and minimize likelihoods.

In deep autoencoders, layers use a Restricted Boltzmann machine, then fine-tuned using backpropagation.

In deep autoencoders, dataset of curves are randomly generated by selecting 3 points from the plane.

A CNN-based autoencoder would have a same dimensions as a transpose of a convolution.

VARIATIONAL AUTOENCODERS

Variational Autoencoders (VAEs) are a type of neural network that can learn to generate new data that is like the data it was trained on. Variational Autoencoders are a type of generative model that learns to represent the input data in a compressed form called the "latent space." It consists of an encoder network that maps the input data to the latent space, and a decoder network that maps the latent variables back to the input data. The key innovation of Variational Autoencoders is that they learn to optimize a lower bound on the log-likelihood of the data, which encourages the latent space to have desirable properties such as smoothness and continuity.

Variational Autoencoder is a Type of Generative model. This is basically a model that can be sampled from a learned distribution. One can desire a model that can generate new handwritten digits, one can also build a generative model in a more sophisticated context, for example sampling synthetic human faces by learning a generative model based on the data from humans. It’s useful for robotics, and scientific imaging among others.

Some models are generated by latent code.

                                                         

In some generative models, some samples are generated by a net applied to a random latent code. We get some image X, which we should think of as a sample from the distribution of the images, and these were high-dimensional. Z is latent code, viewed as a random variable. There is a distribution we want to learn and to deal with the distribution we need to deal with the probabilities. We can determine a random Z to determine the random sample from the distribution.

Before we talk about variational Autoencoders let’s talk about plain autoencoders, which reconstruct input signals/images b6y learning mappings from a code.

We would choose an architecture and then try to choose parameters to minimize the reconstruction of an image Xi from Xi itself, trying to reconstruct input signals/images by learning mappings from the code.

 

A plain autoencoder is not a generative model because it doesn’t define a distribution. We just made the neural network to make images that fit in that distribution.

We want to maximize the likelihood. The nonzero likelihood is mainly on a low-dimensional subset of space. It will induce a joint distribution on images X and Latent Code Z, and we will write and expression for image likelihood as a result.

Given data {Xi} I = 1 … n, train a generative model to maximize the likelihood of the observed data.

If generative model, Gθ : Rk -> Rd then p(x) = 0 almost everywhere so as a result we can’t directly optimize the likelihood.

To have a nonzero likelihood everywhere, we will define a noisy observation model

Pθ(X|Z) = N(X; Gθ(X), µI). Under a simple prior p(z), this induces a joint distribution Pθ(X,Z).

The idea for a variational autoencoder is to optimize a lower bound. We can write the likelihood of X P(x) be the integral of all z of the likelihood of Z multiplied by the likelihood of X given Z.

P(x) = ∫p(z) p(x|z) dz

The data we are generating has a simple prior distribution.

Given X we find Z through a process called inference.

We use Q which acts as an intractable proxy of P(z|x)

 

We keep track using a novel method.

 

We will find z through an inference using z, and this will be using an encoder.

We will calculate the distribution of Z given x using a novel network with separately learned parameters.

 

We want to find the lower bound of the likelihood of X to the parameters θ.

Z space is encoded, X space is decoded.

The KL divergence is going to be the expectation of q(z) dividence of p(z)

The KL of divergence of (p||q) != KL divergence of (q ||p) and this measures how far P is from q.

DKL (q || p) >= 0 and is 0 if p = q.

We want to try to optimize the variational lower bound.

 

Here is another way of interpreting this term:

The regularization term is the negative KL Divergence. The term is regularizing Qφ and promoting it to be more like the standard norm.

We want to promote it and smooth it out at the same time.

 

So, we want to maximize the Variational Lower bound, and it maximizes the KL divergence and the true likelihood.

 

Instead of optimizing the sum of the log likelihoods, we want to maximize the sum of all of the variational lower bounds of this slides.

We want to maximize the variational lower bound over φ and θ.

For each Xu, we want to find the best q by taking multiple gradient steps in the direction of φ. Once we get a good estimation for Q φ, then we can get a reasonable gradient descent approximation in space.

Instead, we want to use an amortization trick by learning an inference neural network and assume a Gaussian model.

 

We amortize the inference costs by learning an inference net.

Here, we assume an inference model with the distribution denoted above. This learning model is represented by a neural network, which has a set of parameters which maps X.

We can amortize the inference costs by learning a certain inference net:

 

The parameters of the inference models are shared between the data points.

.

We can now show the variational autoencoder architecture.

The MS architecture in these we want to separate the random source from differentiable quantities. Mu and Log Sigma will depend on Phi, and X’ will depend on parameter theta. We want to update the parameters Theta and Phi.

We are trying to maximize the sum of the lower bounds of the log likelihoods.

This is based on an expectation, so one would need to survey all the Z’s to get an estimate of the value, and differentiate the quantity.

We say “while the gradient of the lower bound is intractable, there are unbiased estimates of it”.

 

The gradient is a quantity computable using standard deep learning libraries.

Instead of running gradient descent down the expectation, we are going to use stochastic gradient descent on a randomly chosen Z.

Since we are trying to take the gradient of the expectation, we are going to get an unbiased estimate.

The gradient of the quantity both expects on the change of the quantity and change of the distribution.

 

Recall we have a particular model for inference with learned mean and covariance.

As a result, we can optimize a variational autoencoder with stochastic gradients.

 

The variational autoencoders optimize the lower bound for data likelihood. We need to determine a reconstruction error and regularization. 

 

They help to optimize a lower bound to likelihood. The lower bound has terms for reconstruction and regularization. We can maintain an inference model for Z | X in place of intractable true distribution. The reparameterization trick allows the backpropagation on mean and the variance of the inference model. These autoencoders finally have been trained with photorealistic outputs.

They tend to produce blurry or low-quality samples compared to other generative models such as GANs. However, they have the advantage of being more interpretable and easier to train.

They are a powerful and versatile tool in the machine learning toolbox, and are widely used in both academia and industry.

Let’s start with the Monte Carlo Integration. Give a continuous random variable X and a continuous function f(X), then

This follows from the observation where we define {Xi} as a set of random varialbes with the same distribution as X, so we will get

By law of large numbers, the covariance of F(N) will also decrease as N increasing, hence making F(N) an unbiased consistent estimator of F.

Monte Carlo Integration is useful in models where you may need to do some marginalization over a random variable. This is needed when the analytical formula for integral is not available and numerical integration over the sample space is prohibited because high dimensionality between the variables.

The KL divergence measures how much loss we lose if we encode a random variable using q(x) vs p(x) such it provides an indication of the difference between these distributions. The KL divergence is

D(P || Q) != D(Q || P)

 

Here is a diagram of the KL Divergence:

.

The Autoencoders encodes an input on a set of hidden/latent variables which are then used for reconstructed the input.

 

The encoder can be modelled as

 

And the top terms correspond to a generative model and the prior.

The variational autoencoder, or ELBO, is

 

The encoder Term is

,

And the decoder term is

.

This can be computer through the Monte Carlo Estimation.

DRAW (Deep Recurrent Attentive Writer)

The DRAW network aims to generate images in 3 different ways. It uses a recurrent encoder and decoder networks. The decoder sequentially add to the output instead of a single shot. An attention mechanism is used to determine where to read, what to write, and where to write. It is a recurrent neural network for image generation. Here is the architecture:

And here are the specific equations:

We can have iterative improvements over the entire image without attention. With attention, a localized operation is parameterized by the center of the ROI (gX, GY) and a stride δ, and a scale parameter for gaussian filter σ and a scalar intensity γ. It helps to generate various digits.