Chapter 1: Introduction to Variational Autoencoders

This article provides a thorough exploration of Variational Autoencoders (VAEs), a vital component of Deep Generative Models, akin to the well-known Generative Adversarial Networks (GANs). Unlike GANs, which utilize a Generator-Discriminator framework, VAEs rely on an Autoencoder architecture. This distinction makes the concepts behind VAEs relatively easy to grasp, especially for those familiar with Autoencoders.

If you're interested in receiving updates about future articles on Neural Networks, including GANs, consider subscribing for email notifications.

Chapter 2: VAEs in the Machine Learning Landscape

To understand the role of VAEs in Machine Learning, it helps to visualize the various algorithms available. Organizing them can be tricky since they can be categorized in multiple ways based on their structure or the problems they aim to solve.

I have created a chart that reflects these dimensions, placing Neural Networks into a separate category. While Neural Networks are commonly applied in a supervised manner, it's important to recognize that some, like Autoencoders, operate more like Unsupervised or Self-Supervised algorithms.

Though VAEs share objectives with GANs, their architecture is more aligned with traditional Autoencoders, such as Undercomplete Autoencoders. You can explore VAEs in the Autoencoders section of the interactive chart below.

Section 3.1: Undercomplete Autoencoder Overview

An Undercomplete AE aims to effectively encode input data into a lower-dimensional latent space, also known as the bottleneck. This is accomplished by ensuring that the original inputs can be reconstructed with minimal loss through the decoder.

During training, the same dataset is provided to both the input and output layers as we seek to identify the optimal parameters for the latent space.

Section 3.2: Variational Autoencoder Architecture

Now, let's explore how the VAE diverges from an Undercomplete AE.

In a VAE, the latent space comprises distributions rather than individual points. Specifically, the inputs are mapped to a Normal distribution characterized by parameters Zμ (mean) and Zσ (variance), which are determined during the model's training.

The latent vector Z is then sampled from this distribution and sent to the decoder to generate the predicted outputs. The continuous nature of the VAE's latent space allows for sampling from any region, enabling the generation of new outputs, such as images.

Section 3.3: The Importance of Regularization

To produce "meaningful" outputs, encoding inputs into a distribution is only half the battle. Regularization is achieved through a term defined as the Kullback-Leibler divergence (KL divergence), which will be discussed in greater detail in the Python implementation section.

Section 3.4: Visualizing Latent Space

To illustrate how information is distributed within the latent space, consider this visualization:

Mapping data as isolated points fails to train the model to comprehend the relationships between them, making it impossible to generate new, meaningful data. In contrast, VAEs map data as distributions and regularize the latent space, providing a "gradient" or "smooth transition" that allows for generating new data closely aligned with the training set.

Section 4.1: Initial Setup

We will require the following resources:

• MNIST handwritten digit dataset (copyright held by Yann LeCun and Corinna Cortes under the Creative Commons Attribution-Share Alike 3.0 license; source: The MNIST Database)
• Numpy for data manipulation
• Matplotlib, Graphviz, and Plotly for visualizations
• TensorFlow/Keras for Neural Networks

Let's import the necessary libraries:

# Example code for importing libraries

import numpy as np

import matplotlib.pyplot as plt

import tensorflow as tf

This code displays the versions of the libraries used in this example.

Section 4.2: Loading and Preparing the Data

We will load the MNIST dataset and display the first ten digits. It's important to note that we will only utilize digit labels (y_train, y_test) for visualizations, not for model training.

The dataset contains 60,000 images for training and 10,000 images for testing, all sized at 28 x 28 pixels.

Next, we reshape the images from 28x28 to a flat array of 784 pixels.

# Reshaping the data

X_train = X_train.reshape((60000, 784))

X_test = X_test.reshape((10000, 784))

Section 4.3: Constructing the VAE Model

We will define a function for sampling from the latent space distribution Z using a reparameterization trick that allows loss to backpropagate through the mean (z-mean) and variance (z-log-sigma) nodes.

Next, we will build the Encoder model:

# Code to create the encoder model

The encoder outputs three sets of results: Z-mean, Z-log-sigma, and Z.

Now, let's create the Decoder model:

# Code to create the decoder model

Finally, we will combine the Encoder and Decoder models to form a complete VAE.

Section 4.4: Custom Loss Function

Before training the VAE, we need to define a custom loss function that incorporates KL divergence alongside the standard reconstruction loss (MSE) to ensure input and output images remain closely aligned.

Section 4.5: Training the VAE

With the Variational Autoencoder model assembled, we will proceed to train it for 25 epochs and visualize the loss chart.

In this video, we explore the concept of Variational Autoencoders and their practical applications in generating new images.

Section 4.6: Visualizing Latent Space and Generating New Digits

Since our latent space is two-dimensional, we can visualize the distribution of different digits within it.

For example, to generate a new image of the digit 3, we can select coordinates [0, 2.5] to produce an image resembling a digit 3.

# Code to generate new digits based on latent space sampling

Now, let's create 900 new digits from various areas of the latent space.

This video provides a deeper understanding of how VAEs can be utilized for image generation and manipulation.

Section 4.7: Conclusion

It's worth mentioning that VAEs can encode and generate significantly more complex data than just MNIST digits. I encourage you to build upon this simple tutorial by applying it to real-world datasets that are relevant to your field.

You can find the complete Jupyter Notebook containing all the code in my GitHub repository.

If you’d like to receive notifications when I publish new articles on Machine Learning or Neural Networks, such as Generative Adversarial Networks (GANs), please subscribe for updates.

Feel free to reach out with any questions or suggestions!

Best regards! 🤓

Saul Dobilas