11

Introduction to Deep Learning

In the previous chapter, we learned how to use modern machine learning models to tackle time series forecasting. Now, let’s focus our attention on a subfield of machine learning that has shown a lot of promise in the last few years—deep learning. We will be trying to demystify deep learning and go into why it is popular nowadays. We will also break down deep learning into major components and learn about the workhorse behind deep learning—gradient descent.

In this chapter, we will be covering these main topics:

Technical requirements

You will need to set up the Anaconda environment following the instructions in the Preface of the book to get a working environment with all the libraries and datasets required for the code in this book. Any additional libraries will be installed while running the notebooks.

The associated code for the chapter can be found at https://github.com/PacktPublishing/Modern-Time-Series-Forecasting-with-Python-2E/tree/main/notebooks/Chapter11.

What is deep learning and why now?

In Chapter 5, Time Series Forecasting as Regression, we talked about machine learning and borrowed a definition from Arthur Samuel: “Machine Learning is a field of study that gives computers the ability to learn without being explicitly programmed.” And we further saw how we can learn useful functions from data using machine learning. Deep learning is a subfield of this same field of study. The objective of deep learning is also to learn useful functions from data but with a few specifications on how it does that.

Before we talk about what is special about deep learning, let’s answer another question first. Why are we talking about this subfield of machine learning as a separate topic? The answer to that lies in the unreasonable effectiveness of deep learning methods in countless applications. Deep learning has taken the world of machine learning by storm, overthrowing state-of-the-art systems across types of data such as images, videos, text, and so on. If you remember the speech recognition systems on phones a decade ago, they were more meme-worthy than really useful. But today, you can say Hey Google, play Pink Floyd, and Comfortably Numb will start playing on your phone or speakers. Multiple deep learning systems made this process possible in a smooth way. The voice assistant on your phone, self-driving cars, web search, language translation—the list of applications of deep learning in our day-to-day lives just keeps on going.

By now, you might be wondering what this new technology called deep learning is all about, right? Deep learning is not a new technology. The origins of deep learning can be traced way back to the late 1940s and early 1950s. It only appears to be new because of the recent surge in popularity of the field.

Let’s quickly see why deep learning is suddenly popular.

Why now?

There are two main reasons why deep learning has gained a lot of ground in the last two decades:

Let’s discuss the preceding points in detail in the following sections.

Increase in compute availability

Back in 1960, Frank Rosenblatt wrote a paper (Reference 5) about a three-layer neural network and stated that it went a long way in demonstrating the ability of neural networks as a pattern-recognizing device. But in the same paper, he noted that the burden on a digital computer (of the 1960s) was too great as we increased the number of connections. However, in the decades that followed, computer hardware showed close to 50,000 times more improvement, which provided a good boost to neural networks and deep learning. However, it was still not enough as neural networks were still not considered to be good enough for large-scale applications.

This is when a particular type of hardware, which was initially developed for gaming, came to the rescue—GPUs. It’s not entirely clear who started using GPUs for deep learning. Kyoung-Su Oh and Keechul Jung published a paper titled GPU implementation of neural networks back in 2004, which seems to be the first to show massive speed-ups in using GPUs for deep learning. One of the earliest and more popular research papers on the topic came from Rajat Raina, Anand Madhavan, and Andrew Ng, who published a paper titled Large-scale deep unsupervised learning using graphics processors back in 2009. It showed the effectiveness of GPUs for deep learning.

Although many groups led by LeCun, Schmidhuber, Bengio, and so on were playing around with using GPUs, the turning point came when Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton used a GPU-based deep learning system that outperformed all the other competing technologies in an image recognition contest called the ImageNet Large Scale Visual Recognition Challenge 2012.

The introduction of GPUs provided a much-needed boost to the widespread use of deep learning and accelerated the progress in the field.

Reference check:

The research papers GPU implementation of neural networks, Large-scale deep unsupervised learning using graphics processors, and ImageNet Classification with Deep Convolutional Neural Networks are cited in the References section under 1, 2, and 3, respectively.

Increase in data availability

In addition to the skyrocketing compute capability, the other main factor that helped deep learning was the sheer increase in data. As the world became more and more digitized, the amount of data that we generated increased drastically. Tables that had hundreds and thousands of rows now exploded into millions and billions of rows, and the ever-decreasing cost of storage helped this explosion of data collection.

And why would an increase in data availability help deep learning? This lies in the way deep learning works. Deep learning is quite data-hungry and needs large amounts of data to learn good models. Therefore, if we keep increasing the data that we provide to a deep learning model, the model will be able to learn better and better functions. However, the same can’t be said for traditional machine learning models. Let’s cement this learning with a chart that Andrew Ng, a world-renowned machine learning educator and an adjunct professor at Stanford, popularized in his famous machine learning course—Machine Learning by Stanford University on Coursera (https://www.coursera.org/specializations/machine-learning-introduction) (Figure 11.1).

Figure 11.1 – Deep learning versus traditional machine learning as we increase the data size

Figure 11.1: Deep learning versus traditional machine learning as we increase the data size

In Figure 11.1, which was popularized by Andrew Ng, we can see that as we increase the data size, traditional machine learning hits a plateau and won’t improve anymore.

It has been proven empirically that there are significant benefits to the overparameterization of a deep learning model. Overparameterization means that there are more parameters in the model than the number of data points available to train. In classical statistics, this is a big no-no because, under this scenario, the model invariably overfits. But deep learning seems to flaunt this rule with ease. One of the examples of overparameterization is the current state-of-the-art image recognition system, NoisyStudent. It has 480 million parameters, but it was trained on ImageNet with 1.2 million data points.

It has been argued that the way deep learning models are trained (stochastic gradient descent, which we will be explaining soon) is the key because it has a regularizing effect. In a research paper titled The Computational Limits of Deep Learning, Niel C. Thompson and others tried to illustrate this using a simple experiment. They set up a dataset with 1,000 features, but only 10 of them had any signal in them. Then, they tried to learn four models based on the dataset using varying dataset sizes:

Let’s see Figure 11.2 from the research paper with the study:

Figure 11.2 – The chart shows how different models perform under different sizes of data

Figure 11.2: The chart shows how different models perform under different sizes of data

The chart has a number of data points used on the x-axis and the performance (-log(Mean Squared Error)) on the y-axis. The different colored lines show the different types of models. The Oracle model sets the upper limit for learning because it has access to perfect information.

The expert model plateaus because there is a definite lack of information as it doesn’t have access to one of the ten features that are important. The flexible model (which uses all 1,000 features) takes a large number of data points to start recognizing the important ones but still keeps approaching the Oracle performance as the data size gets bigger. The regularized model (which is a proxy for deep learning models) keeps improving as we give the model more and more data. This model uses regularization to figure out which of these 1,000 features is relevant to the problem and starts to leverage them with much fewer data points, and performance keeps increasing with more data points. This strengthens the concept Andrew Ng popularized once more—with more data, deep learning starts to outperform traditional machine learning.

A lot more factors, apart from compute and data availability, have contributed to the success of deep learning. Sara Hooker, in her essay The Hardware Lottery (Reference 9), talks about how an idea wins not necessarily because it is superior to other ideas but because it is suited to the software and hardware available at the time. And once a research direction wins the lottery, it snowballs because more funding and big research organizations get behind that idea and it eventually becomes the most prominent idea in the space of ideas.

We have talked about deep learning for some time but have still not understood what it is. Let’s do that now.

What is deep learning?

There is no single definition of deep learning because it means slightly different things to different people. However, a large majority of people agree on one thing: a model is called deep learning when it involves automatic feature learning from raw data. As Yoshua Bengio (a Turing Award winner and one of the godfathers of AI) explains in his 2021 paper titled Deep Learning of Representations for Unsupervised and Transfer Learning:

”Deep learning algorithms seek to exploit the unknown structure in the input distribution in order to discover good representations, often at multiple levels, with higher-level learned features defined in terms of lower-level features.”

In a 2016 presentation, Deep Learning and Understandability versus Software Engineering and Verification, Peter Norvig, the Director of Research at Google, had a similar but simpler definition:

”A kind of learning where the representation you form have (sic) several levels of abstraction, rather than a direct input to output.”

Another key feature of deep learning a lot of people agree upon is compositionality. Yann LeCun, a Turing Award winner and another one of the godfathers of AI, has a slightly more complex but more exact definition of deep learning (tweet from @ylecun on January 9th, 2020):

”DL is methodology: building a model by assembling parameterized modules into (possibly dynamic) graphs and optimizing it with gradient-based methods.”

The key points we would like to highlight here are as follows:

If you have read anything about deep learning before, you may have seen neural networks and deep learning used together or interchangeably. But we haven’t talked about neural networks till now. Before we do that, let’s look at a fundamental unit of any neural network.

Perceptron, the first neural network

A lot of what we call deep learning and neural networks are deeply influenced by the human brain and its inner workings. Although recent studies have shown very little similarity between human brains and artificial neural networks, the seed behind the idea was inspired by human biology. The human desire to create intelligent beings like themselves was manifested as early as back in Greek mythology (Galatea and Pandora). And owing to this desire, humans have studied and looked for inspiration from human anatomy for years. One of the organs of the human body that has been studied intensely is the brain because it is the center of intelligence, creativity, and everything else that makes a human.

Even though we still don’t know a lot about the brain, we do know a bit about it, and we use that little information to design artificial systems. The fundamental unit of the human brain is something we call a neuron, as shown here:

Figure 11.3 – A biological neuron

Figure 11.3: A biological neuron

Many of you might have come across this in biology or in the context of machine learning as well. But let’s refresh this anyway. The biological neuron has the following parts:

McCulloch and Pitts (1943) were the first to design a mathematical model for the biological neuron. However, the McCulloch-Pitts model had a few limitations:

In 1957, Frank Rosenblatt generalized the McCulloch-Pitts model and made it a full model whose parameters could be learned. The similarity between modern deep learning networks and the human brain ends here. The fundamental unit of learning that started this line of research was inspired by human biology and was a rather cheap imitation of it as well.

Reference check:

The original research paper for Frank Rosenblatt’s perceptron is cited in References under reference 5.

Let’s understand the perceptron in detail because it is the fundamental building block of all neural networks:

Figure 11.4 – Perceptron

Figure 11.4: Perceptron

As we see from Figure 11.4, the perceptron has the following components:

We can write the perceptron in the mathematical form as follows:

Figure 11.5: Perceptron, a math perspective

As shown in Figure 11.5, the perceptron output is defined by the weighted sum of inputs, which is passed in through a non-linear function. Now, we can think of this using linear algebra as well. This is an important perspective for two reasons:

If we consider the inputs as vectors in the feature space (vector space with m-dimensions), the term $\sum_{i=1}^{m} x_i w_i$ is nothing but a linear combination of input vectors. We can rewrite the equation in vector form as below:

$$\hat{y} = g(\mathbb{X}^T \mathbb{W})$$

where, $\mathbb{X} = \begin{bmatrix} 1 \\ x_1 \\ ... \\ x_m \end{bmatrix}, \ \mathbb{W} = \begin{bmatrix} w_0 \\ w_1 \\ ... \\ w_m \end{bmatrix}$

The bias is also included here as a dummy input with a fixed value of 1 and adding $w_0$ to the $\mathbb{W}$ vector.

Now that we have had an introduction to deep learning, let us recall one of the aspects of deep learning we discussed earlier—compositionality—and explore it a bit more deeply in the next section.

Components of a deep learning system

Let us recall Yann LeCun’s definition of deep learning:

”Deep learning is a methodology: building a model by assembling parameterized modules into (possibly dynamic) graphs and optimizing it with gradient-based methods.”

The core idea here is that deep learning is an extremely modular system. Deep learning is not just one model but, rather, a language to express any model in terms of a few parametrized modules with these specific properties:

  1. It should be able to produce an output from a given input through a series of computations.
  2. If the desired output is given, it should be able to pass on information to its inputs on how to change, to arrive at the desired output. For instance, if the output is lower than what is desired, the module should be able to tell its inputs to change in some direction so that the output becomes closer to the desired one.

The more mathematically inclined may have figured out the connection to the second point of differentiation. And you would be correct. To optimize these kinds of systems, we predominantly use gradient-based optimization methods. Therefore, condensing the two properties into one, we can say that these parameterized modules should be differentiable functions.

Let’s take the help of a visual to aid further discussion.

Figure 11.6 – A deep learning system

Figure 11.6: A deep learning system

As shown in Figure 11.6, deep learning can be thought of as a system that takes in raw input data through a series of linear and non-linear transforms to provide us with an output. It also can adjust its internal parameters to make the output as close as possible to the desired output through learning. To make the diagram simpler, we have chosen a paradigm that fits most of the popular deep learning systems. It all starts with raw input data. The raw input data goes through N blocks of linear and non-linear functions that do representation learning. Let’s explore these blocks in some detail.

Representation learning

Representation learning, informally, learns the best features by which we can make the problem linearly separable. Linearly separable means when we can separate the different classes (in a classification problem) with a straight line (Figure 11.7):

Figure 11.7 – Transforming non-linearly separable data into linearly separable using a function,

Figure 11.7: Transforming non-linearly separable data into linearly separable using a function, $\phi$

The representation learning block in Figure 11.6 may have multiple linear and non-linear functions stacked on top of each other, and the overall function of the block is to learn a function, $\phi$, which transforms the raw input into good features that make the problem linearly separable.

Another way to look at this is through the lens of linear algebra. As we explored earlier in the chapter, matrix multiplication can be thought of as a linear transformation of vectors. And if we extend that intuition to the vector spaces, we can see that matrix multiplication warps the vector space in some way or another. When we stack multiple linear and non-linear transformations on top of each other, we are essentially warping, twisting, and squeezing the input vector space (with the features) into another space. When we ask a parameterized system to warp the input space (pixels of images) in such a way as to perform a particular task (such as the classification of dogs versus cats), the representation learning block learn the right transformations, which makes the task (separating cats from dogs) easier.

I have created a video illustrating this because nothing establishes intuition better than a video of what is happening. I’ve taken a sample dataset that is not linearly separable, trained a neural network on the problem to classify, and then visualized how the input space was transformed by the model into a linearly separable representation. You can find the video here: https://www.youtube.com/watch?v=5xYEa9PPDTE.

Now, let’s look inside the representation learning block. We can see there is a linear transformation and a non-linear activation.

Linear transformation

Linear transformations are just transformations that are applied to the vector space. When we say linear transformation in a neural network context, we actually mean affine transformations.

A linear transformation fixes the origin while applying the transformation, but an affine transformation doesn’t. Rotation, reflection, scaling, and so on are purely linear transformations because the origin won’t change while we do this. But something like a translation, which moves the vector space, is an affine transformation. Therefore, A*XT is a linear transformation, but A*XT +b is an affine transformation.

So, linear transformations are simply matrix multiplications that transform the input vector space, and this is at the heart of any neural network or deep learning system today.

What happens if we stack linear transformations on top of each other? For instance, we first multiply the input, X, with a transformation matrix, A, and then multiply the results with another transformation matrix, B:

$$T = B \cdot (A \cdot X)$$

By the associative property (which is applicable to linear algebra as well), we can rewrite this equation as follows:

$$T = (B \cdot A) \cdot X$$

Generalizing this to a stack of N transformation matrices, we can see that it all works out to be a single linear transformation. This kind of defeats the purpose of stacking N layers, doesn’t it?

This is where the non-linearity becomes essential and we introduce non-linearities by using a non-linear function, which we call activation functions.

Activation functions

Activation functions are non-linear differentiable functions. In a biological neuron, the axon hillock decides whether to fire a signal based on the inputs. The activation functions serve a similar function and are key to the neural network’s ability to model non-linear data. In other words, activation functions are key in neural networks’ ability to transform input vector space (which is linearly inseparable) to a linearly separable vector space, informally. To unwarp a space such that linearly inseparable points become linearly separable, we need to have non-linear transformations.

We repeated the same experiment we did in the last section, where we visualized the trained transformation of a neural network on the input vector space, but this time, without any non-linearities. The resulting video can be found here: https://www.youtube.com/watch?v=z-nV8oBpH2w. The best transformation that the model learned is just not sufficient and the points are still linearly inseparable.

Theoretically, an activation function can be any non-linear differentiable (differentiable almost everywhere, to be exact) function. However, over the course of time, there are a few non-linear functions that are popularly used as activation functions. Let’s look at a few of them.

Sigmoid

Sigmoid is one of the most common activation functions around, and probably one of the oldest. It is also known as the logistic function. When we discussed perceptron, we mentioned a step (also called Heaviside in literature) function as the activation function. The step function is not a continuous function and hence is not differentiable everywhere. A very close substitute is the sigmoid function.

It is defined as follows:

$$g(x) = \frac{1}{1 + e^{-x}}$$

Where g is the sigmoid function and x is the input value.

Sigmoid is a continuous function and therefore is differentiable everywhere. The derivative is also computationally simpler to calculate. Because of these properties of the sigmoid, it was adopted widely in the early days of deep learning as a standard activation function.

Let’s see what a sigmoid function looks like and how it transforms a vector space:

Figure 11.8 – Sigmoid activation function (left) and original, and activated vector space (middle and right)

Figure 11.8: Sigmoid activation function (left) and original and activated vector space (middle and right)

The sigmoid function squashes the input between 0 and 1, as seen in Figure 11.8 (left). We can observe the same phenomenon in the vector space. One of the drawbacks of the sigmoid function is that the gradients tend to zero on the flat portions of the sigmoid. When a neuron approaches this area in the function, the gradients that it receives and propagates become negligible and the unit stops learning. We call this saturating of the activation. Because of this, nowadays, sigmoid is not typically used in deep learning, except in the output layer (we will be talking about this usage soon).

Hyperbolic tangent (tanh)

Hyperbolic tangents are another popular activation. They can be easily defined as follows:

$$\tanh(x) = \frac{\sinh(x)}{\cosh(x)}$$

It is very similar to sigmoid. In fact, we can express tanh as a function of sigmoid. Let’s see what the activation function looks like:

Figure 11.9 – TanH activation function (left) and original, and activated vector space (middle and right)

Figure 11.9: Tanh activation function (left) and original and activated vector space (middle and right)

We can see that the shape is similar to sigmoid, although a bit sharper. But the key difference is that the tanh function outputs a value between -1 and 1. And because of the sharpness, we can also see the vector space getting pushed out to the edges as well. The fact that the function outputs a value that is symmetrical around the origin (0) works well with the optimization of the network and hence tanh was preferred over sigmoid. But since the tanh function is also a saturating function, the same problem of very small gradients hampering the flow of gradients and, in turn, learning plagues tanh activations as well.

Rectified linear units and variants

As neuroscience gained more information about the human brain, researchers found out that only one to four percent of neurons in the brain are activated at any time. But with all the activation functions such as sigmoid or tanh, almost half of the neurons in a network are activated. In 2010, Vinod Nair and Geoffrey Hinton proposed rectified linear units (ReLUs) in the seminal paper Rectified Linear Units Improve Restricted Boltzmann Machines. Since then, ReLUs have taken over as the de facto activation functions for deep neural networks.

ReLU

A ReLU is defined as follows:

g(x) = max(x, 0)

It is just a linear function but with a kink at zero. Any value greater than zero is retained as is, but all values below zero are squashed to zero. The range of the output goes from 0 to $\infty$. Let’s see how it looks visually:

Figure 11.10 – ReLU activation function (left) and original, and activated vector space (middle and right)

Figure 11.10: ReLU activation function (left) and original and activated vector space (middle and right)

We can see that the points in the left and bottom quadrants are all pushed into the axes’ lines. This squashing is what gives the non-linearity to the activation function. Because of the way the activation sharply becomes zero and does not tend to zero like the sigmoid or tanh, ReLUs are non-saturating.

Reference check:

The research paper that proposed ReLU is cited in References under reference 7.

There are a few advantages to using ReLUs:

But ReLUs are not without problems:

Let’s see a few variants that try to resolve the problems we discussed for ReLU.

Leaky ReLU and parametrized ReLU

Leaky ReLU is a variant of standard ReLU that resolves the dead ReLU problem. It was proposed by Maas and others in 2013. A leaky ReLU can be defined as follows:

$$g(x) = \begin{cases} x, & \text{if } x \geq 0 \\ \alpha x, & \text{if } x \lt 0 \end{cases}$$

Here, $\alpha$ is the slope parameter (typically set to a very small value such as 0.001) and is considered a hyperparameter. This makes sure the gradients are not zero when x < 0 and thereby ensures there are no dead ReLUs. But the sparsity that ReLU provides is lost here because there is no zero output that turns off a unit completely. Let’s visualize this activation function:

Figure 10.11 – Leaky ReLU activation function (left) and original, and activated vector space (middle and right)

Figure 11.11: Leaky ReLU activation function (left) and original and activated vector space (middle and right)

In 2015, K. He and others proposed another minor modification to leaky ReLU called parametrized ReLU. In parametrized ReLU, instead of considering $\alpha$ as a hyperparameter, they considered it as a learnable parameter.

Rerefence check:

The research paper that proposed leaky ReLU is cited in References under reference 8, and parametrized ReLU is cited under reference 9.

There are many other activation functions that are less popularly used but still have enough use cases to be included in PyTorch. You can find a list of them here: https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity. We encourage you to use the notebook titled 02-Activation_Functions.ipynb in the Chapter 11 folder to try out different activation functions and see how they warp the vector space.

And with that, we now have an idea of the components of the first block in Figure 11.6, representation learning. The next block in there is the linear classifier, which has a linear transformation and an output activation. We already know what a linear transformation is, but what is an output activation?

Output activation functions

Output activation functions are functions that enforce a few desirable properties to the output of the network.

Additional reading:

These functions have a deeper connection with maximum likelihood estimation (MLE) and the chosen loss function, but we will not be getting into that because it is out of the scope of this book. We have linked to the book Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville in the Further reading section. If you are interested in a deeper understanding of deep learning, we suggest you use the book to that effect.

If we want the neural network to predict a continuous number in the case of regression, we just use a linear activation function (which is like saying there is no activation function). The raw output from the network is considered the prediction and fed into the loss function.

But in the case of classification, the desired output is a class out of all possible classes. If there are only two classes, we can use our old friend, the sigmoid function, which has an output between 0 and 1. We can also use tanh because its output is going to be between -1 and 1. The sigmoid function is preferred because of the intuitive probabilistic interpretation that comes along with it. The closer the value is to one, the more confident the network is about that prediction.

Now, sigmoid works for binary classification. What about multiclass classification where the possible classes are more than two?

Softmax

Softmax is a function that converts a vector of K real values into another K-positive real value, which sums up to 1. Softmax is defined as follows:

$$Softmax(x_i) = \frac{e^{x_i}}{\sum_{j=1}^{K} e^{x_j}}$$

This function converts the raw output from a network into something that resembles a probability across K classes. This has a strong relation with sigmoid—sigmoid is a special case of softmax when K = 2. In the following figure, let’s see how a random vector of size 3 is converted into probabilities that add up to 1:

Figure 11.12 – Raw output versus softmax output

Figure 11.12: Raw output versus softmax output

If we look closely, we can see that in addition to converting the real values into something that resembles probability, it also increases the relative gap between the maximum and the rest of the values. This activation is a standard output activation for multiclass classification problems.

Now, there is only one major component left in the diagram (Figure 11.6)—the loss function.

Loss function

The loss function we touched upon in Chapter 5, Time Series Forecasting as Regression, translates nicely to deep learning. In deep learning also, the loss function is a way to tell how good the predictions of the model are. If the predictions are way off the target, the loss function will be higher, and as we get closer to the truth, it becomes smaller. In the deep learning paradigm, we just have one more additional requirement of the loss function—it should be differentiable.

Common loss functions from classical machine learning, such as mean squared error or mean absolute error, are valid in deep learning as well. In fact, in regression tasks, they are the default choices that practitioners adopt. For classification tasks, we adopt a concept borrowed from information theory called cross-entropy loss. However, since deep learning is a very flexible framework, we can use any loss function as long as it is differentiable. There are a lot of loss functions people have already tried and found work in many situations. A lot of them are part of PyTorch’s API as well. You can find them here: https://pytorch.org/docs/stable/nn.html#loss-functions.

Now that we have covered all the components of a deep learning system, let’s also briefly look at how we train the whole system.

Forward and backward propagation

In Figure 11.6, we can see two sets of arrows, one going toward the desired output from the input, marked as Forward Computation, and another going backward to the input from the desired output, marked Backward Computation. These two steps are at the core of learning a deep learning system. In Forward Computation, popularly known as forward propagation, we use the series of computations that are defined in the layers and propagate the input all the way through the network to get the output. Now that we have the output, we would use the loss function to assess how close or far we are from the desired output. This information is now used in Backward Computation, popularly known as backward propagation, to calculate the gradient with respect to all the parameters.

Now, what is a gradient and why do we need it? In high school math, we might have come across gradients or derivatives in another form called slope. It is the rate of change of a quantity when we change a variable by unit measure. Derivatives inform us of the local slope of a scalar function. While derivatives are always with respect to a single variable, gradients are a generalization of derivatives to multivariate functions. Intuitively, both gradients and derivatives inform us of the local slope of the function. And with the gradient of the loss function, we can use one of the techniques from mathematical optimization, called gradient descent, to optimize our loss function.

Let’s see this with an example.

Gradient descent

Any machine learning or deep learning model can be thought of as a function that converts an input, x, to an output, $\hat{y}$, using a few parameters, $\theta$. Here, $\theta$ can be the collection of all the matrix transformations that we do to the input throughout the network. But to simplify the example, let’s assume there are only two parameters, a and b. If we think about the whole process of learning a bit, we will see that by keeping the input and expected output the same, the way to change your loss would be by changing the parameters of the model. Therefore, we can postulate the loss function to be parameterized by the parameters–in this case, a and b.

Notebook alert:

To follow along with the complete code, use the notebook named 03-Gradient_Descent.ipynb in the Chapter11 folder and the code in the src folder.

Let’s assume the loss function takes the following form:

$$\mathcal{L}(a, b) = (a - 8)^2 + (b - 2)^2$$

Let’s see what the function looks like. We can use a three-dimensional plot to visualize a function with two parameters, as seen in Figure 11.13. Two dimensions will be used to denote the two parameters and, at each point in that two-dimensional mesh, we can plot the loss value in the third dimension.

This kind of plot of the loss function is also called a loss curve (in univariate settings), or loss surface (in multivariate settings).

Figure 11.13 – Loss surface plot

Figure 11.13: Loss surface plot

The lighter portion of the 3D shape is where the loss function is less, and as we move away from there, it increases.

In machine learning, our aim is to minimize the loss function, or in other words, find the parameters that make our predicted output as close as possible to the ground truth. This falls under the realm of mathematical optimization, and a particular technique lends itself suitable for this approach—gradient descent.

Gradient descent is a mathematical optimization algorithm used to minimize a cost function by iteratively moving in the direction of the steepest descent. In a univariate function, the derivative (or the slope) gives us the direction (and magnitude) of the steepest ascent. For instance, if we know that the slope of a function is 1, we know if we move to the right, we are climbing up the slope, and moving to the left, we will be climbing down. Similarly, in the multivariate setting, the gradient of a function at any point will give us the direction (and magnitude) of the steepest ascent. And since we are concerned with minimizing a loss function, we will be using the negative gradient, which will point us in the direction of the steepest descent.

So, let’s define the gradient for our loss function. We are using high school calculus, but even if you are not comfortable, you don’t need to worry:

$$\nabla f(a, b) = \begin{bmatrix} \frac{df}{da} \\ \frac{df}{db} \end{bmatrix} = \begin{bmatrix} 2(a - 8) \\ 2(b - 2) \end{bmatrix}$$

Now, how does the algorithm work? Very simply, as follows:

  1. Initialize the parameters to random values.
  2. Compute the gradient at that point.
  3. Make a step in the direction opposite to the gradient.
  4. Repeat steps 2 and 3 until it converges or we reach maximum iterations.

There is just one more aspect that needs more clarity: how much of a step do we take in each iteration?

Ideally, the magnitude of the gradient tells you how fast the function is changing in that direction, and we should just take the step equal to the gradient. But there is a property of the gradient that makes that a bad idea. The gradient only defines the direction and magnitude of the steepest ascent in the infinitesimally small locality of the current point and is blind to what happens beyond it. Therefore, we use a hyperparameter, commonly called the learning rate, to temper the steps we take in each iteration. Therefore, instead of taking a step equal to the gradient, we take a step equal to the learning rate multiplied by the gradient.

Mathematically, if $\theta$ is the vector of parameters, at each iteration, we update the parameters using the following formula:

$$\theta = \theta - \eta \times \Delta f(a, b)$$

Here, $\eta$ is the learning rate, and $\Delta f$ is the gradient at the point.

Let’s see a very simple implementation of gradient descent (refer to the notebook in the GitHub repository for the end-to-end definition and working code). First, let’s define a function that returns the gradient at any point:

def gradient(a, b):
    return 2*(a-8), 2*(b-2)

Now we define a few initial parameters such as the maximum iterations, learning rate, and initial value of a and b:

# maximum number of iterations that can be done
maximum_iterations = 500
# current iteration
current_iteration = 0
# Learning Rate
learning_rate = 0.01
#Initial value of a, b
current_a_value = 28
current_b_value = 27
Now, all that is left is the actual process of gradient descent:
while current_iteration < maximum_iterations:
    previous_a_value = current_a_value
    previous_b_value = current_b_value
    # Calculating the gradients at current values
    gradient_a, gradient_b = gradient(previous_a_value, previous_b_value)
    # Adjusting the parameters using the gradients
    current_a_value = current_a_value - learning_rate * gradient_a * (previous_a_value)
    current_b_value = current_b_value - learning_rate * gradient_b * (previous_b_value)
    current_iteration = current_iteration + 1

We know the minimum for this function will be at a = 8 and b = 2 because that would make the loss function zero. And gradient descent finds a solution that is pretty accurate—a = 8.000000000000005 and b = 2.000000002230101. We can also visualize the path it took to reach the minimum, as seen in Figure 11.14:

Figure 11.14 – Gradient descent optimization on the loss surface

Figure 11.14: Gradient descent optimization on the loss surface

We can see that even though we initialized the parameters far from the actual origin, the optimization algorithm makes a direct path to the optimal point. At each point, the algorithm looks at the gradient of the point, moves in the opposite direction, and eventually converges on the optimum.

When gradient descent is adopted in a learning task, there are a few kinks to be noted. Let’s say we have a dataset of N samples. There are three popular variants of gradient descent that are used in learning and each of them has its pros and cons.

Batch gradient descent

We run all N samples through the network and average the losses across all N instances. Now, we use this loss to calculate the gradient, make a step in the right direction, and repeat.

The pros are as follows:

The cons are as follows:

Stochastic gradient descent (SGD)

In SGD, we randomly sample one instance from N samples, calculate the loss and gradients, and then update the parameters.

The pros are as follows:

The cons are as follows:

Mini-batch gradient descent

Mini-batch gradient descent is a technique that falls somewhere between the spectrum of batch gradient descent and SGD. In this variant, we have another quality called mini-batch size (or simply batch size), b. In each optimization step, we randomly pick b instances from N samples and calculate gradients on the average loss of all b instances. With b = N, we have batch gradient descent, and with b = 1, we have stochastic gradient descent. This is the most popular way neural networks are trained today. By varying the batch size, we can travel between the two variants and manage the pros and cons of each option.

Nothing develops intuition better than a visual playground where we can see the effects of the different components we discussed. Tensorflow Playground is an excellent resource (see the link in the Further reading section) to do just that. I strongly urge you to head over there and play with the tool, train a few neural networks right in the browser, and see in real time how the learning happens.

Summary

We kicked off a new part of the book with an introduction to deep learning. We started with a bit of history to understand why deep learning is so popular today and we also explored its humble beginnings in perceptron. We understood the composability of deep learning and understood and dissected the different components of deep learning, such as the representation learning block, linear layers, activation functions, and so on. Finally, we rounded off the discussion by looking at how a deep learning system uses gradient descent to learn from data. With that understanding, we are now ready to move on to the next chapter, where we will drive the narrative toward time series models.

References

Following is the list of references used throughout this chapter:

  1. Kyoung-Su Oh and Keechul Jung. (2004), GPU implementation of neural networks. Pattern Recognition, Volume 37, Issue 6, 2004: https://doi.org/10.1016/j.patcog.2004.01.013.
  2. Rajat Raina, Anand Madhavan, and Andrew Y. Ng. (2009), Large-scale deep unsupervised learning using graphics processors. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML ‘09): https://doi.org/10.1145/1553374.1553486.
  3. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. (2012), ImageNet Classification with Deep Convolutional Neural Networks. Commun. ACM 60, 6 (June 2017), 84–90: https://doi.org/10.1145/3065386.
  4. Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. (2020). The Computational Limits of Deep Learning. arXiv:2007.05558v1 [cs.LG]: https://arxiv.org/abs/2007.05558v1.
  5. Frank Rosenblatt. (1957), The perceptron – A perceiving and recognizing automaton, Technical Report 85-460-1, Cornell Aeronautical Laboratory.
  6. Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim. (2001). On the Surprising Behavior of Distance Metrics in High Dimensional Spaces. In Proceedings of the 8th International Conference on Database Theory (ICDT ‘01). Springer-Verlag, Berlin, Heidelberg, 420–434: https://dl.acm.org/doi/10.5555/645504.656414.
  7. Nair, V. and Hinton, G.E. (2010). Rectified Linear Units Improve Restricted Boltzmann Machines. ICML: https://icml.cc/Conferences/2010/papers/432.pdf.
  8. Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. (2013). Rectifier nonlinearities improve neural network acoustic models. ICML Workshop on Deep Learning for Audio, Speech, and Language Processing: https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf.
  9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. 2015 IEEE International Conference on Computer Vision (ICCV), 1026-1034: https://ieeexplore.ieee.org/document/7410480.
  10. Sara Hooker. (2021). The hardware lottery. Commun. ACM, Volume 64: https://doi.org/10.1145/3467017.

Further reading

You can check out the following sources if you want to read more about a few topics covered in this chapter: