Pytorch 1/21: Introduction


PyTorch is an open source machine learning library used for developing and training neural network based deep learning models. It is primarily developed by Facebook’s AI research group. PyTorch can be used with Python as well as a C++. Naturally, the Python interface is more polished. PyTorch (backed by Facebook) is immensely popular in research labs. Its popularity is growing on the production servers as well.

Let's take a look at some of the interesting aspects of the PyTorch library.

Starting with Basics, we look into the installation and some core concepts of PyTorch. Then, we dive deeper by implementing some of the common usecases in PyTorch. That can give us an idea of how to start with the library. After that, we will look at the PyTorch 3D and PyTorch Audio extensions - for image and audio processing using PyTorch. That can provide a sufficient introduction to start working on a PyTorch project.Beyond this, you can look up the PyTorch documentation.

Introduction


This meme aptly describes the growing popularity of PyTorch

loading...

Unlike TensorFlow and other such frameworks, Pytorch is a lot more intuitive and yet extremely powerful in its performance. It is designed for use on GPU's and hence provides a great performance. It is based on dynamic graphs, compared to the static graphs in TensorFlow. That causes a slight dip in the performance, but the gap is bridging pretty fast. It's syntax is a lot simpler than TensorFlow. Hence it is pretty useful for developing complex models. In most cases, the final deployment is on a C++ version of PyTorch - hence the performance is not a bottleneck anymore.

Let us now start with the basics of PyTorch. The foremost step is to install the library. If you are working on Google Colab, or other such platform, it is quite likely that PyTorch comes preinstalled. If you are working locally, you will have to install it.

Installation


Most DataScience/AI developers work with Anaconda, rather than a basic Python distribution. That is because Anaconda comes with a lot of features that we need. PyTorch can be installed on either.

$ conda install pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/vikas/dev/anaconda3

  added / updated specs:
    - pytorch


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _pytorch_select-0.2        |            gpu_0           2 KB
    conda-4.8.2                |           py37_0         2.8 MB
    cudatoolkit-10.0.130       |                0       261.2 MB
    cudnn-7.6.5                |       cuda10.0_0       165.0 MB
    ninja-1.9.0                |   py37hfd86e86_0         1.2 MB
    pytorch-1.3.1              |cuda100py37h53c1284_0       169.0 MB
    ------------------------------------------------------------
                                           Total:       599.2 MB

The following NEW packages will be INSTALLED:

  _pytorch_select    pkgs/main/linux-64::_pytorch_select-0.2-gpu_0
  cudatoolkit        pkgs/main/linux-64::cudatoolkit-10.0.130-0
  cudnn              pkgs/main/linux-64::cudnn-7.6.5-cuda10.0_0
  ninja              pkgs/main/linux-64::ninja-1.9.0-py37hfd86e86_0
  pytorch            pkgs/main/linux-64::pytorch-1.3.1-cuda100py37h53c1284_0

The following packages will be UPDATED:

  conda                                       4.7.12-py37_0 --> 4.8.2-py37_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
_pytorch_select-0.2  | 2 KB      | ################################################################################################################################################################## | 100% 
pytorch-1.3.1        | 169.0 MB  | ################################################################################################################################################################## | 100% 
cudatoolkit-10.0.130 | 261.2 MB  | ################################################################################################################################################################## | 100% 
conda-4.8.2          | 2.8 MB    | ################################################################################################################################################################## | 100% 
ninja-1.9.0          | 1.2 MB    | ################################################################################################################################################################## | 100% 
cudnn-7.6.5          | 165.0 MB  | ################################################################################################################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

$

The exact details of the output logs would be different, depending upon the status and version of your Anaconda installation. Now, we have a good installation of the PyTorch library. Let us now get back to some theory, to understand some of the core concepts of PyTorch.

Basic Syntax


PyTorch started as an alternative to NumPy - tailored for the GPU. It defines has a basic object - tensor. This is the counterpart of the NumPy multidimensional array. As we would expect, a PyTorch tensor provides a lot more than the NumPy array. We will check that later.

On the similarities, we can instantiate and initialize a PyTorch Tensor, just like a NumPy array.

Empty Tensor


We can create an empty (uninitialized tensor for a given shape), using:

import torch

x = torch.empty(2,3)
print(x)
.....
tensor([[4.2678e-17, 3.0852e-41, 4.2675e-17],
        [3.0852e-41, 8.9683e-44, 0.0000e+00]])

Note that the values in the uninitialized tensor are not created by PyTorch. This is the junk data that was already present in the memory that is allocated to the newly created tensor object.

Random Tensor


We can also create a tensor, initialized with random values.

import torch

x = torch.rand(2,3)
print(x)
.....
tensor([[0.3612, 0.3983, 0.9686],
        [0.6017, 0.6697, 0.8142]])

These are random values generated by PyTorch, and assigned to the individual elements of the tensor.

Zero Tensors


PyTorch provides us to create a new tensor initialized with 0's using the torch.zeros() method

import torch

x = torch.zeros(2,3)
print(x)
.....
tensor([[0., 0., 0.],
        [0., 0., 0.]])

Note that the datatype of each element is a float. We can insist on an int or a long by explicitly specifying the datatype as we create the tensor.

x = torch.zeros(2,3, dtype=torch.int)
print(x)
.....
tensor([[0, 0, 0],
        [0, 0, 0]], dtype=torch.int32)

Tensor from Data


The above methods created tensors based on predefined template values. We can explicitly specify the data to be filled in - by instantiating the tensor based on the data.

import torch

x = torch.tensor([5, 3, -1])
print(x)
.....
tensor([ 5,  3, -1])

x = torch.tensor([.5, 3, -1])
print(x)
.....
tensor([ 0.5000,  3.0000, -1.0000])

Note that PyTorch automatically infers the tensor datatype, based on the data types in the input array.

Tensor Operations


If you are reading this blog, I guess you already know what is addition or multiplication. I won't waste time, disk space, network bandwidth and browser memory, in trying to explain that. But there are some queer aspects of PyTorch that we should discuss here. PyTorch provides for multiple types of operation syntax.

import torch

# Initialize two tensors with random values.
x = torch.rand(2,3)
y = torch.rand(2,3)
print(x)
print(y)
.....
tensor([[0.4933, 0.3527, 0.1636],
        [0.9583, 0.3568, 0.8859]])
tensor([[0.7720, 0.1015, 0.2516],
        [0.9815, 0.9426, 0.9421]])

Simple, traditional addition operation

print(x + y)
.....
tensor([[1.2652, 0.4541, 0.4152],
        [1.9398, 1.2995, 1.8280]])

The torch version of addition

print(torch.add(x, y))
.....
tensor([[1.2652, 0.4541, 0.4152],
        [1.9398, 1.2995, 1.8280]])

This is another

z = torch.empty(2, 3) torch.add(x, y, out=z) print(z) ..... tensor([[1.2652, 0.4541, 0.4152], [1.9398, 1.2995, 1.8280]])

Yet Another.

print(x.add(y))
print(x)
.....
tensor([[1.2652, 0.4541, 0.4152],
        [1.9398, 1.2995, 1.8280]])
tensor([[0.4933, 0.3527, 0.1636],
        [0.9583, 0.3568, 0.8859]])

Note that after all this, the value of x has not changed. But, if we add an _ to the function name, the tensor is modified in place.

print(x.add_(y))
print(x)
.....
tensor([[1.2652, 0.4541, 0.4152],
        [1.9398, 1.2995, 1.8280]])
tensor([[1.2652, 0.4541, 0.4152],
        [1.9398, 1.2995, 1.8280]])

Indexing & Reshaping


PyTorch tensors support all the standard indexing operations of NumPy arrays.

print(x[:, 1])
.....
tensor([1.4541, 1.2995])

And also, reshaping tensors with the view() operator

import torch

x = torch.rand(2,3)
print(x)
.....
tensor([[0.5462, 0.5335, 0.3462],
        [0.9560, 0.4732, 0.6421]])

We can reshape it to a single dimension tensor

print(x.view(6))
.....
tensor([0.5462, 0.5335, 0.3462, 0.9560, 0.4732, 0.6421])

Or we can reshape it to another two dimensional tensor. Note that the dimension -1 indicates that its value should be inferred from the rest of the dimensions. In this case, it has to be 3.

print(x.view(-1,2))
.....
tensor([[0.5462, 0.5335],
        [0.3462, 0.9560],
        [0.4732, 0.6421]])

Item


If we have a one element tensor, we can use the item() method to extract the value.

import torch

x = torch.rand(1)
print(x)
print(x.item())
......
tensor([-0.4520])
-0.45202213525772095

PyTorch provides for many more of these operations that can simplify our code. For a detailed tutorial of these operators, check out the PyTorch Documentation.

NumPy Bridge


Similarly, we can convert PyTorch tensors to and from NumPy arrays using the simple NumPy bridge.

import torch
import numpy as np

a = torch.rand(4)
print(a)
.....
tensor([0.3368, 0.6063, 0.4710, 0.3740])

We can now get the NumPy array using the numpy() method

b = a.numpy()
print(b)
.....
[0.33675885 0.60627323 0.4709654  0.37395465]

This NumPy array is based out of the same data as the original tensor. It points to the same memory location. Thus, modifying the tensor in place will alter the array as well. Here is how:

print(a.add_(1))
print(b)
.....
tensor([1.3368, 1.6063, 1.4710, 1.3740])
[1.3367589 1.6062732 1.4709654 1.3739547]

Changing the elements in a, causes the corresponding change in b as well - because both point to the same memory. NumPy does not support modification in place. So the problem of reverse modification does not arise.

Similarly, we can extract a PyTorch tensor from a NumPy array using from_numpy(). Here again, modifying the tensor modifies the NumPy array.

import torch
import numpy as np

x = np.ones(5)
y = torch.from_numpy(x)

y.add_(1)
print(x)
.....
tensor([2., 2., 2., 2., 2.], dtype=torch.float64)
[2. 2. 2. 2. 2.]