What Is PyTorch Autograd?

This beginner-friendly Pytorch code introduces you to PyTorch autograd and explains how it works using a simple PyTorch example.

Objective: What Is PyTorch Autograd?

This PyTorch code example will introduce you to PyTorch Autograd with the help of a PyTorch tensor example. 

What Is Autograd in PyTorch?

PyTorch Autograd is a package for automatic differentiation in PyTorch. It allows you to compute the gradients of any PyTorch computation with respect to any PyTorch tensor. Autograd works by tracking all of the operations performed on PyTorch tensors. Once you have finished computing a loss function, you can call the backward() method on the output tensor to compute the gradients of the loss function with respect to all of the input tensors.

How To Optimize Neural Networks Using Autograd PyTorch?

There are several ways to optimize neural networks using PyTorch Autograd. Some of the most common methods include-

  • Using a gradient optimizer algorithm that uses the loss function's gradients to update the neural network's parameters.

  • Using a learning rate scheduler algorithm that adjusts the learning rate of the gradient optimizer over time.

  • Using regularization techniques, such as L1 and L2 regularization, to prevent the neural network from overfitting.

How To Initialize A PyTorch Autograd Variable?

You can initialize a PyTorch Autograd variable using the torch.tensor() function. When you call torch.tensor() with the requires_grad=True keyword argument, you create a variable that PyTorch Autograd will track.

For example,

import torch

x = torch.tensor(1.0, requires_grad=True)

Setting An Existing Tensor To Autograd PyTorch

You can set an existing tensor to Autograd in PyTorch using the .requires_grad_() method. This method will set the requires_grad attribute of the tensor to True, which will enable Autograd to track the tensor.

import torch

x = torch.tensor(1.0)

x.requires_grad_()

PyTorch: How To Extract The Gradient From Autograd?

You can extract the gradient from Autograd in PyTorch using the .grad attribute. The .grad attribute is a tensor that stores the gradient of the tensor with respect to all of its input tensors.

import torch

def f(x):

  y = x**2

  return y

x = torch.tensor(1.0, requires_grad=True)

y = f(x)

# Compute the gradients of y with respect to x

y.backward()

# Extract the gradient of y with respect to x

gradient = x.grad

print(gradient)

How To Perform PyTorch Disable Autograd?

There are three ways to disable PyTorch Autograd-

  • Set the requires_grad attribute of a tensor to False. This will prevent the tensor from being tracked by Autograd.

  • Use the torch.no_grad() context manager. This will disable Autograd for all tensors inside the context manager.

  • Use the .detach() method to create a copy of a tensor that Autograd does not track.

Steps Involved In The PyTorch Autograd Function Example

The following steps will show you how PyTorch Autograd works with the help of an easy-to-understand PyTorch tensor example.

Step 1 - Import Library For PyTorch Tensor Autograd

First, you must import the required libraries.

import torch

Step 2 - Take Sample PyTorch Tensors

In the next step, you will take two sample tensors with tensor_1 and tensor_2, where you must keep requires_grad = True for both tensors. This will send signals to autograd that every operation should be tracked.

tensor_1 = torch.tensor([4., 5.], requires_grad=True)

tensor_2 = torch.tensor([7., 9.], requires_grad=True)

Get Closer To Your Dream of Becoming a Data Scientist with Solved End-to-End PyTorch Projects

Step 3 - Perform Calculations For PyTorch Tensor Autograd

You will take the cube of tensor_1 and square of tensor_2.

calc = 3*tensor_1**3 - tensor_2**2

Step 4 - Call PyTorch Autograd backward() Function

In this step, you will call the backward() PyTorch Autograd function, as shown in the code below. Let us assume that the tensors you have taken, "tensor_1" and "tensor_2", are parameters of a neural network where the "calc" is the error. In the neural network training, you need the error gradient with respect to parameters, i.e., dQ/dtensor_1 = 9tensor_1^2 dQ/dtensor_2 = -2tensor_2. When you call the ".backward()" on the "calc" then the autograd calculates these gradients and stores them in the respective tensors which ".grad" attribute. You need to explicitly pass the "gradient" attribute in the "calc.backward()" because it's a vector. The "gradient" is a tensor of the same shape as the "calc" and it represents the gradient of Q w.r.t. itself, i.e. dQ/dQ = 1

new_grad = torch.tensor([1.,1.])

calc.backward(gradient=new_grad)

Step 5 - Check The Collected Gradients

The final step is to print the collected gradients.

print(9*tensor_1**2 == tensor_1.grad)

print(-2**tensor_2 == tensor_2.grad)

The output of the above code is-

tensor([True, True])

tensor([False, False])

PyTorch Autograd Backward Error

A PyTorch Autograd backward error is an error that occurs when Autograd is unable to compute the gradients of a PyTorch computation. Some of the most common errors include-

  • RuntimeError: Trying to go backward through the graph a second time. This error occurs when you call the backward() method on a tensor that has already been backpropagated.

  • RuntimeError: Function <function_name> returned an invalid gradient at index <index>. This error occurs when a PyTorch function returns a gradient that is not valid. 

  • ValueError: Cannot call backward on a scalar tensor. This error occurs when you try to call the backward() method on a scalar tensor. 

Learn More About PyTorch Autograd Implementation With ProjectPro

This PyTorch Autograd example offers a comprehensive understanding of PyTorch's autograd mechanism, which is fundamental for automatic differentiation in neural network training and other machine learning tasks. Engaging in PyTorch projects through ProjectPro offers a unique opportunity to gain practical experience and insights into employing PyTorch effectively for real-world data science and machine learning solutions. These end-to-end solved projects provide hands-on experience and problem-solving skills crucial for building robust data science and machine learning solutions.

What Users are saying..

profile image

Ameeruddin Mohammed

ETL (Abintio) developer at IBM
linkedin profile url

I come from a background in Marketing and Analytics and when I developed an interest in Machine Learning algorithms, I did multiple in-class courses from reputed institutions though I got good... Read More

Relevant Projects

NLP Project for Beginners on Text Processing and Classification
This Project Explains the Basic Text Preprocessing and How to Build a Classification Model in Python

PyTorch Project to Build a GAN Model on MNIST Dataset
In this deep learning project, you will learn how to build a GAN Model on MNIST Dataset for generating new images of handwritten digits.

AWS MLOps Project to Deploy Multiple Linear Regression Model
Build and Deploy a Multiple Linear Regression Model in Python on AWS

Time Series Forecasting Project-Building ARIMA Model in Python
Build a time series ARIMA model in Python to forecast the use of arrival rate density to support staffing decisions at call centres.

Learn to Build a Neural network from Scratch using NumPy
In this deep learning project, you will learn to build a neural network from scratch using NumPy

Deploy Transformer-BART Model on Paperspace Cloud
In this MLOps Project you will learn how to deploy a Tranaformer BART Model for Abstractive Text Summarization on Paperspace Private Cloud

MLOps Project for a Mask R-CNN on GCP using uWSGI Flask
MLOps on GCP - Solved end-to-end MLOps Project to deploy a Mask RCNN Model for Image Segmentation as a Web Application using uWSGI Flask, Docker, and TensorFlow.

Deep Learning Project for Text Detection in Images using Python
CV2 Text Detection Code for Images using Python -Build a CRNN deep learning model to predict the single-line text in a given image.

Loan Eligibility Prediction using Gradient Boosting Classifier
This data science in python project predicts if a loan should be given to an applicant or not. We predict if the customer is eligible for loan based on several factors like credit score and past history.

Build a Graph Based Recommendation System in Python -Part 1
Python Recommender Systems Project - Learn to build a graph based recommendation system in eCommerce to recommend products.