what are the different types of activation function in neural network

This recipe explains what are the different types of activation function in neural network

Recipe Objective - What are the different types of activation function?

Activation functions gives the output of the neural network in between 0 to 1 or -1 to 1 that is depending upon the function used. Linear activation function and Non-linear activation functions are the two types of activation functions. Linear activation function is linear in shape and the output of function is not confined between any range. Non-linear activation function are the mostly and widely used activation functions which consists of Sigmoid activation function, Tanh activation function, ReLU(Rectified Linear Unit) activation function, Leaky ReLU etc.

A Deep Dive into the Types of Neural Networks

Explanation of Activation Functions.

Sigmoid or Logistic activation function is the widely used activation function. It exists between 0 and 1. Sigmoid activation function is differentiable that is slope of sigmoid curve at any two points is available. The function is monotonic and derivative of sigmoid activation function is not monotonic.

Tanh or hyperbolic tangent activation function ranges between -1 to 1 and is S-shape sigmoid function. Tanh activation function is differentiable. The function is monotonic also while derivative of the function is not monotonic.

ReLU activation function is rectified from the bottom and is widely used in buiding convolutional neural networks and deep learning models. ReLU activation function is differentiable. Both the ReLU activation function and it's derivative is monotonic.

Leaky ReLU activation function attempts to solve leaky problem in ReLU activation function. The range of leaky relu lies between -infinity to infinity. Leaky ReLU activation function is differentiable.Both the Leaky ReLU function and it's derivative is monotonic.

What Users are saying..

profile image

Savvy Sahai

Data Science Intern, Capgemini
linkedin profile url

As a student looking to break into the field of data engineering and data science, one can get really confused as to which path to take. Very few ways to do it are Google, YouTube, etc. I was one of... Read More

Relevant Projects

AWS MLOps Project for ARCH and GARCH Time Series Models
Build and deploy ARCH and GARCH time series forecasting models in Python on AWS .

Build CI/CD Pipeline for Machine Learning Projects using Jenkins
In this project, you will learn how to create a CI/CD pipeline for a search engine application using Jenkins.

Recommender System Machine Learning Project for Beginners-2
Recommender System Machine Learning Project for Beginners Part 2- Learn how to build a recommender system for market basket analysis using association rule mining.

Tensorflow Transfer Learning Model for Image Classification
Image Classification Project - Build an Image Classification Model on a Dataset of T-Shirt Images for Binary Classification

LLM Project to Build and Fine Tune a Large Language Model
In this LLM project for beginners, you will learn to build a knowledge-grounded chatbot using LLM's and learn how to fine tune it.

Digit Recognition using CNN for MNIST Dataset in Python
In this deep learning project, you will build a convolutional neural network using MNIST dataset for handwritten digit recognition.

Locality Sensitive Hashing Python Code for Look-Alike Modelling
In this deep learning project, you will find similar images (lookalikes) using deep learning and locality sensitive hashing to find customers who are most likely to click on an ad.

Learn to Build a Siamese Neural Network for Image Similarity
In this Deep Learning Project, you will learn how to build a siamese neural network with Keras and Tensorflow for Image Similarity.

Machine Learning project for Retail Price Optimization
In this machine learning pricing project, we implement a retail price optimization algorithm using regression trees. This is one of the first steps to building a dynamic pricing model.

House Price Prediction Project using Machine Learning in Python
Use the Zillow Zestimate Dataset to build a machine learning model for house price prediction.