Here are some of the most famous activation functions used in neural networks, along with their advantages and disadvantages:

1. Sigmoid Function:

• Output: Ranges between 0 and 1 (squashes the input values between 0 and 1).
• Advantages:
  • Smooth output, making it suitable for modeling probabilities (often used in output layer for binary classification).
  • Well-behaved gradients for backpropagation (a technique used to train neural networks).
• Disadvantages:
  • Output saturates for large positive/negative inputs (vanishing gradients). This can slow down the training process.
  • Not zero-centered, which can affect the learning process in some cases.

2. Hyperbolic Tangent (tanh) Function:

• Output: Ranges between -1 and 1 (squashes the input values between -1 and 1).
• Advantages:
  • Zero-centered output, which can be helpful for some neural network architectures.
  • Well-behaved gradients for backpropagation.
• Disadvantages:
  • Similar saturation issues as sigmoid for large positive/negative inputs.

3. Rectified Linear Unit (ReLU):

• Output: Max(0, input) – Simply sets all negative inputs to zero and keeps positive inputs unchanged.
• Advantages:
  • Fast computation (no complex mathematical operations involved).
  • Avoids vanishing gradients: Because ReLU only allows positive values, it avoids the gradients vanishing to zero during backpropagation, which can speed up training.
• Disadvantages:
  • Dead neurons: ReLU neurons can become inactive (stuck at zero) if they receive constant negative inputs. This can limit the network’s ability to learn.

Choosing the Right Activation Function:

The best activation function for a specific task depends on various factors like the type of problem you’re trying to solve and the architecture of your neural network. Here’s a brief guideline:

• Use sigmoid or tanh for output layers where you want probabilities (e.g., binary classification).
• Use ReLU for hidden layers in most cases due to its computational efficiency and ability to avoid vanishing gradients.

References:

• https://www.analyticsvidhya.com/blog/2022/03/introductory-guide-on-the-activation-functions/ This article provides a good overview of activation functions and when to use them, including sigmoid, tanh, and ReLU.
• https://www.linkedin.com/pulse/demystifying-activation-functions-neural-networks-ravindra-bajpai-6oc7c This article offers a broader look at various activation functions beyond the three classics mentioned.