![Pros And Cons of ELU Activation function
Pros
1. ELU becomes smooth slowly until its
output equal to -α whereas RELU
sharply smoothes.
2. ELU is a strong alternative to ReLU.
3. Unlike to ReLU, ELU can produce
negative outputs.
Cons
1. For x > 0, it can blow up the activation
with the output range of [0, inf].](https://image.slidesharecdn.com/vkkajcdqeugo0amsntq9-signature-d6c9c09aa3528af14349e65e2ed4ff95984195ec14b5eb58eafc662914ccb582-poli-200520153327/85/Activation-functions-11-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Activation functions (20)
Ad
Recently uploaded (20)
Ad
Activation functions
- 1. Activation Functions DEEP LEARNING By Prateek Sahu
- 2. 1. SIGMOID ACTIVATION FUNCTION Sigmoidal functions are frequently used in machine learning, specifically in the testing of artificial neural networks, as a way of understanding the output of a node or “neuron.” A sigmoid function is a type of activation function, and more specifically defined as a squashing function. Squashing functions limit the output to a range between 0 and 1.
- 3. Pros And Cons of Sigmoid Activation function Pros 1. The performance of Binary Classification is very well as compare to other activation function. 2. Clear predictions, i.e very close to 1 or 0. Cons 1. The calculation in sigmoid function is complex. 2. It is not useful in multiclass classification . 3. For negative values of x-axis gives 0. 4. It become constant and gives 1 for any high positive values. 5. Function output is not zero-centered
- 4. Hypertangent activation Function This function is easily defined as the ratio between the hyperbolic sine and the cosine functions
- 5. Pros And Cons of Tanh Activation function Pros 1. The gradient is stronger for tanh than sigmoid ( derivatives are steeper). 2. The output interval of tanh is 1, and the whole function is 0-centric, which is better than sigmod Cons 1. Tanh also has the vanishing gradient problem.
- 6. ReLu Activation Function The ReLU function is actually a function that takes the maximum value
- 7. Pros And Cons of ReLu function Pros 1. When the input is positive, there is no gradient saturation problem. 2. The calculation speed is much faster. 3. The ReLU function has only a linear relationship. 4. Whether it is forward or backward, it is much faster than sigmod and tanh. Cons 1. When the input is negative, ReLU is completely inactive, which means that once a negative number is entered, ReLU will die 2. We find that the output of the ReLU function is either 0 or a positive number, which means that the ReLU function is not a 0-centric function.
- 8. Leaky ReLu Function It is an attempt to solve the dying ReLU problem The leak helps to increase the range of the ReLU function. Usually, the value of a is 0.01 or so.
- 9. Pros And Cons of Leaky ReLu Activation function Pros 1. There will be no problems with Dead ReLU. 2. A parameter-based method, Parametric ReLU : f(x)= max(alpha x,x), which alpha can be learned from back propagation. Cons 1. It has not been fully proved that Leaky ReLU is always better than ReLU.
- 10. ELU (Exponential Linear Units) function ELU is very similiar to RELU except negative inputs. They are both in identity function form for non- negative inputs. On the other hand, ELU becomes smooth slowly until its output equal to -α whereas RELU sharply smoothes.
- 11. Pros And Cons of ELU Activation function Pros 1. ELU becomes smooth slowly until its output equal to -α whereas RELU sharply smoothes. 2. ELU is a strong alternative to ReLU. 3. Unlike to ReLU, ELU can produce negative outputs. Cons 1. For x > 0, it can blow up the activation with the output range of [0, inf].
- 12. Softmax Function Softmax function calculates the probabilities distribution of the event over ‘n’ different events. In general way of saying, this function will calculate the probabilities of each target class over all possible target classes.
- 13. Pros And Cons of Softmax Activation function Pros 1. It mimics the one hot encoded labels better than the absolute values. 2. If we use the absolute (modulus) values we would lose information, while the exponential intrinsically takes care of this. Cons 1. The softmax function should not be used for multi-label classification. 2. the sigmoid function (discussed later) is preferred for multi-label classification. 3. The Softmax function should not be used for a regression task as well.
- 14. Swish Function Swish's design was inspired by the use of sigmoid functions for gating in LSTMs and highway networks. We use the same value for gating to simplify the gating mechanism, which is called self-gating.
- 15. Pros And Cons of Swish Activation function Pros 1. No dying ReLU. 2. Increase in accuracy over ReLU 3. Outperforms ReLU in every batch size. Cons 1. Slightly more computationally expensive. 2. More problems with the algorithm will probably arise given time.
- 16. Maxout Function The Maxout activation is a generalization of the ReLU and the leaky ReLU functions. It is a learnable activation function. The Maxout activation function is defined as follows
- 17. Pros And Cons of Maxout Activation function Pros 1. It is a learnable activation function. Cons 1. It doubles the total number of parameters for each neuron and hence, a higher total number of parameters need to be trained.
- 18. Softplus Activation Funtion The softplus function is similar to the ReLU function, but it is relatively smooth.It is unilateral suppression like ReLU.It has a wide acceptance range (0, + inf). Softplus function: f(x) = ln(1+exp x)
- 19. Pros And Cons of Softplus Activation function Pros 1. It is relatively smooth. 2. It is unilateral suppression like ReLU. 3. It has a wide acceptance range (0, + inf). Cons 1. Leaky ReLU is a piecewise linear function, just as for ReLU, so quick to compute. ELU has the advantage over softmax and ReLU that it's mean output is closer to zero, which improves learning.