Relationship between Sigmoid and Gaussing Distribution

Question

I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?":

The assumption of a dependent variable to follow a sigmoid function inherently assumes a Gaussian distribution for the independent variable which is a general distribution we see for a lot of randomly occurring events and this is a good generic distribution to start with.

Could someone elaborate on this relationship between the two?

Jeanba · Answer

Yes sure, the core idea is that the sigmoid function approximate very nicely the cumulative density function of a Gaussian distribution. Therefore, if the input of a neuron is a continuous Gaussian distribution, the probability of the output being inferior to a certain value will follow (almost) the shape of the sigmoid function.

A detailed explanation can be found p16 of the following article:

https://www.researchgate.net/publication/220233479_Catastrophic_Forgetting_and_the_Pseudorehearsal_Solution_in_Hopfield-type_Networks

Relationship between Sigmoid and Gaussing Distribution

One Answer

Add your own answers!

Ask a Question