Optimizers in Machine Learning

Optimizer is an algorithm or a function that is used to modify parameters such as weights and learning rate to improve the performance of model by reducing loss and improving the accuracy.

Different optimizers

1. Gradient Descent
2. Stochastic Gradient Descent
3. Stochastic Gradient Descent with Momentum
4. Mini Batch Gradient Descent
5. Adagrad
6. RMSProp
7. AdaDelta
8. Adam

Activation Functions

 Activation function defines the output of the node basis on input.

Types of Activation Functions

1) Step Function

If x >=0, y=1

If x<0, y=0

2) Sigmoid Function 

This function ranges from (0,1) and is defined as 

S(x) =  1/(1+e^(-x))

3) ReLU Function

Rectified linear unit, it will output the input directly if input is positive, else output will be zero.

y = max(0,x)

4) Leaky ReLU

Leaky Rectified linear unit, this activation function provides small slope for negative values instead of flat slope.

y = ax if x < 0

y  = x

5) ELU

Exponential Linear Unit

y = x if x x> 0

y = a((e^x)-1))

If x value is less than 0, output will be slightly less than 0.

Artificial Neural Network

  Artificial Neural Networks (ANN) are inspired by the human brain. ANN is made up of three layers: an input layer, a hidden layer or layers, and an output layer. These networks learn from training data and gradually improve their accuracy. Once they are trained, these networks become powerful tools to recognize patterns.
Some important terminology in ANN 

• Weights

• Bias

• Learning Rate

• Threshold

How does neural network work?
 
Once input is fed through the input layer, weights are assigned, and these weights help to understand the importance of each variable. The higher the weight, the greater the importance.
After weights are assigned, weights are multiplied with individual variables and summation is done if summation crosses a given threshold and the basis of activation function output is determined.
 
The input layer's output is routed through the hidden layer(s) and then to the output layer in a process known as "feed forward neural network."
A loss is calculated based on output and actual value. The objective is to minimize the loss value.