Activation Functions

Linear Activation Function

Exhibit a linear relationship between input and output, often used in regression tasks. The output is simply the weighted sum of inputs plus bias.

General Formula

f(x) = wx + b

m: Slope, reprents the rate of change of the output with respect to the input.
b: Intercept, the output when the input is zero.

Non-Linear Activation Functions

Non-linear activation functions introduce non-linearity into the model, allowing it to learn complex patterns. and Don’t follow a straight-line pattern.

Common examples:

Polynomial functions
Exponential functions
Logarithmic functions
Trigonometric functions

Sigmoid: Maps input values to a range between 0 and 1, useful for binary classification.
ReLU (Rectified Linear Unit): Outputs the input directly if positive; otherwise, it outputs zero. It helps mitigate the vanishing gradient problem.
Tanh: Maps input values to a range between -1 and 1, useful for zero-centered data.
Softmax: Converts a vector of values into probabilities, often used in multi-class classification tasks.