AI · Theory

Activation Functions in Neural Networks

Theory

118/215

Activation Functions in Neural Networks

Activation adds non-linearity. Without it, stacked layers collapse to one linear map. ReLU = max(0, x) is the workhorse. Sigmoid/tanh squash; they saturate (vanishing gradients). Softmax on the last layer for multi-class probabilities.

Viva — why activation + name ReLU + one old one (sigmoid).

Activation Functions in Neural Networks — output: -2 → 0 / 0 → 0 / 3 → 3. Without activation, deep layers collapse to linear.

Exam tip

Why no activation = pointless deep net.

Example

# ReLU
def relu(x):
    return x if x > 0 else 0
for v in [-2, 0, 3]:
    print(v, "→", relu(v))

Activation Functions in Neural Networks — output: -2 → 0 / 0 → 0 / 3 → 3. Without activation, deep layers collapse to linear.

Short notes

DefNon-linearity. ReLU common.
RuleSoftmax for classes.

AI · Theory

Activation Functions in Neural Networks

← All stacks

Theory

118/215

Activation Functions in Neural Networks

Viva — why activation + name ReLU + one old one (sigmoid).

Activation Functions in Neural Networks — output: -2 → 0 / 0 → 0 / 3 → 3. Without activation, deep layers collapse to linear.

Exam tip

Why no activation = pointless deep net.

Example

# ReLU
def relu(x):
    return x if x > 0 else 0
for v in [-2, 0, 3]:
    print(v, "→", relu(v))

Activation Functions in Neural Networks — output: -2 → 0 / 0 → 0 / 3 → 3. Without activation, deep layers collapse to linear.

Short notes

DefNon-linearity. ReLU common.
RuleSoftmax for classes.

Activation Functions in Neural Networks

Example

Short notes

Questions

Activation Functions in Neural Networks

Example

Short notes

Questions