Activation Functions & SwiGLU

Interactive exploration of ReLU, GELU, Swish, and the SwiGLU gating mechanism

📈 Function Comparison

Show:

ReLU

f(x) = max(0, x)

Simple but causes "dead neurons" — zero gradient for all x < 0

GELU

f(x) = x · Φ(x)

Smooth, probabilistic gating. Used in BERT and GPT-2

Swish / SiLU

f(x) = x · σ(x)

Smooth, non-monotonic. Used inside SwiGLU in all modern LLMs

⚡ Derivative Comparison

Show:

ReLU Dead Zone

0.000

Gradient for all x < 0
Once dead, a neuron never recovers

Swish at x = −2

−0.090

Non-zero gradient even for negative inputs
Neurons can always recover

Why this matters: In deep networks, if a ReLU neuron receives a large negative input (e.g., from a bad weight update), its gradient becomes exactly 0. With zero gradient, the weights can never update — the neuron is permanently "dead." Swish and GELU have small but non-zero gradients everywhere, so neurons can always recover.

🔀 SwiGLU Gating Mechanism

SwiGLU(x) = (Swish(xW₁) ⊙ xW₃) · W₂ — the gate controls which features pass through

Input x:

1.50

GATE Swish(x · W₁) — which features to open

⊙

VALUE x · W₃ — what information is available

↓

OUTPUT gate ⊙ value — gated result

Standard FFN (ReLU)

ReLU(x · W₁) — binary on/off, no gating

SwiGLU FFN

Swish(x · W₁) ⊙ x · W₃ — smooth, selective gating

Key insight: ReLU is a hard switch — features are either fully on or fully off. SwiGLU uses the Swish gate as a smooth dimmer: each hidden feature has an independent, learnable gate that controls how much of the corresponding value passes through. This gives the FFN much richer, more expressive control over information flow.