Previously: From Desmos to CaLU: My Activation Function Tinkering Notes (I) ran 45 controlled runs on MNIST and collected eleven activations plus CaLU. This post picks up from there.
At the end of the last post I said the next step was VGG on CIFAR-10. I never got around to it; co-op has kept me busy the past two weeks. In my spare moments I was tidying up old notes, and I figured I might as well rewrite those eleven activation formulas cleanly so they’d be easy to look up later. Halfway through, I realized I didn’t need to copy them one by one.
Turns out I didn’t need to: they’re all the same expression. Start with the most familiar one: ReLU is defined as
Expand the by cases, and it becomes times an on/off switch:
is the Heaviside step function .
The rest follow the same pattern:
That leading stays the same throughout. I call it the value path, and the term behind it the gate.
All eleven, written out:
| Activation | Gate | Floor | Notes |
|---|---|---|---|
| ReLU | Heaviside | ||
| Leaky ReLU | Heaviside | ||
| PReLU | Heaviside | (learnable) | |
| ELU | Heaviside | saturating | negative side is a scaled whose slope decays to 0; not really a linear leak |
| SELU | Heaviside | saturating | ELU times |
| SiLU | logistic | ||
| GELU | Gaussian CDF | ||
| CaLU | Cauchy CDF | ||
| Leaky CaLU | Cauchy CDF | ||
| RoCLU | log-transformed | ||
| Mish |
All the variation lives in the gate and its floor: the gate is a curve climbing from 0 to 1, be it Heaviside, logistic, Gaussian CDF, or Cauchy CDF; the floor is what remains when the gate is fully closed, and it decides how completely the gate shuts. Every row is the value path times a gate. In rows with a linear leak, even a fully closed gate still leaves .
One item didn’t make the table: softplus, , which doesn’t look like times a gate. It’s really the integral of a gate, since . The disappears in the integration, so I left it out; few people use it as a hidden-layer activation anyway.
Once I’d written out the table, I landed on a simple conclusion: designing a new activation function mostly means designing a new gate.
That conclusion only describes; it doesn’t explain. What gave me pause was two more basic questions: why the value path is always , when nobody gates or ; and why gates are always these few curves, Heaviside, logistic, Gaussian CDF, or Cauchy CDF, rather than any S-curve you like.
Why the value path can only be
The first question comes down to asymptotics. A usable hidden-layer activation has to asymptote to on the positive side and to a straight line with nonnegative slope on the negative side (slope 0 falls back to ReLU). The reason is familiar: magnitudes have to get through, and so do gradients. If saturates as , then “strong” and “stronger” come out as the same number, and the information is lost right there; meanwhile goes to zero, killing backprop at that point too.
For positive there’s only one line through the origin with slope 1: . There’s no alternative for the value path.
Look at tanh and sigmoid for the flip side. When used as activations, they have no value path; they are the squashing curve themselves. For the effective gain is , which tends to zero as : the stronger the signal, the harder it gets squashed. Vanishing gradients are just this fact in backprop language, not its cause.
Put that same sigmoid inside and the problem goes away. What it does hasn’t changed; it still climbs from 0 to 1, except it no longer has to carry the signal to the next layer. The same “boundedness” is a virtue in a gate and a defect in an activation. In the last post, sigmoid ranked last on its own and second inside SiLU: two sides of the same coin.
So far this only explains why the value path is locked to . What the gate actually shuts off, and what the network loses when it does, are still open.
Which column the gate shuts off
First, let’s get the FC layer straight. One linear layer computes , with and . This is often described as “mapping the input to another space.” More concretely: each row of is a direction, and is the component of the input along that direction. Since activations act coordinate-wise, gates switch one component at a time.
What happens algebraically when a gate shuts depends on the next layer. The next layer’s pre-activation is
where is the -th column of . When the -th gate closes fully (), the term drops out of the sum, which is the same as deleting the -th column of . So one neuron’s contribution to the whole network is the rank-one atom formed by the input direction and the output direction :
The gate decides whether that atom counts for this input. Selection and assembly come apart here: the atom itself is the feature, one input direction paired with one output direction; the gate picks, the sum assembles. Between layers the network just recombines the selected atoms.
The atoms are generally not orthogonal. Shutting one gate doesn’t cleanly remove one feature; the remaining atoms still leak a little in that direction, and how much leaks depends on their overlap with . That’s what sparse coding means by a low-coherence dictionary.
How the gates open determines how many regimes the network has. With , a shut gate outputs an exact zero, and each input turns on one set of gates, so the network picks one subnetwork for that input: it uses a submatrix of , and the mapping stays linear, just cut into many pieces. With , no column is ever deleted; the shut ones are just scaled by , the matrix stays full rank, and the condition number worsens. Both versions train; the difference is how much the network truly stops using those atoms.
This also changes how to read a neuron’s output. A linear layer returns a nonzero number for any input, so it has no “absent” state; once the gate shuts, absence can be represented for the first time. A neuron that speaks up only on some inputs carries information in its silence; one that never shuts up can’t signal “present” vs. “absent” in its output. Which direction it prefers is up to ; the gate decides when it sits out.
If all gates stay open, every atom contributes to the output, the layer collapses to a fixed linear map, and “this coordinate is a feature” breaks down, because every coordinate mixes all directions. Only a sparse code aligns coordinates with features. A big part of why ReLU ruled for so many years is that it shuts decisively: the output is genuinely sparse, and the next layer gets a list of who is present.
These gates are all cumulative distribution functions: Heaviside is a degenerate distribution concentrated at 0, and , , are the logistic, Gaussian, and Cauchy CDFs. So reads as an expectation: take independent of , keep when and drop it otherwise, and the expectation is exactly . One motivation given in the GELU paper was to multiply by a 0/1 mask drawn from . The gate’s output is this component’s cumulative probability (percentile) under its own distribution.
Choosing a gate amounts to a hypothesis about this coordinate: how big it usually is, and how heavy its tail is. Heaviside is a point mass with no tail, which is why ReLU is so harsh; logistic means exponential tails, Gaussian means Gaussian tails, Cauchy means polynomial tails. Read that way, the gate comparison figure from the last post is several distributional assumptions side by side: Cauchy has a heavy tail, large values are no surprise, so the gate crawls toward 1; Gaussian is steepest in the middle, so GELU is most sensitive to mid-sized activations.
This reading has its limits: each gate looks only at its own component, which amounts to assuming independence across components. Real features aren’t independent, and can’t express “shut B off when A is on.”
What the optimizer and low precision see
Everything so far started from the function itself. The same formula looks like two different things to the optimizer and to numeric formats.
First, the optimizer. As gradients flow back, the activation contributes a local gain . One layer is like that; stack layers and the gain along one path lands in . The bound is crude, but it spells out the depth-vs-leak tradeoff: with , once a path shuts, nothing flows back through it; with there is still a narrow road of width . Narrow, but not zero.
Then, scale. ReLU is positively homogeneous of degree one, for . Ignoring biases, scaling up a layer’s weights scales its output along with it and leaves the shape alone. SiLU and GELU don’t have this: blowing up the weights pushes further right on the gating curve and washes the nonlinearity out. With BN this effect is mostly held down, since each layer’s scale gets normalized back; without BN it stays active, and weight decay slowly reshapes the activation’s effective form. That last claim is reasoning only; I haven’t tested it.
The number-format side is more direct. Piecewise-linear, bounded operators can be written entirely in integer arithmetic, which is where ReLU6 and hardswish come from: the former clips the range to , the latter approximates SiLU with to skip exponentials and division. They caught on on mobile because the hardware can compute the whole operator exactly, which has little to do with whether the shape is nice.
SiLU has an awkward range. The negative side dips to about , and asymmetric quantization has to cover that stretch, which eats into the resolution on the positive side. To actually use it you usually need a zero point that covers the negative values.
The real trouble is the leak itself. lives happily in floating point, but in int8, if the shut-off value falls below half a quantization step, it rounds to 0. You think you’re running Leaky ReLU, but you’re really running ReLU; soft support degrades to hard support, the submatrix gets truly deleted, and the dead-unit risk comes back in full. fp16 has its own version: underflows to 0 around , so the gate on that side shuts completely as well.
So low precision isn’t just about accuracy loss; it rewrites the activation’s effective parameters. is continuous on paper, but not on hardware.
Who the gate looks at
Back to that limitation. In , the gate and the gated value are the same scalar, so the only information the gate can use is “how big is this component.” That is a magnitude criterion: keep the big ones, shut the small ones. It can’t express “let B through when A is present.”
Make a vector instead of a scalar and the restriction loosens. GLU does exactly this. GLU stands for Gated Linear Unit: two linear projections of the same input, one as the value, one through a sigmoid as the gate.
Here is a vector, and are two matrices, is elementwise multiplication, biases omitted. The value path is , the gate is . The gate on the -th component is decided by , which is a linear combination of all input components. The gate looks along another direction, not at its own component.
Once the gate can see other components, selection becomes conditional: whether one feature stays can depend on whether another feature is there.
Check it against the earlier template and nothing has changed: the value path is still linear, the gate is still logistic. SwiGLU and GeGLU only swap the gate for SiLU and GELU. Same family. Only one thing changed: the gate’s input went from its own component to another projection.
If the earlier reading holds, this is the remaining degree of freedom in gate design. A one-dimensional gate can only haggle over “how big counts as big,” and the curves can only differ so much; give the gate more to look at, and what it can express jumps by an order of magnitude. The question the gate asks changes too: from “is this component big” to “given that component, should this one stay.” My guess is that this is why the GLU family keeps winning in language models: what wins isn’t the gate’s shape, it’s what the gate gets to see. The price is an extra projection, costing extra parameters and compute; whether that trade pays off depends on the task.
Closing
The VGG experiment is still pending. But before that, I looked at the table again and the whole story boils down to one line: the value path is , the gate is some CDF, and the only thing you can change is who the gate looks at.
If this reading is right, “inventing a new activation” is mostly swapping a new gate curve into the same template, and swapping curves may matter less than swapping what the gate gets to see. The 45 runs in the last post never separated, staying within noise, and looking back that is probably why: I kept tuning the less important parameter.
