Activation Layers

Applies the CELU activation (continuously differentiable ELU).

CELU(x) = max(0, x) + min(0, alpha * (exp(x / alpha) - 1))

Applies the Exponential Linear Unit (ELU) activation.

ELU(x) = x if x > 0, else alpha * (exp(x) - 1)

Applies the Gaussian Error Linear Unit (GELU) activation.

GELU(x) = x * Phi(x) where Phi is the CDF of the standard normal distribution.

With approximate: "tanh" (the default) the value is computed with 0.5 x (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))), which matches PyTorch's GELU(approximate="tanh"). With approximate: "none" it is the exact erf form 0.5 x (1 + erf(x / sqrt(2))), which is PyTorch's default nn.GELU(). The two differ by up to about 5e-4.

Unlike PyTorch, the default here is "tanh". Pass { approximate: "none" } to get the values of torch.nn.GELU().

GLU (Gated Linear Unit) activation.

GLU(a, b) = a ⊗ σ(b) where [a, b] = split(input, dim)

Splits the input tensor in half along dim and applies sigmoid gating. Used in modern NLP architectures.

Applies the Hardshrink activation.

Hardshrink(x) = x if |x| > lambd, otherwise 0.

Hardsigmoid activation: piecewise linear approximation of sigmoid.

Hardsigmoid(x) = clamp(x/6 + 0.5, 0, 1)

Efficient for mobile/embedded (MobileNetV3).

Hardswish activation: piecewise linear approximation of swish.

Hardswish(x) = x * Hardsigmoid(x) = x * clamp(x/6 + 0.5, 0, 1)

Efficient for mobile/embedded (MobileNetV3).

Applies the HardTanh activation function.

HardTanh(x) = max(minVal, min(maxVal, x))

Applies the Leaky Rectified Linear Unit (Leaky ReLU) activation.

LeakyReLU(x) = x if x > 0, else alpha * x

Applies the LogSigmoid activation: log(1 / (1 + exp(-x))).

Computed as min(x, 0) - log1p(exp(-|x|)), so it neither overflows for very negative inputs nor loses precision for large positive ones.

Applies the Log Softmax activation function.

LogSoftmax(x_i) = log(exp(x_i) / sum(exp(x_j)))

Applies the Mish activation function.

Mish(x) = x * tanh(softplus(x))

PReLU (Parametric ReLU) activation with learnable slope parameter.

PReLU(x) = max(0, x) + a * min(0, x)

Where a is learnable. With numParameters = 1 a single slope is shared by all elements. With numParameters = C there is one slope per channel, taken along axis 1 of the input (axis 0 for 1-D input), as in PyTorch.

The slope is a parameter named weight in the layer dtype (float32 unless the dtype option or the global default dtype says otherwise). The layer computes in that dtype and casts the input to it (differentiably). A GradTensor input gives a GradTensor; a plain Tensor input gives a GradTensor while the slope requires grad and gradient tracking is on, and a plain Tensor otherwise.

Applies the Rectified Linear Unit (ReLU) activation function element-wise.

ReLU(x) = max(0, x)

Applies the ReLU6 activation: min(max(0, x), 6).

Used by MobileNet. NaN propagates. The gradient is 1 strictly between 0 and 6 and 0 elsewhere, including at the two kinks (as in PyTorch).

SELU (Scaled Exponential Linear Unit) activation for self-normalizing networks.

SELU(x) = scale * (max(0, x) + min(0, alpha * (exp(x) - 1)))

With specific alpha and scale values that enable self-normalization.

Applies the Sigmoid activation function element-wise.

Sigmoid(x) = 1 / (1 + exp(-x))

SiLU (Sigmoid Linear Unit) activation, an alias for Swish.

SiLU(x) = x * sigmoid(x)

Standard name in PyTorch. Identical to Swish.

Applies the Softmax activation function.

Softmax(x_i) = exp(x_i) / sum(exp(x_j))

Applies softmax over the channel dimension at every spatial location.

Accepts (C, H, W) or (N, C, H, W) input (as PyTorch's Softmax2d) and normalizes along the channel axis (ndim - 3), so the values at each pixel sum to 1 across channels.

Applies the Softmin activation function.

Softmin(x_i) = softmax(-x_i)

Applies the Softplus activation function.

Softplus(x) = log(1 + exp(beta * x)) / beta

A smooth approximation of ReLU. It is evaluated without overflow for large inputs, in both the tensor and the autograd path.

Applies the Softshrink activation.

Softshrink(x) = x - lambd if x > lambd, x + lambd if x < -lambd, otherwise 0.

Softsign activation function.

Softsign(x) = x / (1 + |x|)

Applies the Swish activation function (also known as SiLU).

Swish(x) = x * sigmoid(x)

Applies the Hyperbolic Tangent (Tanh) activation function element-wise.

Tanh(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x))

Applies the Tanhshrink activation function element-wise.

Tanhshrink(x) = x - tanh(x)

Replaces every element that is not above a threshold with a fixed value.

Threshold(x) = x if x > threshold, otherwise value. NaN passes through unchanged. The gradient is 1 where x > threshold and 0 elsewhere.