Skip to content

deepbox/nn

Normalization & Regularization

Batch, layer, group, instance, RMS and local response normalization, and the dropout layers.

  • Normalization

Overview

  • Normalization layers take eps (default 1e-5). The batch and instance variants also take momentum (0.1), affine and trackRunningStats. Batch normalization keeps running statistics in training mode and uses them after model.eval(). InstanceNorm defaults to affine: true, where PyTorch uses false. RMSNorm defaults to eps: 1e-5, where PyTorch uses the machine epsilon of the dtype.
  • LayerNorm(normalizedShape) takes the sizes of the trailing dimensions, as in new LayerNorm(32) or new LayerNorm([32]). Without elementwiseAffine it has no parameters and returns a plain Tensor for plain input.
  • Dropout layers are active by default, because modules start in training mode. Call model.eval() for inference. Dropout defaults to p = 0.5 and p must be in [0, 1). Surviving values are scaled by 1 / (1 - p). Dropout draws from the global generator, so setSeed makes a run repeatable.

Classes

14

AlphaDropout

extends Module

Applies Alpha Dropout during training, designed for SELU-activated networks.

Unlike standard Dropout which zeros elements, AlphaDropout replaces dropped elements with the negative saturation value of SELU, then applies an affine transformation to maintain self-normalizing properties.

Mathematical Formulation: During training: 1. Generate binary mask with keep probability (1 - p) 2. Replace dropped values with α' = -λα ≈ -1.7580993408 3. Apply affine transform: y = a * (x * mask + α' * (1 - mask)) + b where a = 1 / sqrt((1 - p) * (1 + p * α'²)) and b = -a * α' * p, which keep mean 0 and variance 1 for standard-normal inputs

During evaluation:

y = x

Purpose: - Maintains the self-normalizing property of SELU networks - Keeps mean ≈ 0 and variance ≈ 1 after dropout

Example
import { AlphaDropout, SELU, Linear, Sequential } from 'deepbox/nn'; const model = new Sequential([  new Linear(784, 256),  new SELU(),  new AlphaDropout(0.1),  new Linear(256, 10),]);
Members (6)
  • constructor(p?: number)

    Create a new AlphaDropout layer.

  • dropoutRate: number
  • forward(input: GradTensor): GradTensor

    Forward pass: apply alpha dropout during training, identity during evaluation.

  • forward(input: Tensor): Tensor
  • forward(input: AnyTensor): AnyTensor
  • toString(): string

Dropout

extends Module

Applies Dropout regularization during training.

Mathematical Formulation: During training:

y = x * mask / (1 - p)

where mask is a binary tensor with probability (1-p) of being 1.

During evaluation:

y = x

Purpose: - Prevents overfitting by randomly zeroing elements during training - Forces network to learn redundant representations - Improves generalization performance

Scaling: The output is scaled by 1/(1-p) during training to maintain expected value. This is called "inverted dropout" and eliminates the need for scaling during inference.

Example
import { Dropout } from 'deepbox/nn';import { tensor } from 'deepbox/ndarray'; const dropout = new Dropout(0.5); // Drop 50% of neuronsconst input = tensor([[1, 2, 3, 4]]); // Training mode: randomly zeros ~50% of elementsdropout.train();const output = dropout.forward(input); // Evaluation mode: passes input unchangeddropout.eval();const output2 = dropout.forward(input); // Same as input
Members (6)
  • constructor(p?: number)

    Create a new Dropout layer.

  • dropoutRate: number

    Get the dropout probability.

  • forward(input: GradTensor): GradTensor

    Forward pass: apply dropout during training, identity during evaluation.

  • forward(input: Tensor): Tensor
  • forward(input: AnyTensor): AnyTensor
  • toString(): string

    Get string representation of the layer.

Dropout2d

extends Module

Applies 2D channel-wise Dropout during training.

Randomly zeros entire channels (feature maps) of the input. Input is expected to be 4-D: (N, C, H, W), or 3-D (C, H, W) for a single sample. Each channel is either entirely kept or entirely zeroed, and kept channels are scaled by 1 / (1 - p). In evaluation mode (or with p = 0) the input is returned as is.

Example
const dropout2d = new Dropout2d(0.5);const input = tensor([[[[1,2],[3,4]], [[5,6],[7,8]]]]);dropout2d.train();const output = dropout2d.forward(input);
Members (6)
  • constructor(p?: number)
  • dropoutRate: number
  • forward(input: GradTensor): GradTensor

    Forward pass: zero whole channels during training, identity during evaluation.

  • forward(input: Tensor): Tensor
  • forward(input: AnyTensor): AnyTensor
  • toString(): string

BatchNorm1d

extends _BatchNorm

Batch Normalization layer.

Normalizes the input over the batch dimension for faster and more stable training.

Formula: y = (x - E[x]) / sqrt(Var[x] + eps) * gamma + beta

During training, uses batch statistics (biased variance for the normalization, unbiased variance for the running estimate). During evaluation, uses running statistics unless trackRunningStats=false, in which case batch statistics are always used. Batch statistics need more than one value per channel, otherwise an InvalidParameterError is thrown.

Accepts (N, C) or (N, C, L) input.

Example
import { BatchNorm1d } from 'deepbox/nn';import { tensor } from 'deepbox/ndarray'; const bn = new BatchNorm1d(3);const x = tensor([[1, 2, 3], [4, 5, 6]]);const y = bn.forward(x);
Members (1)
  • constructor(numFeatures: number, options?: { readonly eps?: number; readonly momentum?: number; readonly affine?: boolean; readonly trackRunningStats?: boolean; readonly dtype?: FloatDType; })

BatchNorm2d

extends _BatchNorm

2D Batch Normalization.

Normalizes over (N, H, W) dimensions for each channel, designed for 4D input tensors (batch, channels, height, width).

Example
const bn2d = new BatchNorm2d(64);// input: (N, 64, H, W) -> normalized output
Members (1)
  • constructor(numFeatures: number, options?: { readonly eps?: number; readonly momentum?: number; readonly affine?: boolean; readonly trackRunningStats?: boolean; readonly dtype?: FloatDType; })

BatchNorm3d

extends _BatchNorm

3D Batch Normalization.

Normalizes over (N, D1, D2, D3) dimensions for each channel, designed for 5D input tensors (batch, channels, depth, height, width).

Example
const bn3d = new BatchNorm3d(64);// input: (N, 64, D, H, W) -> normalized output
Members (1)
  • constructor(numFeatures: number, options?: { readonly eps?: number; readonly momentum?: number; readonly affine?: boolean; readonly trackRunningStats?: boolean; readonly dtype?: FloatDType; })

GroupNorm

extends Module

Group Normalization.

Divides channels into groups and normalizes within each group. Works well with small batch sizes where BatchNorm struggles.

Accepts (N, C, *) input with any number of trailing spatial dimensions.

Formula: y = (x - E[x]) / sqrt(Var[x] + eps) * gamma + beta

Example
const gn = new GroupNorm(32, 256); // 32 groups, 256 channels
Members (7)
  • constructor(numGroups: number, numChannels: number, options?: { readonly eps?: number; readonly affine?: boolean; readonly dtype?: FloatDType; })
  • weight: GradTensor | undefined

    Per-channel scale (gamma), or undefined when affine is false.

  • bias: GradTensor | undefined

    Per-channel shift (beta), or undefined when affine is false.

  • forward(x: GradTensor): GradTensor
  • forward(x: TensorClass): AnyTensor
  • forward(x: AnyTensor): AnyTensor
  • toString(): string

InstanceNorm

extends _InstanceNorm

Instance Normalization.

Normalizes each channel of each sample independently. Equivalent to GroupNorm with numGroups = numChannels. Used in style transfer and image generation.

Accepts (N, C, *) input with at least one spatial dimension that holds more than one element. Unlike PyTorch, affine defaults to true; pass { affine: false } to match torch.nn.InstanceNorm.

Example
const inorm = new InstanceNorm(64); // 64 channels
Members (1)
  • constructor(numFeatures: number, options?: { readonly eps?: number; readonly affine?: boolean; readonly dtype?: FloatDType; })

InstanceNorm1d

extends _InstanceNorm

1D Instance Normalization.

Normalizes each channel independently per sample for 3D inputs (N, C, L). An unbatched 2D input (C, L) is also accepted. Equivalent to GroupNorm where numGroups == numChannels. Unlike PyTorch, affine defaults to true; pass { affine: false } to match torch.nn.InstanceNorm1d.

Example
const inorm = new InstanceNorm1d(64);// input: (N, 64, L) -> normalized output
Members (1)
  • constructor(numFeatures: number, options?: { readonly eps?: number; readonly affine?: boolean; readonly dtype?: FloatDType; })

InstanceNorm2d

extends _InstanceNorm

2D Instance Normalization.

Normalizes each channel independently per sample for 4D inputs (N, C, H, W). An unbatched 3D input (C, H, W) is also accepted. Equivalent to GroupNorm where numGroups == numChannels. Unlike PyTorch, affine defaults to true; pass { affine: false } to match torch.nn.InstanceNorm2d.

Example
const inorm = new InstanceNorm2d(64);// input: (N, 64, H, W) -> normalized output
Members (1)
  • constructor(numFeatures: number, options?: { readonly eps?: number; readonly affine?: boolean; readonly dtype?: FloatDType; })

InstanceNorm3d

extends _InstanceNorm

3D Instance Normalization.

Normalizes each channel independently per sample for 5D inputs (N, C, D, H, W). An unbatched 4D input (C, D, H, W) is also accepted. Equivalent to GroupNorm where numGroups == numChannels. Unlike PyTorch, affine defaults to true; pass { affine: false } to match torch.nn.InstanceNorm3d.

Example
const inorm = new InstanceNorm3d(64);// input: (N, 64, D, H, W) -> normalized output
Members (1)
  • constructor(numFeatures: number, options?: { readonly eps?: number; readonly affine?: boolean; readonly dtype?: FloatDType; })

LayerNorm

extends Module

Layer Normalization.

Normalizes across the feature dimensions (trailing dimensions specified by normalizedShape) for each sample independently. Unlike BatchNorm, LayerNorm works the same way during training and evaluation.

Formula: y = (x - E[x]) / sqrt(Var[x] + eps) * gamma + beta

Example
import { LayerNorm } from 'deepbox/nn';import { tensor } from 'deepbox/ndarray'; const ln = new LayerNorm(3);const x = tensor([[1, 2, 3]]);const y = ln.forward(x);
Members (5)
  • constructor(normalizedShape: number | readonly number[], options?: { readonly eps?: number; readonly elementwiseAffine?: boolean; readonly bias?: boolean; readonly dtype?: FloatDType; })
  • forward(x: GradTensor): GradTensor
  • forward(x: TensorClass): AnyTensor
  • forward(x: AnyTensor): AnyTensor
  • toString(): string

LocalResponseNorm

extends Module

Local Response Normalization.

Applies local response normalization over an input signal, as described in the AlexNet paper. For each element, the normalization is computed across nearby channels.

Formula: out_i = x_i / (k + alpha/size * sum(x_j^2))^beta where the sum runs over size neighbouring channels: channel i itself, floor(size / 2) channels before it and floor((size - 1) / 2) after it. Channels outside the input count as zero, and the divisor is always size. This matches torch.nn.LocalResponseNorm, including for even size.

Example
const lrn = new LocalResponseNorm(5);// input: (N, C, ...) -> normalized output
Members (5)
  • constructor(size: number, options?: { readonly alpha?: number; readonly beta?: number; readonly k?: number; readonly dtype?: FloatDType; })
  • forward(x: GradTensor): GradTensor
  • forward(x: TensorClass): TensorClass
  • forward(x: AnyTensor): AnyTensor
  • toString(): string

RMSNorm

extends Module

Root Mean Square Layer Normalization (RMSNorm).

Normalizes the input by the RMS of the feature values, without centering. Used in LLaMA, Gemma, and other modern LLM architectures.

Formula: y = x / RMS(x) * gamma where RMS(x) = sqrt(mean(x^2) + eps)

The default eps is 1e-5; PyTorch's RMSNorm defaults to the machine epsilon of the input dtype, so pass eps explicitly when comparing outputs.

Example
const rms = new RMSNorm(512);
Members (5)
  • constructor(normalizedShape: number | readonly number[], options?: { readonly eps?: number; readonly elementwiseAffine?: boolean; readonly dtype?: FloatDType; })
  • forward(x: GradTensor): GradTensor
  • forward(x: TensorClass): AnyTensor
  • forward(x: AnyTensor): AnyTensor
  • toString(): string
nn-normalization.ts
import { tensor } from "deepbox/ndarray";import { setSeed } from "deepbox/random";import { BatchNorm1d, Dropout, LayerNorm, RMSNorm } from "deepbox/nn"; setSeed(1);const x = tensor([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 13]]); const batchNorm = new BatchNorm1d(4);console.log(batchNorm.forward(x).shape); // uses the batch statisticsbatchNorm.eval();console.log(batchNorm.forward(x).shape); // uses the running statistics console.log(new LayerNorm([4]).forward(x).shape);console.log(new RMSNorm(4).forward(x).shape); const dropout = new Dropout(0.5);console.log(dropout.forward(x).toString()); // some values zeroed, the rest doubleddropout.eval();console.log(dropout.forward(x).toString()); // identity in evaluation mode

Found a mistake? Open an issue.