Dense, Conv & Utility Layers

1D Adaptive Average Pooling.

Produces fixed output length regardless of input size. Output cell o averages the input range [floor(o * L / out), ceil((o + 1) * L / out)). Input: (N, C, L) -> Output: (N, C, outputSize)

2D Adaptive Average Pooling.

Produces output of specified size regardless of input dimensions. Output cell o averages the input range [floor(o * in / out), ceil((o + 1) * in / out)), so neighbouring windows overlap when in is not a multiple of out. Typical use is a global average pool ([1, 1]) before a classifier head, so that the head does not depend on the input resolution.

1D Adaptive Max Pooling.

Produces fixed output length regardless of input size, taking the maximum of each adaptive window (the windows are the same as in AdaptiveAvgPool1d). Input: (N, C, L) -> Output: (N, C, outputSize)

2D Adaptive Max Pooling.

Produces output of specified size regardless of input dimensions, selecting the maximum value from each adaptive window (the windows are the same as in AdaptiveAvgPool2d).

1D Average Pooling.

Applies average pooling over a 1D signal. Input: (N, C, L) -> Output: (N, C, L_out)

By default padded zeros count towards the average (PyTorch's count_include_pad=True); pass countIncludePad: false to divide by the number of real elements in each window. The stride defaults to the kernel size and padding may be at most half of kernelSize.

2D Average Pooling Layer.

Applies a 2D average pooling over an input of shape (batch, channels, height, width). By default padded zeros count towards the average (PyTorch's count_include_pad=True); pass countIncludePad: false to divide by the number of real elements in each window.

The stride defaults to the kernel size. As in PyTorch, padding may be at most half of kernelSize.

3D Average Pooling Layer.

Applies average pooling over a 3D input signal (volumetric data). Input: (N, C, D, H, W) -> Output: (N, C, D_out, H_out, W_out)

By default padded zeros count towards the average (PyTorch's count_include_pad=True); pass countIncludePad: false to divide by the number of real elements in each window. The stride defaults to the kernel size and padding may be at most half of kernelSize.

1D Convolutional Layer.

Applies a 1D cross-correlation over an input of shape (batch, inChannels, length) and returns (batch, outChannels, outLength) with outLength = floor((length + 2 * padding - dilation * (kernelSize - 1) - 1) / stride) + 1.

Weights have shape (outChannels, inChannels / groups, kernelSize). As in PyTorch, weights and bias are drawn from U(-1/sqrt(fanIn), 1/sqrt(fanIn)) with fanIn = inChannels / groups * kernelSize.

dilation spaces out the kernel taps, groups splits the channels into independent groups (groups = inChannels gives a depthwise convolution), and padding accepts "valid" (no padding) or "same" (output length equals input length; stride 1 only).

The layer computes in its parameter dtype (float32 unless dtype or the global default dtype says otherwise) and casts the input to it. A GradTensor input gives a GradTensor; a plain Tensor input gives a GradTensor that tracks the weights while they require grad and gradient tracking is on, and a plain Tensor otherwise (inside noGrad() or with frozen weights).

2D Convolutional Layer.

Applies a 2D cross-correlation over an input of shape (batch, inChannels, height, width) and returns (batch, outChannels, outH, outW) with outH = floor((height + 2 * padH - dilH * (kH - 1) - 1) / strideH) + 1 (and likewise for the width).

Weights have shape (outChannels, inChannels / groups, kH, kW). As in PyTorch, weights and bias are drawn from U(-1/sqrt(fanIn), 1/sqrt(fanIn)) with fanIn = inChannels / groups * kH * kW.

dilation spaces out the kernel taps, groups splits the channels into independent groups (groups = inChannels gives a depthwise convolution), and padding accepts "valid" (no padding) or "same" (output size equals input size; stride 1 only).

The layer computes in its parameter dtype (float32 unless dtype or the global default dtype says otherwise) and casts the input to it. A GradTensor input gives a GradTensor; a plain Tensor input gives a GradTensor that tracks the weights while they require grad and gradient tracking is on, and a plain Tensor otherwise (inside noGrad() or with frozen weights).

3D Convolutional Layer.

Applies a 3D cross-correlation over an input signal composed of several input planes. Used for video processing, medical imaging (CT/MRI), and 3D point clouds.

Input: (N, C_in, D, H, W) -> Output: (N, C_out, D_out, H_out, W_out), where each output size is floor((in + 2 * padding - dilation * (kernel - 1) - 1) / stride) + 1.

Weights have shape (outChannels, inChannels / groups, kD, kH, kW). As in PyTorch, weights and bias are drawn from U(-1/sqrt(fanIn), 1/sqrt(fanIn)) with fanIn = inChannels / groups * kD * kH * kW.

dilation spaces out the kernel taps, groups splits the channels into independent groups, and padding accepts "valid" (no padding) or "same" (output size equals input size; stride 1 only).

The layer computes in its parameter dtype and casts the input to it. A GradTensor input gives a GradTensor; a plain Tensor input gives a GradTensor while the weights require grad and gradient tracking is on, and a plain Tensor otherwise.

1D Transposed Convolution Layer (Deconvolution).

Applies a transposed 1D convolution over an input signal. Used for upsampling in sequence generation, audio synthesis, etc.

Output size: outL = (inL - 1) * stride - 2 * padding + kernelSize + outputPadding

Weights have shape (inChannels, outChannels, kernelSize). outputPadding must be smaller than the stride. As in PyTorch, weights and bias are drawn from U(-1/sqrt(fanIn), 1/sqrt(fanIn)) with fanIn = outChannels * kernelSize.

The layer computes in its parameter dtype and casts the input to it. A GradTensor input gives a GradTensor; a plain Tensor input gives a GradTensor while the weights require grad and gradient tracking is on, and a plain Tensor otherwise.

2D Transposed Convolution Layer (Deconvolution).

Applies a transposed 2D convolution operator over an input image. Used in image generation (GANs), semantic segmentation (U-Net), autoencoders, and super-resolution networks.

Output size:

outH = (inH - 1) * strideH - 2 * padH + kernelH + outputPadHoutW = (inW - 1) * strideW - 2 * padW + kernelW + outputPadW

Weights have shape (inChannels, outChannels, kH, kW). outputPadding must be smaller than the stride; it only resolves the output-size ambiguity of strided convolutions and adds no values of its own. As in PyTorch, weights and bias are drawn from U(-1/sqrt(fanIn), 1/sqrt(fanIn)) with fanIn = outChannels * kH * kW.

The layer computes in its parameter dtype and casts the input to it. A GradTensor input gives a GradTensor; a plain Tensor input gives a GradTensor while the weights require grad and gradient tracking is on, and a plain Tensor otherwise.

1D Max Pooling.

Applies max pooling over a 1D signal (e.g., sequences, audio). Input: (N, C, L) -> Output: (N, C, L_out)

The stride defaults to the kernel size. Padding is ignored by the maximum, a NaN inside a window propagates, and padding may be at most half of kernelSize.

2D Max Pooling Layer.

Applies a 2D max pooling over an input of shape (batch, channels, height, width). Padding is ignored by the maximum (it behaves like -Infinity), a NaN inside a window propagates to the output, and the gradient flows to the first maximum of each window.

The stride defaults to the kernel size. As in PyTorch, padding may be at most half of kernelSize.

3D Max Pooling Layer.

Applies max pooling over a 3D input signal (volumetric data). Input: (N, C, D, H, W) -> Output: (N, C, D_out, H_out, W_out)

The stride defaults to the kernel size. Padding is ignored by the maximum, a NaN inside a window propagates, and padding may be at most half of kernelSize.

A lookup table that stores embeddings of a fixed dictionary and size.

This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.

Mathematical Formulation:

output[i] = weight[indices[i]]

The weight table is initialized from a standard normal distribution. With paddingIdx, that row is initialized to zeros and receives no gradient. As in PyTorch, the lookup returns the stored row, which stays zero unless you write to it.

The output has the dtype of the weight table. A GradTensor index gives a GradTensor; a plain index tensor gives a GradTensor that tracks the weight while it requires grad and gradient tracking is on, and a plain Tensor otherwise (for example inside noGrad() or for a frozen table).

Computes sums, means, or maxes of "bags" of embeddings, without instantiating the intermediate per-embedding matrix.

Indices can be given as a 1-D tensor together with offsets that mark where each bag starts, or as a 2-D tensor (numBags, bagSize) without offsets. Entries equal to paddingIdx are left out of the reduction (and receive no gradient). An empty bag, or a bag made only of padding, gives a row of zeros.

Applies a linear transformation to the incoming data: y = xA^T + b

This is also known as a fully connected layer or dense layer.

Mathematical Formulation:

y = x * W^T + b

Where: - x is the input tensor of shape (*, in_features) - W is the weight matrix of shape (out_features, in_features) - b is the bias vector of shape (out_features,) - y is the output tensor of shape (*, out_features)

Shape Conventions: - Input: (*, in_features) where * means any number of leading dimensions - 1D: (in_features) → Output: (out_features) - 2D: (batch, in_features) → Output: (batch, out_features) - 3D: (batch, seq_len, in_features) → Output: (batch, seq_len, out_features) - The last dimension must equal in_features - All leading dimensions are preserved in the output

Parameters: - inFeatures: Size of each input sample - outFeatures: Size of each output sample - bias: If true, adds a learnable bias to the output

Attributes: - weight: Learnable weights of shape (out_features, in_features) - bias: Learnable bias of shape (out_features,) if bias=true

Initialization: As in PyTorch, weights and biases are drawn from the uniform distribution U(-1/sqrt(in_features), 1/sqrt(in_features)) (Kaiming uniform with a = sqrt(5) for the weights). Seed the global generator with manualSeed for reproducible weights.

Input dtype: The layer computes in its parameter dtype (float32 unless dtype is given, or the global default dtype), so integer, boolean and float64 inputs all work. For a GradTensor input the conversion is differentiable.

Gradient tracking: With a GradTensor input the result is a GradTensor. With a plain Tensor input the result is a GradTensor that tracks the weights while gradient tracking is on and a weight requires grad (the input itself is not tracked), so a training loop needs no wrapping of the data. Inside noGrad(), or when every weight is frozen, a plain Tensor is returned. eval() alone does not switch tracking off.

Pads the input tensor using a constant value.

The padding is [left, right, top, bottom], or a single number applied to all four sides. Negative values crop. For integer and boolean inputs the value is truncated to the input dtype.

Pads using reflection of the input boundary (the edge row or column is not repeated). Each pad must be smaller than the corresponding input size.

Pads using replication of the input boundary values.

Pads the input tensor using zeros.

The padding is [left, right, top, bottom], or a single number applied to all four sides. Negative values crop. Accepts (N, C, H, W) and unbatched (C, H, W) input.

Applies Spectral Normalization to a weight parameter of a module.

Spectral normalization constrains the spectral norm (largest singular value) of a weight matrix to 1, stabilizing the training of GANs and other networks.

Mathematical Formulation:

W_SN = W / σ(W)

where σ(W) is the largest singular value of W, estimated via power iteration.

Behavior: - The wrapped module is registered as the child module, so parameters() and stateDict() include its weights (for example module.weight). - The raw weight stays in the wrapped module and is never modified. During forward the module sees the normalized weight, and gradients flow back to the raw weight through the normalization, treating the power-iteration vectors as constants (as PyTorch does). - The power iteration vectors u and v are stored as buffers named <weightName>_u and <weightName>_v. They are refined by nPowerIterations steps on each training-mode forward pass and are left unchanged in evaluation mode. Fifteen warm-up iterations run at construction. - Weights with more than two dimensions are reshaped to (shape[0], rest).

Purpose: - Stabilizes GAN training by controlling the Lipschitz constant - Prevents mode collapse in generative models - More efficient than full SVD: uses power iteration (O(mn) per step)

Upsamples a 4D input (N, C, H, W) using nearest-neighbor or bilinear interpolation.

Specify either scaleFactor or size, not both. With scaleFactor, the output height and width are floor(input * scaleFactor). The output keeps the input's float dtype (integer and bool inputs produce float64) and is differentiable with respect to the input.

Bilinear mode follows alignCorners. The default is true (corner pixels of input and output coincide), which is the convention Deepbox has always used. PyTorch's nn.Upsample defaults to false; pass alignCorners: false for results that match it.

Flattens a contiguous range of dims into a single dim.

Negative startDim / endDim count from the last dimension. A 0-d input is flattened to shape [1], as in PyTorch.

Identity layer: returns its input unchanged (the same object, no copy).

Useful for skip connections and placeholder layers.

Unflattens a single dim into multiple dims.

At most one entry of unflattenedSize may be -1; it is inferred from the size of the dimension being split.