deepbox/ndarray

Autograd

Reverse-mode automatic differentiation with GradTensor, trainable leaves from parameter, noGrad and customOp.

Autograd
GradTensor

Overview

parameter(data) creates a leaf with requiresGrad: true. Use it for hand-written optimization problems. For network training you do not wrap data: model.forward(tensor) returns a GradTensor that tracks the weights.
backward() adds to .grad on every leaf, as in PyTorch, so clear gradients between steps with optimizer.zeroGrad(), module.zeroGrad() or x.zeroGrad(). Gradients of intermediate nodes are not kept. On a non-scalar result backward() seeds ones, and you can pass your own seed tensor.
noGrad(fn) runs a synchronous callback with recording off and returns its result. An async callback throws DeepboxError. model.eval() does not turn tracking off, so wrap inference in noGrad.
GradTensor has the same methods as Tensor. Operations that produce no gradient, such as argmax, comparisons and isnan, return a plain Tensor. A plain Tensor exposes requiresGrad (always false), grad (null) and a backward() that throws, so code that receives AnyTensor can call them without narrowing.
customOp attaches a hand-written backward rule to a precomputed result. The *Grad helpers (softmaxGrad, dropoutGrad, im2colGrad and others) are the differentiable building blocks that the nn layers use.

Types

GradTensorOptions

type

Options for creating a GradTensor.

readonly requiresGrad?: boolean
readonly dtype?: Exclude<DType, "string">

Classes

GradTensor

Tensor wrapper that records a computation graph for reverse-mode autodiff.

Gradients accumulate into leaf tensors across backward() calls until zeroGrad() is called; gradients of intermediate nodes are recomputed on every pass.

The method surface matches Tensor, so code reads the same with or without gradient tracking. Results that cannot carry a gradient (argmax, comparisons, argsort, any, all) are plain Tensors, as in PyTorch.

ExampleTypeScript

const w = parameter([1, 2]);const loss = w.mul(w).sum();loss.backward();console.log(loss.item()); // 5console.log(w.grad?.toArray()); // [2, 4]

Members (104)

constructor(data: number | number[] | number[][] | number[][][], options?: GradTensorOptions)
constructor(args: { readonly tensor: Tensor; readonly requiresGrad: boolean; readonly prev: readonly GradTensor[]; readonly backward: BackwardFn; })
readonly tensor: Tensor
requiresGrad: boolean
shape: Shape
Get the shape of the underlying tensor.
size: number
Get the total number of elements.
ndim: number
Get the number of dimensions.
dtype: DType
Get the data type of the underlying tensor.
device: Tensor["device"]
Get the device where the tensor resides.
isDeviceTensor: boolean
true when the tensor's storage lives in device memory (see Tensor.isDeviceTensor).
strides: readonly number[]
Get the memory strides of the underlying tensor.
offset: number
Get the offset into the underlying data buffer.
data: TypedArray
Get the underlying data buffer.
grad: Tensor | null
Get the accumulated gradient for this tensor.
T: GradTensor
Transpose with the axes reversed (like .T in NumPy and PyTorch).
static isGradTensor(value: unknown): value is GradTensor
Check if a value is a GradTensor (works across module boundaries).
static create(args: { readonly tensor: Tensor; readonly requiresGrad: boolean; readonly prev: readonly GradTensor[]; readonly backward: BackwardFn; }): GradTensor
Build a GradTensor from an existing tensor, graph parents and backward rule.
static fromTensor(t: Tensor, options?: GradTensorOptions): GradTensor
Wrap a tensor as a leaf GradTensor without copying its data.
static scalar(value: number, options?: GradTensorOptions): GradTensor
Create a 0-d GradTensor holding value (default dtype float32).
toArray(): ReturnType<Tensor["toArray"]>
Convert to a nested JS array (delegates to the underlying tensor).
at(...indices: number[]): ReturnType<Tensor["at"]>
Read a single element by multi-index (delegates to the underlying tensor).
astype(dtype: DType): GradTensor
Cast to another numeric dtype, differentiably.
setGrad(grad: Tensor): void
Replace the stored gradient with grad.
zeroGrad(): void
Reset the gradient to zeros (shape and dtype of the tensor).
detach(): GradTensor
Return a GradTensor that shares this tensor's data but is cut off from the graph (requiresGrad is false).
setRequiresGrad(value: boolean): void
Turn gradient tracking on or off for this tensor.
hasGrad(): boolean
Whether a gradient has been stored on this tensor.
backward(grad?: Tensor): void
Backpropagate gradients from this node through the recorded graph.
add(otherArg: number | Tensor | GradTensor): GradTensor
Elementwise sum with broadcasting.
sub(otherArg: number | Tensor | GradTensor): GradTensor
Elementwise difference with broadcasting.
mul(otherArg: number | Tensor | GradTensor): GradTensor
Elementwise product with broadcasting.
neg(): GradTensor
Elementwise negation.
sum(axisArg?: Axis | readonly Axis[], keepdims?: boolean): GradTensor
Sum over axis, or over all elements when axis is omitted.
div(otherArg: number | Tensor | GradTensor): GradTensor
Elementwise quotient with broadcasting.
pow(exponent: number | Tensor | GradTensor): GradTensor
Raise every element to a constant power.
sqrt(): GradTensor
Elementwise square root, pow(0.5).
matmul(otherArg: Tensor | GradTensor): GradTensor
Matrix product with numpy.matmul / torch.matmul semantics.
relu(): GradTensor
Rectified linear unit, max(x, 0); the gradient at 0 is 0.
sigmoid(): GradTensor
Logistic function 1 / (1 + exp(-x)).
square(): GradTensor
Elementwise square, pow(2).
exp(): GradTensor
Elementwise exponential.
log(): GradTensor
Elementwise natural logarithm.
tanh(): GradTensor
Elementwise hyperbolic tangent (keeps the input's float dtype).
slice(...args: SliceRange[]): GradTensor
Slice the tensor (see slice); the gradient is scattered back into a zero tensor of the original shape.
gather(indicesArg: Tensor | GradTensor, axis: Axis): GradTensor
Select entries along axis using a 1-D index tensor.
mean(axis?: Axis | readonly Axis[], keepdims?: boolean): GradTensor
Mean over axis, or over all elements when axis is omitted.
max(axis?: Axis | readonly Axis[], keepdims?: boolean): GradTensor
Maximum over axis, or over all elements when axis is omitted.
reshape(newShape: Shape): GradTensor
Reshape the GradTensor to a new shape without copying data.
flatten(): GradTensor
Flatten the GradTensor to a 1-dimensional array.
view(shape: Shape, strides?: readonly number[], offset?: number): GradTensor
Create a view of the GradTensor with a different shape.
transpose(axes?: readonly number[]): GradTensor
Permute the axes (reverse them when axes is omitted).
min(axis?: Axis | readonly Axis[], keepdims?: boolean): GradTensor
Minimum over axis, or over all elements when axis is omitted.
abs(): GradTensor
Elementwise absolute value; the gradient at 0 is 0.
clip(minVal?: number, maxVal?: number): GradTensor
Clamp to [minVal, maxVal]; either bound may be omitted.
leakyRelu(negativeSlope?: number): GradTensor
Leaky ReLU: x for x > 0, otherwise negativeSlope * x.
elu(alpha?: number): GradTensor
Exponential linear unit: x for x > 0, otherwise alpha * (exp(x) - 1).
gelu(options?: GeluApproximation | GeluOptions): GradTensor
Gaussian error linear unit.
hardtanh(minVal?: number, maxVal?: number): GradTensor
Clamp to [minVal, maxVal]; the gradient is 1 strictly inside the interval and 0 elsewhere.
tanhshrink(): GradTensor
x - tanh(x).
sin(): GradTensor
Elementwise sine.
cos(): GradTensor
Elementwise cosine.
tan(): GradTensor
Elementwise tangent.
log1p(): GradTensor
log(1 + x), accurate for small x.
expm1(): GradTensor
exp(x) - 1, accurate for small x.
softplus(): GradTensor
Softplus, log(1 + exp(x)), computed without overflow.
logSigmoid(): GradTensor
Log of the logistic function, -softplus(-x), stable for large |x|.
swish(): GradTensor
Swish (SiLU), x * sigmoid(x).
mish(): GradTensor
Mish, x * tanh(softplus(x)).
selu(): GradTensor
Scaled ELU, 1.0507... * elu(x, 1.6732...), the self-normalizing activation.
softmax(axis?: Axis): GradTensor
Softmax along axis (default last).
logSoftmax(axis?: Axis): GradTensor
Log-softmax along axis (default last).
cumsum(axis?: Axis): GradTensor
Cumulative sum along axis; the input is flattened first when axis is omitted (NumPy semantics).
prod(axis?: Axis | readonly Axis[], keepdims?: boolean): GradTensor
Product over axis, a list of axes, or all elements.
var(axis?: Axis | readonly Axis[], keepdims?: boolean, ddof?: number): GradTensor
Variance over axis, a list of axes, or all elements.
std(axis?: Axis | readonly Axis[], keepdims?: boolean, ddof?: number): GradTensor
Standard deviation over axis, a list of axes, or all elements.
maximum(other: number | Tensor | GradTensor): GradTensor
Elementwise maximum with broadcasting.
minimum(other: number | Tensor | GradTensor): GradTensor
Elementwise minimum with broadcasting.
static where(condition: GradTensor | Tensor, a: GradTensor | number, b: GradTensor | number): GradTensor
Select between two tensors by a condition, with broadcasting: condition ? a : b.
clone(): GradTensor
Differentiable copy: the result owns its own buffer and the gradient flows through unchanged (like torch.clone).
item(): number | bigint
Value of a single-element tensor as a JS number (or bigint for int64), like torch.Tensor.item().
squeeze(axis?: Axis | readonly Axis[]): GradTensor
Remove axes of size 1 (functional form: squeeze).
unsqueeze(axis: number): GradTensor
Insert an axis of size 1 (functional form: unsqueeze).
argmax(axis?: Axis, keepdims?: boolean): Tensor
Index of the maximum (functional form: argmax).
argmin(axis?: Axis, keepdims?: boolean): Tensor
Index of the minimum (functional form: argmin).
eq(other: number | Tensor | GradTensor): Tensor
Element-wise equality, giving a plain bool Tensor (functional form: equal).
ne(other: number | Tensor | GradTensor): Tensor
Element-wise inequality, giving a plain bool Tensor (functional form: notEqual).
gt(other: number | Tensor | GradTensor): Tensor
Element-wise greater-than, giving a plain bool Tensor (functional form: greater).
ge(other: number | Tensor | GradTensor): Tensor
Element-wise greater-or-equal, giving a plain bool Tensor (functional form: greaterEqual).
lt(other: number | Tensor | GradTensor): Tensor
Element-wise less-than, giving a plain bool Tensor (functional form: less).
le(other: number | Tensor | GradTensor): Tensor
Element-wise less-or-equal, giving a plain bool Tensor (functional form: lessEqual).
isnan(): Tensor
Element-wise NaN test, giving a plain bool Tensor (functional form: isnan).
any(axis?: Axis | readonly Axis[], keepdims?: boolean): Tensor
True where any element is non-zero, as a plain bool Tensor (functional form: any).
all(axis?: Axis | readonly Axis[], keepdims?: boolean): Tensor
True where all elements are non-zero, as a plain bool Tensor (functional form: all).
floor(): GradTensor
Round down (functional form: floor).
ceil(): GradTensor
Round up (functional form: ceil).
round(decimals?: number): GradTensor
Round half to even (functional form: round).
flip(axes?: Axis | readonly Axis[]): GradTensor
Reverse the order of elements along axes (functional form: flip).
sort(axis?: Axis | undefined, descending?: boolean): GradTensor
Sorted values along an axis (functional form: sort), like torch.sort(x).values.
argsort(axis?: Axis | undefined, descending?: boolean): Tensor
Indices that would sort the tensor along an axis (functional form: argsort).
dot(other: Tensor | GradTensor): GradTensor
Dot product, the same as GradTensor.matmul.
to(device: Device): Promise<GradTensor>
Move the tensor, and its gradient if it has one, to another device (see Tensor.to).
cpu(): Promise<GradTensor>
Move the tensor and its gradient to the CPU.
dispose(): void
Release the device memory of the tensor and of its gradient (see Tensor.dispose).
toString(maxElements?: number): string
Return a human-readable string representation of this GradTensor.

Functions

col2imGrad

function col2imGrad(cols: GradTensor, outputShape: readonly number[], kernelSize: [ number, number ], stride: [ number, number ], padding: [ number, number ]): GradTensor;

Column to Image operation for GradTensor (transpose/adjoint of im2col). The backward of col2im is im2col.

Parameters

cols GradTensor
outputShape readonly number[]
kernelSize [ number, number ]
stride [ number, number ]
padding [ number, number ]

Returns

GradTensor

concatGrad

function concatGrad(parts: readonly GradTensor[], axis?: number): GradTensor;

Concatenate GradTensors along an existing axis. Backward slices the gradient back to each input's segment.

Parameters

parts readonly GradTensor[]: Tensors to join; shapes must agree on every axis but axis.
axisoptional number: Axis to join along; negative values count from the end.

Returns

GradTensor

Throws

DeepboxErrorIf parts is empty.

customOp

function customOp(output: Tensor, grads: ReadonlyArray<readonly [ GradTensor, (outGrad: Tensor) => Tensor ]>): GradTensor;

Wrap a precomputed output tensor with an explicit reverse-mode rule.

Details

Layers that compute their forward value with hand-written kernels (Conv3d, ConvTranspose, pooling, Embedding, RNN cells, …) use this to attach a correct backward instead of returning a detached leaf. Each entry in grads is [input, gradFn]; gradFn(outGrad) returns that input's gradient tensor (already reduced to the input's shape). Honors the global no-grad context and only records the graph when some input requires grad.

Parameters

output Tensor
grads ReadonlyArray<readonly [ GradTensor, (outGrad: Tensor) => Tensor ]>

Returns

GradTensor

dropoutGrad

function dropoutGrad(input: GradTensor, p?: number, training?: boolean): GradTensor;

Inverted dropout: during training each element is zeroed with probability p and the survivors are scaled by 1 / (1 - p), so the expected value is unchanged. Returns input itself when training is false or p is 0. Integer and bool inputs are converted to float32 first, because the scale is not an integer in general.

Parameters

input GradTensor: Input tensor.
poptional number: Drop probability in [0, 1) (default 0.5).
trainingoptional boolean: Apply dropout only when true (default true).

Returns

GradTensor

Throws

InvalidParameterErrorIf p is outside [0, 1).

im2colGrad

function im2colGrad(input: GradTensor, kernelSize: [ number, number ], stride: [ number, number ], padding: [ number, number ]): GradTensor;

Differentiable im2col: unfold sliding kernel windows of an NCHW input into columns. The backward pass is col2imGrad's forward (col2im).

Parameters

input GradTensor
kernelSize [ number, number ]
stride [ number, number ]
padding [ number, number ]

Returns

GradTensor

logSoftmaxGrad

function logSoftmaxGrad(source: GradTensor, axis?: Axis): GradTensor;

Differentiable log-softmax along axis, computed as x - max - log(sum(exp(x - max))) (log-sum-exp form, stable for large inputs).

Host float32 and float64 tensors use a fused kernel (one sweep per lane, in float64, with the closed-form gradient g - exp(y) * sum(g)); other inputs use the composite formulation from differentiable primitives.

Parameters

source GradTensor: Logits. Integer and bool inputs are converted to float32.
axisoptional Axis: Axis to normalize over (default: last).

Returns

GradTensor

noGrad

function noGrad<T>(fn: () => T): T;

Run fn with gradient tracking disabled and return its result.

Details

Operations executed inside do not record a graph, so their results have requiresGrad false. Leaf tensors created inside (for example with parameter) keep the requiresGrad flag they were created with, as in PyTorch. The previous state is restored afterwards, also when fn throws.

Important: The callback must be synchronous. Passing an async function will cause gradEnabled to be restored before the awaited work finishes, silently breaking gradient tracking inside the async continuation.

Parameters

fn () => T

Returns

T

Throws

DeepboxErrorIf the callback returns a Promise or other thenable

parameter

function parameter(data: number | number[] | number[][] | number[][][] | Tensor, options?: GradTensorOptions): GradTensor;

Create a leaf GradTensor with requiresGrad = true. The flag also holds when the tensor is created inside noGrad (PyTorch semantics); only operations run inside noGrad skip recording.

Parameters

data number | number[] | number[][] | number[][][] | Tensor: Nested numbers or an existing tensor.
optionsoptional GradTensorOptions: Optional dtype used when data is not already a tensor.

Returns

GradTensor

softmaxGrad

function softmaxGrad(source: GradTensor, axis?: Axis): GradTensor;

Differentiable softmax along axis.

Host float32 and float64 tensors use a fused kernel (one sweep per lane, in float64, with the closed-form gradient). Other inputs, such as device tensors, are computed as exp(x - max) / sum(exp(x - max)) from differentiable primitives with the maximum treated as a constant (softmax is shift-invariant), so large inputs do not overflow either way.

Parameters

source GradTensor: Logits. Integer and bool inputs are converted to float32.
axisoptional Axis: Axis to normalize over (default: last).

Returns

GradTensor

stackGrad

function stackGrad(parts: readonly GradTensor[]): GradTensor;

Stack a list of same-shape GradTensors along a new leading axis. Backward splits the upstream gradient back to each input. Used by the recurrent layers to assemble per-timestep hidden states differentiably.

Parameters

parts readonly GradTensor[]: Tensors to stack; all must have the same shape. The result takes the dtype and device of the first one.

Returns

GradTensor

Throws

DeepboxErrorIf parts is empty.
ShapeErrorIf the shapes differ.

varianceGrad

function varianceGrad(input: GradTensor, axis?: Axis, correction?: number): GradTensor;

Differentiable variance.

Parameters

input GradTensor: Input tensor.
axisoptional Axis: Axis to reduce; when omitted, all elements are reduced.
correctionoptional number: Delta degrees of freedom: the divisor is n - correction (default 1, the unbiased estimator). The divisor is clamped at 0, so n <= correction yields NaN or Infinity instead of a negative variance.

Returns

GradTensor

ndarray-autograd.tsTypeScript

import { GradTensor, noGrad, parameter } from "deepbox/ndarray"; const x = parameter([[1, 2], [3, 4]]);const w = parameter([[0.5], [0.25]]); const y = x.matmul(w).sum();y.backward();console.log(x.grad?.toString());console.log(w.grad?.toString()); w.zeroGrad(); // gradients accumulate until you clear themconst prediction = noGrad(() => x.matmul(w));console.log(prediction.requiresGrad); // falseconsole.log(x instanceof GradTensor);

Found a mistake? Open an issue.