Skip to content

deepbox/core

Utilities, Serialization & Parallelism

JSON serialization for tensors, module state and estimators, a chunking worker pool, and the validate* argument checks.

  • Serialization
  • Validation

Overview

  • toJSON and fromJSON handle three payload shapes: Tensor, ModuleState and Estimator. bigint, NaN, Infinity and -0 are stored as tagged objects, so the round trip is exact. Both functions validate the payload and throw DataValidationError when it is malformed.
  • save(path, payload) and load(path) are asynchronous and need Node.js, because they import node:fs on demand. They write and read the same JSON text.
  • WorkerPool splits an array into at most maxWorkers chunks and runs them in order on the calling thread. It does not start threads, so it organizes work but does not make it faster. taskTimeout is stored but not enforced, because a running task cannot be interrupted.
  • validateShape, validateDtype, validateInteger, validatePositive, validateRange and validateOneOf throw DataValidationError with the argument name in the message. normalizeAxis converts a negative axis and throws InvalidParameterError when it is out of range.

Types

7

PoolStatus

type

Status information about the worker pool.

  • readonly maxWorkers: number
  • readonly activeWorkers: number
  • readonly pendingTasks: number

    Always 0: chunks run as soon as they are scheduled.

  • readonly completedTasks: number
  • readonly isTerminated: boolean

TaskResult

type

Result of a parallel task execution.

  • readonly value: T
  • readonly workerId: number
  • readonly durationMs: number

WorkerPoolOptions

type

Options for creating a WorkerPool.

  • readonly maxWorkers?: number

    Maximum number of chunks work is split into.

  • readonly taskTimeout?: number

    Task timeout in milliseconds (positive).

SerializedEstimator

type

Serialized ML estimator: hyperparameters and fitted arrays.

  • readonly className: string
  • readonly params: Record<string, unknown>
  • readonly state: Record<string, unknown>

SerializedModuleState

type

Serialized nn.Module state: parameters and buffers.

  • readonly parameters: Record<string, { data: Array<number | string | bigint>; dtype: string; shape: number[]; }>
  • readonly buffers: Record<string, { data: Array<number | string | bigint>; dtype: string; shape: number[]; }>

SerializedPayload

type

Union of all serializable payloads.

type SerializedPayload = SerializedTensor | SerializedModuleState | SerializedEstimator;

SerializedTensor

type

Serialized representation of a tensor: flat data in row-major order plus metadata.

  • readonly data: ReadonlyArray<number | string | bigint>
  • readonly shape: readonly number[]
  • readonly dtype: string

Classes

1

WorkerPool

A pool that batches work into chunks and runs them in order on the calling thread.

Results always keep input order. A callback that throws rejects the returned promise and later chunks are not started.

Example
import { WorkerPool } from 'deepbox/core'; const pool = new WorkerPool({ maxWorkers: 4 }); const results = await pool.map([1, 2, 3, 4], (x) => x * x);console.log(results); // [1, 4, 9, 16] const sum = await pool.reduce([1, 2, 3, 4, 5, 6, 7, 8], (a, b) => a + b, 0);console.log(sum); // 36 pool.terminate();
Members (11)
  • constructor(options?: WorkerPoolOptions)
  • readonly taskTimeoutMs: number

    Task timeout in milliseconds.

  • isTerminated: boolean

    Whether the pool has been terminated.

  • status(): PoolStatus

    Get the current status of the pool.

  • exec<T, R>(fn: (arg: T) => R, arg: T): Promise<TaskResult<R>>

    Execute a function on a single item.

  • map<T, R>(items: readonly T[], fn: (item: T) => R): Promise<R[]>

    Map a function over an array of inputs.

  • reduce<T>(items: readonly T[], fn: (a: T, b: T) => T, initial: T): Promise<T>

    Reduce an array to a single value.

  • all<T>(tasks: readonly (() => T)[]): Promise<T[]>

    Execute multiple independent tasks.

  • forEach<T>(items: readonly T[], fn: (item: T, index: number) => void): Promise<void>

    Execute a function for each item, in order.

  • filter<T>(items: readonly T[], predicate: (item: T) => boolean): Promise<T[]>

    Filter an array.

  • terminate(): void

    Mark the pool as terminated.

Functions

23

availableCores

function availableCores(): number;

Get the number of available CPU cores.

Uses os.availableParallelism() in Node.js and navigator.hardwareConcurrency in browsers, falling back to 4 when neither is available.

Returns

numberNumber of available CPU cores (at least 1)

createWorkerPool

function createWorkerPool(maxWorkers?: number): WorkerPool;

Create a WorkerPool with sensible defaults.

Parameters

maxWorkersoptional
number
Maximum number of chunks (defaults to CPU count)

Returns

WorkerPoolA new WorkerPool instance

Throws

  • InvalidParameterErrorIf maxWorkers is not a positive integer

fromJSON

function fromJSON(json: string): SerializedPayload;

Deserialize a JSON string back to a payload object.

Parameters

json
string
JSON string produced by toJSON()

Returns

SerializedPayloadDeserialized payload

Throws

  • DataValidationErrorIf the string is not valid JSON or does not describe a Tensor, ModuleState or Estimator payload (for tensors, the number of data elements must equal the product of the shape unless the dtype is complex)

load

function load(path: string): Promise<SerializedPayload>;

Load a serialized payload from a file (Node.js only).

Parameters

path
string
File path to read from

Returns

Promise<SerializedPayload>Deserialized payload

Throws

  • InvalidParameterErrorIf path is empty
  • DataValidationErrorIf the file content is not a valid serialized payload
  • DeepboxErrorIf the file cannot be read or node:fs is unavailable
Example
import { load } from 'deepbox/core';const payload = await load('./model.json');

save

function save(path: string, payload: SerializedPayload): Promise<void>;

Save a serialized payload to a file (Node.js only).

Uses dynamic import of node:fs so the library remains browser-compatible when file I/O is not used.

Parameters

path
string
File path to write to
payload
SerializedPayload
The payload to serialize and save

Returns

Promise<void>

Throws

  • InvalidParameterErrorIf path is empty
  • DataValidationErrorIf payload is not a valid serialized payload
  • DeepboxErrorIf the file cannot be written or node:fs is unavailable
Example
import { save } from 'deepbox/core';// Save a tensorawait save('./model.json', { __type: 'Tensor', data: [1,2,3], shape: [3], dtype: 'float32' });

toJSON

function toJSON(payload: SerializedPayload): string;

Serialize a payload to a JSON string.

BigInt values, NaN, Infinity, -Infinity and negative zero are encoded as tagged objects so that fromJSON(toJSON(x)) restores them exactly.

Parameters

payload
SerializedPayload
The object to serialize

Returns

stringJSON string

Throws

  • DataValidationErrorIf payload is not a valid serialized payload

checkArray

function checkArray<T>(array: T, options?: { dtype?: DType | readonly DType[]; ensureNdim?: number; allowEmpty?: boolean; }): T;

Input validation on an array-like (Tensor).

Checks dtype, dimensionality, and that the tensor is non-empty (no dimension of size 0 unless allowEmpty is set). 0-d tensors count as non-empty.

Parameters

array
T
Input tensor to validate
optionsoptional
{ dtype?: DType | readonly DType[]; ensureNdim?: number; allowEmpty?: boolean; }
Validation options

Returns

TThe validated tensor (same reference)

Throws

  • DataValidationErrorIf validation fails
Example
const X = checkArray(input, { dtype: 'float32', ensureNdim: 2 });

checkIsFitted

function checkIsFitted(estimator: Record<string, unknown>, attributes?: readonly string[], msgOverride?: string): void;

Check that an estimator has been fitted.

Verifies that the given object has at least one own attribute ending with _ (the sklearn convention for fitted attributes, ignoring names that start with __) whose value is not undefined, or checks for specific attribute names if provided.

Parameters

estimator
Record<string, unknown>
The estimator object to check
attributesoptional
readonly string[]
Optional list of attribute names to check. If not provided, checks for any attribute ending with _.
msgOverrideoptional
string
Optional custom error message

Throws

  • NotFittedErrorIf the estimator has not been fitted
  • DataValidationErrorIf estimator is not an object
Example
checkIsFitted(model); // checks for any fitted attributecheckIsFitted(model, ['coef_', 'intercept_']); // checks specific attrs

checkXY

function checkXY<TX, TY>(X: TX, y: TY, options?: { allowEmpty?: boolean; dtype?: DType; multiOutput?: boolean; }): [ TX, TY ];

Input validation for standard estimators (X and y).

Checks that X is 2D, y is 1D (1D or 2D with multiOutput), neither is empty, and they have the same number of samples.

Parameters

X
TX
Feature matrix
y
TY
Target vector
optionsoptional
{ allowEmpty?: boolean; dtype?: DType; multiOutput?: boolean; }
Validation options

Returns

[ TX, TY ]Tuple of [X, y] after validation

Throws

  • DataValidationErrorIf validation fails
Example
const [Xv, yv] = checkXY(X, y);

dtypeToTypedArrayCtor

function dtypeToTypedArrayCtor(dtype: DType): Float32ArrayConstructor | Float64ArrayConstructor | Int32ArrayConstructor | BigInt64ArrayConstructor | Uint8ArrayConstructor;

Get TypedArray constructor for a given DType.

Details

Maps Deepbox dtype strings to JavaScript TypedArray constructors. Used internally for allocating tensor storage.

Mapping: - float16, bfloat16, float32, complex64 → Float32Array (the working component type; half-precision tensors store through a software-emulated wrapper over Uint16Array; the complex dtypes are reserved, tensors cannot use them yet, and their component type is listed here for completeness) - float64, complex128 → Float64Array - int32 → Int32Array - int64 → BigInt64Array - uint8 → Uint8Array - bool → Uint8Array (1 byte per boolean) - string → Not supported (throws error)

Parameters

dtype
DType
Data type identifier

Returns

Float32ArrayConstructor | Float64ArrayConstructor | Int32ArrayConstructor | BigInt64ArrayConstructor | Uint8ArrayConstructorTypedArray constructor for the given dtype

Throws

  • DTypeErrorIf dtype is 'string' (strings are stored in a plain array)
Example
const Ctor = dtypeToTypedArrayCtor('float32');const arr = new Ctor(10); // Float32Array with 10 elements

ensureNumericDType

function ensureNumericDType(dtype: DType, context?: string): NumericDType;

Ensure a dtype is numeric (non-string).

Parameters

dtype
DType
Data type identifier
contextoptional
string
Context string for error messages

Returns

NumericDTypeThe same dtype narrowed to numeric types

Throws

  • DTypeErrorIf dtype is 'string'

normalizeAxes

function normalizeAxes(axis: Axis | readonly Axis[], ndim: number): number[];

Normalize a list of axes to valid dimension indices.

Checks for duplicates (including a negative axis that resolves to an already listed one, e.g. [0, -2] for ndim=2) and validity of each axis.

Parameters

axis
Axis | readonly Axis[]
Single axis or array of axes
ndim
number
Number of dimensions

Returns

number[]Array of unique non-negative integer dimension indices

Throws

  • InvalidParameterErrorIf any axis is invalid, out of bounds, or duplicated

normalizeAxis

function normalizeAxis(axis: Axis, ndim: number): number;

Normalize an axis argument to a valid dimension index.

Supports negative indexing (Python-style) and string aliases: - "index" or "rows" -> 0 - "columns" -> 1

Parameters

axis
Axis
Axis identifier (index or alias)
ndim
number
Number of dimensions in the tensor

Returns

numberNon-negative integer dimension index

Throws

  • InvalidParameterErrorIf ndim is not a non-negative integer
  • InvalidParameterErrorIf axis is not an integer or is out of bounds (valid range is -ndim to ndim - 1)
  • InvalidParameterErrorIf axis alias is invalid
Example
normalizeAxis(-1, 3);        // 2normalizeAxis("columns", 2); // 1

shapeToSize

function shapeToSize(shape: unknown, name?: string): number;

Parameters

shape
unknown
nameoptional
string

Returns

number

validateArray

function validateArray(arr: unknown, name: string): asserts arr is unknown[];

Validate that a value is an array.

Parameters

arr
unknown
The value to validate
name
string
The parameter name for error messages

Returns

asserts arr is unknown[]

Throws

  • DataValidationErrorIf value is not an array

validateDevice

function validateDevice(device: unknown, name?: string): Device;

Parameters

device
unknown
nameoptional
string

Returns

Device

validateDtype

function validateDtype(dtype: unknown, name?: string): DType;

Parameters

dtype
unknown
nameoptional
string

Returns

DType

validateInteger

function validateInteger(value: number, name: string): void;

Validate that a value is a safe integer.

Parameters

value
number
The value to validate
name
string
The parameter name for error messages

Throws

  • DataValidationErrorIf value is not a safe integer

validateNonNegative

function validateNonNegative(value: number, name: string): void;

Validate that a value is non-negative (>= 0).

Parameters

value
number
The value to validate
name
string
The parameter name for error messages

Throws

  • DataValidationErrorIf value is negative

validateOneOf

function validateOneOf<T extends string>(value: unknown, options: readonly T[], name: string): asserts value is T;

Validate that a value is one of the allowed options.

Parameters

value
unknown
The value to validate
options
readonly T[]
The array of allowed values
name
string
The parameter name for error messages

Returns

asserts value is T

Throws

  • DataValidationErrorIf value is not in options

validatePositive

function validatePositive(value: number, name: string): void;

Validate that a value is positive (> 0).

Parameters

value
number
The value to validate
name
string
The parameter name for error messages

Throws

  • DataValidationErrorIf value is not positive

validateRange

function validateRange(value: number, min: number, max: number, name: string): void;

Validate that a value is within a specified range [min, max].

Parameters

value
number
The value to validate
min
number
The minimum allowed value (inclusive)
max
number
The maximum allowed value (inclusive)
name
string
The parameter name for error messages

Throws

  • DataValidationErrorIf value is out of range or bounds are invalid

validateShape

function validateShape(shape: unknown, name?: string): Shape;

Parameters

shape
unknown
nameoptional
string

Returns

Shape

core-utils.ts
import {  availableCores,  createWorkerPool,  fromJSON,  normalizeAxis,  save,  toJSON,  validatePositive,} from "deepbox/core"; const payload = {  __type: "Tensor",  data: [1, 2, 3, 4],  shape: [2, 2],  dtype: "float32",} as const; const json = toJSON(payload);console.log(fromJSON(json).__type); // Tensorawait save("tensor.json", payload); validatePositive(3, "batchSize");console.log(normalizeAxis(-1, 3)); // 2 const pool = createWorkerPool(Math.min(availableCores(), 4));console.log(await pool.map([1, 2, 3, 4], (x) => x * x)); // [1, 4, 9, 16]console.log(pool.status());pool.terminate();

Found a mistake? Open an issue.