deepbox/ml

Trees & Forests

Decision trees, random forests and extremely randomized trees for classification and regression, with exportText for reading a fitted tree.

Trees

Overview

maxDepth defaults to 10 for trees and forests, where scikit-learn grows trees without a limit. Pass Infinity for full trees.
Forests build 100 trees by default. RandomForestClassifier examines sqrt(nFeatures) features per split and RandomForestRegressor examines all of them. ExtraTrees* draw random thresholds and do not bootstrap by default.
Trees and forests take sampleWeight in fit and the options classWeight, minImpurityDecrease, maxLeafNodes and ccpAlpha (cost-complexity pruning). Class labels may be any finite numbers. predict returns int32 when all labels are integers and float64 otherwise.
Set randomState for repeatable trees, or call setSeed to seed the global generator. featureImportances is a getter, and exportText(tree) prints the learned rules.

Types

ClassificationCriterion

type

Impurity criterion used by the classification trees.

"log_loss" is an alias of "entropy"; both use the base-2 Shannon entropy, as scikit-learn does (this matters for minImpurityDecrease and ccpAlpha).

type ClassificationCriterion = "gini" | "entropy" | "log_loss";

DecisionTreeClassifierOptions

type

Options of DecisionTreeClassifier.

type DecisionTreeClassifierOptions = {    readonly maxDepth?: number;    readonly minSamplesSplit?: number;    readonly minSamplesLeaf?: number;    readonly maxFeatures?: TreeMaxFeatures;    readonly randomState?: number;    readonly criterion?: ClassificationCriterion;    readonly classWeight?: TreeClassWeight;} & TreeGrowthOptions;

DecisionTreeRegressorOptions

type

Options of DecisionTreeRegressor.

type DecisionTreeRegressorOptions = {    readonly maxDepth?: number;    readonly minSamplesSplit?: number;    readonly minSamplesLeaf?: number;    readonly maxFeatures?: TreeMaxFeatures;    readonly randomState?: number;} & TreeGrowthOptions;

ForestClassWeight

type

Class weights of a forest: the options of TreeClassWeight plus "balanced_subsample", which computes the "balanced" weights again on the bootstrap sample of every tree.

type ForestClassWeight = TreeClassWeight | "balanced_subsample";

ForestMaxFeatures

type

Number of features examined at every split of a forest tree.

- an integer >= 1: that many features (capped at the number of features) - a number in (0, 1): that fraction of the features, at least one - "sqrt": floor(sqrt(nFeatures)) - "log2": floor(log2(nFeatures)) - null: every feature (scikit-learn's None), whatever the class default is

Results are never below one feature.

type ForestMaxFeatures = number | "sqrt" | "log2" | null;

RandomForestClassifierOptions

type

Options of RandomForestClassifier.

type RandomForestClassifierOptions = RandomForestOptions & {    readonly criterion?: ClassificationCriterion;    readonly classWeight?: ForestClassWeight;};

RandomForestOptions

type

Options shared by RandomForestClassifier and RandomForestRegressor.

type RandomForestOptions = {    readonly nEstimators?: number;    readonly maxDepth?: number;    readonly minSamplesSplit?: number;    readonly minSamplesLeaf?: number;    readonly maxFeatures?: ForestMaxFeatures;    readonly bootstrap?: boolean;    readonly warmStart?: boolean;    readonly randomState?: number;    readonly maxSamples?: number;    readonly oobScore?: boolean;} & TreeGrowthOptions;

RandomForestRegressorOptions

type

Options of RandomForestRegressor.

type RandomForestRegressorOptions = RandomForestOptions;

TreeClassWeight

type

Class weights of a classification tree: "balanced" or a map from class label to weight.

"balanced" gives class k the weight n_samples / (n_classes * count_k). With a map, classes that are not listed keep the weight 1, and every key must be a class of y. Weights multiply the sampleWeight of fit.

type TreeClassWeight = "balanced" | Readonly<Record<number, number>>;

TreeGrowthOptions

type

Options that control how far a tree grows. They are accepted by the decision trees, random forests, extremely randomized trees and gradient boosting estimators in deepbox/ml.

readonly minImpurityDecrease?: number
A node is split only if the split lowers the weighted impurity by at least this value: (W_node / W_total) * (impurity - W_left / W_node * impurity_left - W_right / W_node * impurity_right) >= minImpurityDecrease, with W the sum of the…
readonly maxLeafNodes?: number
Grow the tree best-first (the split with the largest impurity decrease first) until it has this many leaves.
readonly ccpAlpha?: number
Strength of minimal cost-complexity pruning: the subtrees with the smallest effective alpha are cut away as long as that alpha is not above ccpAlpha.

TreeMaxFeatures

type

Number of features examined at every split: an exact count, "sqrt" (floor(sqrt(nFeatures))) or "log2" (floor(log2(nFeatures))), never below 1.

type TreeMaxFeatures = number | "sqrt" | "log2";

ExtraTreesClassifierOptions

type

Options of ExtraTreesClassifier.

type ExtraTreesClassifierOptions = ExtraTreesOptions & {    readonly criterion?: ClassificationCriterion;    readonly classWeight?: ForestClassWeight;};

ExtraTreesOptions

type

Options shared by ExtraTreesClassifier and ExtraTreesRegressor.

type ExtraTreesOptions = {    readonly nEstimators?: number;    readonly maxDepth?: number;    readonly minSamplesSplit?: number;    readonly minSamplesLeaf?: number;    readonly maxFeatures?: TreeMaxFeatures;    readonly bootstrap?: boolean;    readonly randomState?: number;} & TreeGrowthOptions;

Classes

DecisionTreeClassifier

Decision Tree Classifier.

A non-parametric supervised learning method that learns simple decision rules inferred from the data features.

Algorithm: CART (Classification and Regression Trees) - Splits on Gini impurity (default) or Shannon entropy - Recursively splits data based on feature thresholds (x <= threshold goes left) - Supports maxDepth, minSamplesSplit, minSamplesLeaf, minImpurityDecrease, maxLeafNodes (best-first growth) and ccpAlpha (cost-complexity pruning) for regularization - fit accepts per-sample weights and the classWeight option weights whole classes - Ties between equally good classes resolve to the smallest class label

Class labels may be any finite numbers. predict and classes are int32 when all labels are integers and float64 otherwise.

ExampleTypeScript

import { DecisionTreeClassifier } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X = tensor([[1, 2], [3, 4], [5, 6], [7, 8]]);const y = tensor([0, 0, 1, 1]); const clf = new DecisionTreeClassifier({ maxDepth: 3 });clf.fit(X, y);const predictions = clf.predict(X);

Members (14)

constructor(options?: DecisionTreeClassifierOptions)
classes: Tensor | undefined
Get the unique class labels discovered during fitting.
tree_: TreeNode | undefined
Access the internal tree structure (for export_text).
nFeatures_: number | undefined
Number of features seen during fit.
featureImportances: Tensor
Normalized mean decrease in impurity per feature (sums to 1, or all zeros when the tree never split).
fit(X: Tensor, y: Tensor, sampleWeightArg?: Tensor | void): this
Build a decision tree classifier from the training set (X, y).
predict(X: Tensor): Tensor
Predict class labels for samples in X.
predictProba(X: Tensor): Tensor
Predict class probabilities for samples in X.
score(X: Tensor, y: Tensor): number
Return the mean accuracy on the given test data and labels.
getParams(): Record<string, unknown>
Get hyperparameters for this estimator.
setParams(params: Record<string, unknown>): this
Set the parameters of this estimator.
getDepth(): number
Depth of the fitted tree (a single leaf has depth 0).
getNLeaves(): number
Number of leaves of the fitted tree.
clone(): DecisionTreeClassifier
Create an unfitted copy with the same hyperparameters.

DecisionTreeRegressor

Decision Tree Regressor.

Uses MSE reduction to find optimal splits for regression tasks. Leaves predict the mean target of their training samples; predict returns float64.

ExampleTypeScript

import { DecisionTreeRegressor } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X = tensor([[1], [2], [3], [4]]);const y = tensor([1.5, 1.7, 3.1, 3.3]);const reg = new DecisionTreeRegressor({ maxDepth: 2 }).fit(X, y);const predictions = reg.predict(X);

Members (13)

constructor(options?: DecisionTreeRegressorOptions)
tree_: TreeNode | undefined
Access the internal tree structure (for export_text).
nFeatures_: number | undefined
Number of features seen during fit.
featureImportances: Tensor
Normalized mean decrease in impurity per feature (sums to 1, or all zeros when the tree never split).
fit(X: Tensor, y: Tensor, sampleWeightArg?: Tensor | void): this
Build a decision tree regressor from the training set (X, y).
predict(X: Tensor): Tensor
Predict target values for samples in X.
remapLeaves(fn: (leafValue: number) => number): void
Rewrite each leaf's prediction via fn(oldValue).
score(X: Tensor, y: Tensor): number
Return the R² score on the given test data and target values.
getParams(): Record<string, unknown>
Get hyperparameters for this estimator.
setParams(params: Record<string, unknown>): this
Set the parameters of this estimator.
getDepth(): number
Depth of the fitted tree (a single leaf has depth 0).
getNLeaves(): number
Number of leaves of the fitted tree.
clone(): DecisionTreeRegressor
Create an unfitted copy with the same hyperparameters.

ExtraTreesClassifier

Extremely Randomized Trees Classifier.

Like Random Forest, but uses random split thresholds instead of searching for the best threshold, and by default does not bootstrap. This leads to lower variance at the cost of slightly higher bias.

At every node a random threshold, drawn uniformly between the node's minimum and maximum, is tried for each of maxFeatures randomly chosen features, and the candidate with the lowest weighted impurity is used. predictProba averages the leaf class frequencies of all trees and predict returns the class with the highest average (ties go to the smallest label).

ExampleTypeScript

import { ExtraTreesClassifier } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const clf = new ExtraTreesClassifier({ nEstimators: 100, randomState: 42 });clf.fit(X_train, y_train);const predictions = clf.predict(X_test);

Members (11)

constructor(options?: ExtraTreesClassifierOptions)
classes: Tensor | undefined
Sorted class labels seen during fit (int32 for integer labels, float64 otherwise), or undefined if not fitted.
nFeatures_: number | undefined
Number of features seen during fit.
featureImportances: Tensor
Mean decrease in impurity per feature, averaged over the trees (sums to 1, or all zeros if no tree split).
fit(X: Tensor, y: Tensor, sampleWeightArg?: Tensor | void): this
Fit the ensemble on training data.
predict(X: Tensor): Tensor
Predict class labels: the class with the highest mean probability over the trees.
predictProba(X: Tensor): Tensor
Predict class probabilities: the leaf class frequencies averaged over the trees.
score(X: Tensor, y: Tensor): number
Mean accuracy on the given test data and labels.
getParams(): Record<string, unknown>
Get hyperparameters for this estimator.
setParams(params: Record<string, unknown>): this
Set parameters.
clone(): ExtraTreesClassifier
Create an unfitted copy with the same hyperparameters.

ExtraTreesRegressor

Extremely Randomized Trees Regressor.

Like Random Forest Regressor, but uses random split thresholds and by default does not bootstrap. Predictions are averaged across trees. Every split considers all features unless maxFeatures says otherwise.

ExampleTypeScript

import { ExtraTreesRegressor } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const reg = new ExtraTreesRegressor({ nEstimators: 100, randomState: 42 });reg.fit(X_train, y_train);const predictions = reg.predict(X_test);

Members (9)

constructor(options?: ExtraTreesOptions)
nFeatures_: number | undefined
Number of features seen during fit.
featureImportances: Tensor
Mean decrease in impurity per feature, averaged over the trees (sums to 1, or all zeros if no tree split).
fit(X: Tensor, y: Tensor, sampleWeightArg?: Tensor | void): this
Fit the ensemble on training data.
predict(X: Tensor): Tensor
Predict target values: the mean of the trees' predictions.
score(X: Tensor, y: Tensor): number
R² score on the given test data.
getParams(): Record<string, unknown>
Get hyperparameters for this estimator.
setParams(params: Record<string, unknown>): this
Set parameters.
clone(): ExtraTreesRegressor
Create an unfitted copy with the same hyperparameters.

RandomForestClassifier

Random Forest Classifier.

An ensemble of decision trees trained on bootstrap samples with a random subset of the features examined at every split. Class probabilities are the average of the trees' probabilities and predict returns the class with the highest average probability (the smallest label on ties), as scikit-learn does.

Algorithm: 1. Draw nEstimators bootstrap samples from the training data 2. Grow a decision tree on each sample, examining maxFeatures random features per split 3. Average the trees' class probabilities

Differences from scikit-learn defaults: maxDepth defaults to 10 (not unlimited); pass Infinity to grow full trees.

ExampleTypeScript

import { RandomForestClassifier } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X_train = tensor([[0, 0], [0, 1], [1, 0], [1, 1], [5, 5], [5, 6], [6, 5], [6, 6]]);const y_train = tensor([0, 0, 0, 0, 1, 1, 1, 1]);const clf = new RandomForestClassifier({ nEstimators: 50, randomState: 0 });clf.fit(X_train, y_train);const predictions = clf.predict(tensor([[0.5, 0.5], [5.5, 5.5]]));

Members (13)

constructor(options?: RandomForestClassifierOptions)
oobScore: number
Out-of-bag accuracy: each training sample is classified by the trees whose bootstrap sample did not contain it (probabilities averaged, as in scikit-learn).
oobDecisionFunction: Tensor
Out-of-bag class probabilities of the training samples, shape (n_samples, n_classes).
classes: Tensor | undefined
Get the unique class labels discovered during fitting.
featureImportances: Tensor
Get feature importances averaged across all trees.
fit(X: Tensor, y: Tensor, sampleWeightArg?: Tensor | void): this
Fit the random forest classifier on training data.
predict(X: Tensor): Tensor
Predict class labels for samples in X.
predictProba(X: Tensor): Tensor
Predict class probabilities for samples in X.
predictLogProba(X: Tensor): Tensor
Predict log class probabilities for samples in X (-Infinity for impossible classes).
score(X: Tensor, y: Tensor): number
Return the mean accuracy on the given test data and labels.
getParams(): Record<string, unknown>
Get hyperparameters for this estimator.
setParams(params: Record<string, unknown>): this
Set the parameters of this estimator.
clone(): RandomForestClassifier
Create an unfitted copy with the same hyperparameters.

RandomForestRegressor

Random Forest Regressor.

An ensemble of decision tree regressors trained on bootstrap samples with a random subset of the features examined at every split. Predictions are the mean of the trees' predictions.

By default every feature is examined at every split (maxFeatures 1, which JavaScript cannot tell apart from the float 1.0 that scikit-learn uses for "all features"; null states it explicitly). To examine fewer, pass "sqrt", "log2", an integer >= 2, or a fraction in (0, 1). maxDepth defaults to 10 (not unlimited); pass Infinity to grow full trees.

ExampleTypeScript

import { RandomForestRegressor } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X_train = tensor([[1], [2], [3], [4], [5], [6]]);const y_train = tensor([1.1, 1.9, 3.2, 3.9, 5.1, 6.0]);const reg = new RandomForestRegressor({ nEstimators: 50, randomState: 0 });reg.fit(X_train, y_train);const predictions = reg.predict(tensor([[2.5], [4.5]]));

Members (10)

constructor(options?: RandomForestRegressorOptions)
oobScore: number
Out-of-bag R²: each training sample is predicted by the mean of the trees whose bootstrap sample did not contain it.
oobPrediction: Tensor
Out-of-bag predictions of the training samples, shape (n_samples,); NaN for samples that no tree left out.
featureImportances: Tensor
Get feature importances averaged across all trees.
fit(X: Tensor, y: Tensor, sampleWeightArg?: Tensor | void): this
Fit the random forest regressor on training data.
predict(X: Tensor): Tensor
Predict target values for samples in X.
score(X: Tensor, y: Tensor): number
Return the R² score on the given test data and target values.
getParams(): Record<string, unknown>
Get hyperparameters for this estimator.
setParams(params: Record<string, unknown>): this
Set the parameters of this estimator.
clone(): RandomForestRegressor
Create an unfitted copy with the same hyperparameters.

Functions

exportText

function exportText(tree: DecisionTreeClassifier | DecisionTreeRegressor, options?: { featureNames?: readonly string[]; decimals?: number; }): string;

Build a text representation of a decision tree.

Works with both DecisionTreeClassifier and DecisionTreeRegressor.

Parameters

tree DecisionTreeClassifier | DecisionTreeRegressor: A fitted DecisionTreeClassifier or DecisionTreeRegressor
optionsoptional { featureNames?: readonly string[]; decimals?: number; }: Optional feature names and the number of decimals (0 to 100, default 2)

Returns

stringMulti-line string representation of the tree

Throws

NotFittedErrorIf the tree has not been fitted
InvalidParameterErrorIf decimals is not an integer in [0, 100]

ExampleTypeScript

const clf = new DecisionTreeClassifier({ maxDepth: 3 });clf.fit(X, y);console.log(exportText(clf));// |--- feature_1 <= 3.50// |   |--- class: 0// |--- feature_1 > 3.50// |   |--- class: 1

Deprecated names

These older spellings still work in 1.x. New code should use the names above.

export_textPrefer exportText.

ml-tree.tsTypeScript

import {  DecisionTreeClassifier,  ExtraTreesClassifier,  RandomForestClassifier,  exportText,} from "deepbox/ml";import { tensor } from "deepbox/ndarray"; const X = tensor([[1, 2], [2, 1], [3, 3], [4, 4]]);const y = tensor([0, 0, 1, 1]); const tree = new DecisionTreeClassifier({ maxDepth: 2 }).fit(X, y);console.log(tree.predict(X).toString());console.log(exportText(tree));console.log(tree.featureImportances.toString()); const forest = new RandomForestClassifier({ nEstimators: 20, randomState: 42 }).fit(X, y);console.log(forest.predict(X).toString(), forest.predictProba(X).toString());console.log(new ExtraTreesClassifier({ nEstimators: 20, randomState: 42 }).fit(X, y).predict(X).toString());

Found a mistake? Open an issue.