deepbox/ml

Support Vector Machines

Kernel, linear, nu-formulation and one-class support vector models.

Overview

SVC and SVR default to the rbf kernel with gamma: "scale" and C: 1.0, as scikit-learn does. They solve the dual problem with SMO and cache kernel rows (200 MB by default), so training cost grows faster than linearly with the number of samples. For large linear problems use LinearSVC or LinearSVR.
Two defaults differ from scikit-learn: LinearSVC uses loss: "hinge" (scikit-learn: squared hinge), and LinearSVR uses epsilon: 0.1 (scikit-learn: 0).
OneClassSVM returns 1 for inliers and -1 for outliers. Since 1.5.0, nu controls the share of training outliers correctly: with nu = 0.1 about 10 percent of the training rows are flagged, where 1.0.0 flagged 88 percent.
Support vector machines are sensitive to feature scale. Standardize features first, for example with StandardScaler.

Types

ClassWeightOption

type

Per-class penalty multipliers: "balanced" or a map from class label to weight.

type ClassWeightOption = "balanced" | Record<number, number>;

GammaOption

type

Kernel coefficient: a positive number, "scale" (1 / (n_features * X.var())) or "auto" (1 / n_features).

type GammaOption = number | "scale" | "auto";

KernelType

type

Kernel functions supported by the kernel SVM estimators.

type KernelType = "rbf" | "linear" | "poly" | "sigmoid";

LinearSVCLoss

type

Loss of LinearSVC: "hinge" or "squaredHinge".

type LinearSVCLoss = "hinge" | "squaredHinge";

LinearSVCOptions

type

Constructor options of LinearSVC.

readonly C?: number
Penalty of margin violations, must be positive (default: 1.0).
readonly loss?: LinearSVCLoss
Loss function (default: "hinge").
readonly fitIntercept?: boolean
Fit an intercept (default: true).
readonly interceptScaling?: number
Value of the constant feature that carries the intercept (default: 1).
readonly maxIter?: number
Maximum number of passes over the data (default: 1000).
readonly tol?: number
Stopping tolerance on the projected-gradient range (default: 1e-4).
readonly classWeight?: ClassWeightOption
Per-class multiplier of C: "balanced" or a map from class label to weight.
readonly randomState?: number
Seed of the coordinate order.

LinearSVRLoss

type

Loss of LinearSVR: "epsilonInsensitive" or "squaredEpsilonInsensitive".

type LinearSVRLoss = "epsilonInsensitive" | "squaredEpsilonInsensitive";

LinearSVROptions

type

Constructor options of LinearSVR.

readonly C?: number
Penalty of errors outside the epsilon tube, must be positive (default: 1.0).
readonly epsilon?: number
Half-width of the tube in which errors are not penalized (default: 0.1).
readonly loss?: LinearSVRLoss
Loss function (default: "epsilonInsensitive").
readonly fitIntercept?: boolean
Fit an intercept (default: true).
readonly interceptScaling?: number
Value of the constant feature that carries the intercept (default: 1).
readonly maxIter?: number
Maximum number of passes over the data (default: 1000).
readonly tol?: number
Stopping tolerance relative to the violation of the first pass (default: 1e-4).
readonly randomState?: number
Seed of the coordinate order.

NuSVCOptions

type

Constructor options of NuSVC.

readonly nu?: number
Upper bound on the fraction of margin errors and lower bound on the fraction of support vectors, in (0, 1] (default: 0.5).
readonly kernel?: KernelType
Kernel function (default: "rbf").
readonly gamma?: GammaOption
Kernel coefficient of rbf, poly and sigmoid (default: "scale").
readonly coef0?: number
Independent term of the poly and sigmoid kernels (default: 0).
readonly degree?: number
Degree of the poly kernel, an integer >= 1 (default: 3).
readonly maxIter?: number
Budget of SMO updates, expressed as passes over the data: the solver stops after at most maxIter * n working-set updates per binary problem (default: 1000).
readonly tol?: number
Stopping tolerance on the maximal KKT violation (default: 1e-3).
readonly cacheSize?: number
Size of the kernel row cache in megabytes (default: 200).

NuSVROptions

type

Constructor options of NuSVR.

readonly nu?: number
Upper bound on the fraction of training errors and lower bound on the fraction of support vectors, in (0, 1] (default: 0.5).
readonly C?: number
Penalty of errors, must be positive (default: 1.0).
readonly kernel?: KernelType
Kernel function (default: "rbf").
readonly gamma?: GammaOption
Kernel coefficient of rbf, poly and sigmoid (default: "scale").
readonly coef0?: number
Independent term of the poly and sigmoid kernels (default: 0).
readonly degree?: number
Degree of the poly kernel, an integer >= 1 (default: 3).
readonly maxIter?: number
Budget of SMO updates in passes over the 2 * n dual variables (default: 1000).
readonly tol?: number
Stopping tolerance on the maximal KKT violation (default: 1e-3).
readonly cacheSize?: number
Size of the kernel row cache in megabytes (default: 200).

OneClassSVMOptions

type

Constructor options of OneClassSVM.

type OneClassSVMOptions = NuSVCOptions;

SVCOptions

type

Constructor options of SVC.

readonly C?: number
Penalty of margin violations, must be positive (default: 1.0).
readonly kernel?: KernelType
Kernel function (default: "rbf").
readonly gamma?: GammaOption
Kernel coefficient of rbf, poly and sigmoid (default: "scale").
readonly coef0?: number
Independent term of the poly and sigmoid kernels (default: 0).
readonly degree?: number
Degree of the poly kernel, an integer >= 1 (default: 3).
readonly maxIter?: number
Budget of SMO updates, expressed as passes over the data: the solver stops after at most maxIter * n working-set updates per binary problem (default: 1000).
readonly tol?: number
Stopping tolerance on the maximal KKT violation (default: 1e-3).
readonly classWeight?: ClassWeightOption
Per-class multiplier of C: "balanced" or a map from class label to weight.
readonly cacheSize?: number
Size of the kernel row cache in megabytes (default: 200).

SVROptions

type

Constructor options of SVR.

readonly C?: number
Penalty of errors outside the epsilon tube, must be positive (default: 1.0).
readonly kernel?: KernelType
Kernel function (default: "rbf").
readonly gamma?: GammaOption
Kernel coefficient of rbf, poly and sigmoid (default: "scale").
readonly coef0?: number
Independent term of the poly and sigmoid kernels (default: 0).
readonly degree?: number
Degree of the poly kernel, an integer >= 1 (default: 3).
readonly epsilon?: number
Half-width of the tube in which errors are not penalized (default: 0.1).
readonly maxIter?: number
Budget of SMO updates in passes over the 2 * n dual variables (default: 1000).
readonly tol?: number
Stopping tolerance on the maximal KKT violation (default: 1e-3).
readonly cacheSize?: number
Size of the kernel row cache in megabytes (default: 200).

Classes

LinearSVC

Support Vector Machine classifier with a linear kernel.

Minimizes 0.5 ||w||^2 + C * sum_i loss(y_i, w.x_i + b) with dual coordinate descent (the LIBLINEAR algorithm). More than two classes are handled one-vs-rest.

predictProba returns a monotone squashing of the decision values, not calibrated probabilities.

ExampleTypeScript

import { LinearSVC } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X = tensor([[1, 2], [2, 3], [3, 1], [4, 2]]);const y = tensor([0, 0, 1, 1]); const svm = new LinearSVC({ C: 1.0 });svm.fit(X, y);const predictions = svm.predict(X);

Members (13)

constructor(options?: LinearSVCOptions)
classes: Tensor | undefined
Sorted class labels seen during fit, or undefined before fitting.
coef: Tensor
Weight vectors, shape (1, n_features) for two classes (positive means classes[1]) and (n_classes, n_features) otherwise.
intercept: Tensor
Intercepts, shape (1,) for two classes and (n_classes,) otherwise.
nIter: number
Number of passes over the data used by the slowest binary problem.
fit(X: Tensor, y: Tensor, sampleWeightArg?: Tensor | void): this
Fit the classifier.
predict(X: Tensor): Tensor
Predict class labels.
decisionFunction(X: Tensor): Tensor
Signed distances X @ coef.T + intercept to the separating hyperplanes.
predictProba(X: Tensor): Tensor
Class scores squashed into rows that sum to one: the logistic function of the decision value for two classes, the per-class logistic values normalized to sum to one otherwise.
score(X: Tensor, y: Tensor): number
Mean accuracy on the given data.
getParams(): Record<string, unknown>
Get hyperparameters, including classWeight and randomState only when set.
setParams(params: Record<string, unknown>): this
Set hyperparameters.
clone(): LinearSVC
Create an unfitted copy with the same hyperparameters.

LinearSVR

Support Vector Regression with a linear kernel.

Minimizes 0.5 ||w||^2 + C * sum_i loss(|y_i - w.x_i - b| - epsilon) with dual coordinate descent (the LIBLINEAR algorithm).

ExampleTypeScript

import { LinearSVR } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X = tensor([[1], [2], [3], [4]]);const y = tensor([1.5, 2.5, 3.5, 4.5]); const svr = new LinearSVR({ C: 1.0, epsilon: 0.1 });svr.fit(X, y);const predictions = svr.predict(X);

Members (10)

constructor(options?: LinearSVROptions)
coef: Tensor
Weight vector, shape (n_features,).
intercept: Tensor
Intercept, shape (1,).
nIter: number
Number of passes over the data used by the last fit.
fit(X: Tensor, y: Tensor): this
Fit the regressor.
predict(X: Tensor): Tensor
Predict target values.
score(X: Tensor, y: Tensor): number
Coefficient of determination R^2 on the given data.
getParams(): Record<string, unknown>
Get hyperparameters, including randomState only when set.
setParams(params: Record<string, unknown>): this
Set hyperparameters.
clone(): LinearSVR
Create an unfitted copy with the same hyperparameters.

NuSVC

Nu-Support Vector Classification.

Like SVC but nu in (0, 1] replaces C: it is an upper bound on the fraction of margin errors and a lower bound on the fraction of support vectors. More than two classes are handled with one-vs-one voting.

predictProba returns a monotone squashing of the decision values, not calibrated probabilities.

ExampleTypeScript

import { NuSVC } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X = tensor([[0, 0], [1, 1], [1, 0], [0, 1]]);const y = tensor([0, 0, 1, 1]); const clf = new NuSVC({ nu: 0.5 });clf.fit(X, y);const predictions = clf.predict(X);

Members (15)

constructor(options?: NuSVCOptions)
classes: Tensor | undefined
Sorted class labels seen during fit, or undefined before fitting.
supportVectors: Tensor
Support vectors, shape (n_SV, n_features), grouped by class.
supportIndices: Tensor
Indices of the support vectors in the training data, shape (n_SV,), dtype int32.
nSupport: Tensor
Number of support vectors of each class, shape (n_classes,), dtype int32.
dualCoef: Tensor
Dual coefficients alpha_i * y_i, shape (n_classes - 1, n_SV), scikit-learn layout.
intercept: Tensor
Decision-function offsets, one per class pair, scikit-learn sign convention.
fit(X: Tensor, y: Tensor): this
Fit the classifier.
predict(X: Tensor): Tensor
Predict class labels by one-vs-one voting.
decisionFunction(X: Tensor): Tensor
Signed decision values.
predictProba(X: Tensor): Tensor
Class scores squashed into rows that sum to one.
score(X: Tensor, y: Tensor): number
Mean accuracy on the given data.
getParams(): Record<string, unknown>
Get hyperparameters.
setParams(params: Record<string, unknown>): this
Set hyperparameters.
clone(): NuSVC
Create an unfitted copy with the same hyperparameters.

NuSVR

Nu-Support Vector Regression.

Like SVR but nu in (0, 1] controls the fraction of support vectors and the width of the insensitive tube is found by the solver instead of being given.

ExampleTypeScript

import { NuSVR } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X = tensor([[1], [2], [3], [4], [5]]);const y = tensor([1.2, 2.1, 2.9, 4.0, 5.1]); const svr = new NuSVR({ nu: 0.5 });svr.fit(X, y);const predictions = svr.predict(X);

Members (11)

constructor(options?: NuSVROptions)
supportVectors: Tensor
Support vectors, shape (n_SV, n_features).
supportIndices: Tensor
Indices of the support vectors in the training data, shape (n_SV,), dtype int32.
dualCoef: Tensor
Dual coefficients alpha_i - alpha_i*, shape (1, n_SV).
intercept: Tensor
Intercept of the decision function, shape (1,).
fit(X: Tensor, y: Tensor): this
Fit the regressor.
predict(X: Tensor): Tensor
Predict target values.
score(X: Tensor, y: Tensor): number
Coefficient of determination R^2 on the given data.
getParams(): Record<string, unknown>
Get hyperparameters.
setParams(params: Record<string, unknown>): this
Set hyperparameters.
clone(): NuSVR
Create an unfitted copy with the same hyperparameters.

OneClassSVM

One-Class SVM for unsupervised outlier detection.

Learns a boundary that separates the data from the origin in kernel feature space (Scholkopf et al.). nu in (0, 1] bounds the fraction of training errors from above and the fraction of support vectors from below.

scoreSamples returns the raw kernel expansion sum_i alpha_i K(sv_i, x), decisionFunction returns scoreSamples(X) - offset, and predict returns +1 where the decision function is >= 0 and -1 elsewhere.

ExampleTypeScript

import { OneClassSVM } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X = tensor([[1, 1], [1, 2], [2, 1], [2, 2], [10, 10]]); const ocsvm = new OneClassSVM({ nu: 0.2 });ocsvm.fit(X);const labels = ocsvm.predict(X); // +1 inlier, -1 outlier

Members (13)

constructor(options?: OneClassSVMOptions)
offset: number
Decision threshold in scoreSamples units: samples scoring below it are outliers.
supportVectors: Tensor
Support vectors, shape (n_SV, n_features).
supportIndices: Tensor
Indices of the support vectors in the training data, shape (n_SV,), dtype int32.
dualCoef: Tensor
Dual coefficients alpha_i (sum to nu * n_samples), shape (1, n_SV).
fit(X: Tensor, _y?: Tensor): this
Fit the detector on inlier-dominated data.
predict(X: Tensor): Tensor
Predict whether samples are inliers.
fitPredict(X: Tensor, _y?: Tensor): Tensor
Fit on X, then predict labels for the same samples.
scoreSamples(X: Tensor): Tensor
Raw scores sum_i alpha_i K(sv_i, x): lower values are more abnormal.
decisionFunction(X: Tensor): Tensor
Signed distance to the decision boundary, scoreSamples(X) - offset: negative values are outliers.
getParams(): Record<string, unknown>
Get hyperparameters.
setParams(params: Record<string, unknown>): this
Set hyperparameters.
clone(): OneClassSVM
Create an unfitted copy with the same hyperparameters.

SVC

Support Vector Classification with the kernel trick.

Solves the C-SVC dual with the LIBSVM working-set SMO algorithm and handles more than two classes with one-vs-one voting, like scikit-learn's SVC. Supports RBF, polynomial, sigmoid and linear kernels.

predictProba returns a monotone squashing of the decision values, not calibrated probabilities. Wrap the classifier in CalibratedClassifierCV when calibrated probabilities matter.

ExampleTypeScript

import { SVC } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X = tensor([[0, 0], [1, 1], [1, 0], [0, 1]]);const y = tensor([0, 0, 1, 1]); const svc = new SVC({ kernel: 'rbf', C: 1.0 });svc.fit(X, y);const predictions = svc.predict(X);

Members (16)

constructor(options?: SVCOptions)
classes: Tensor | undefined
Sorted class labels seen during fit, or undefined before fitting.
supportVectors: Tensor
Support vectors, shape (n_SV, n_features), grouped by class.
supportIndices: Tensor
Indices of the support vectors in the training data, shape (n_SV,), dtype int32.
nSupport: Tensor
Number of support vectors of each class, shape (n_classes,), dtype int32.
dualCoef: Tensor
Dual coefficients alpha_i * y_i, shape (n_classes - 1, n_SV), scikit-learn layout.
intercept: Tensor
Decision-function offsets, one per class pair, scikit-learn sign convention.
nIter: number
Largest number of SMO updates used by any binary sub-problem.
fit(X: Tensor, y: Tensor, sampleWeightArg?: Tensor | void): this
Fit the classifier.
predict(X: Tensor): Tensor
Predict class labels by one-vs-one voting.
decisionFunction(X: Tensor): Tensor
Signed decision values.
predictProba(X: Tensor): Tensor
Class scores squashed into rows that sum to one: the logistic function of the decision value for two classes, a softmax of the one-vs-rest scores otherwise.
score(X: Tensor, y: Tensor): number
Mean accuracy on the given data.
getParams(): Record<string, unknown>
Get hyperparameters, including classWeight only when it is set.
setParams(params: Record<string, unknown>): this
Set hyperparameters.
clone(): SVC
Create an unfitted copy with the same hyperparameters.

SVR

Epsilon-Support Vector Regression with the kernel trick.

Solves the epsilon-SVR dual with the LIBSVM working-set SMO algorithm, so the result agrees with scikit-learn's SVR up to the solver tolerance. Supports RBF, polynomial, sigmoid and linear kernels.

ExampleTypeScript

import { SVR } from 'deepbox/ml';import { tensor } from 'deepbox/ndarray'; const X = tensor([[1], [2], [3], [4], [5]]);const y = tensor([1.2, 2.1, 2.9, 4.0, 5.1]); const svr = new SVR({ kernel: 'rbf', C: 10 });svr.fit(X, y);const predictions = svr.predict(X);

Members (11)

constructor(options?: SVROptions)
supportVectors: Tensor
Support vectors, shape (n_SV, n_features).
supportIndices: Tensor
Indices of the support vectors in the training data, shape (n_SV,), dtype int32.
dualCoef: Tensor
Dual coefficients alpha_i - alpha_i*, shape (1, n_SV).
intercept: Tensor
Intercept of the decision function, shape (1,).
fit(X: Tensor, y: Tensor, sampleWeightArg?: Tensor | void): this
Fit the regressor.
predict(X: Tensor): Tensor
Predict target values.
score(X: Tensor, y: Tensor): number
Coefficient of determination R^2 on the given data.
getParams(): Record<string, unknown>
Get hyperparameters.
setParams(params: Record<string, unknown>): this
Set hyperparameters.
clone(): SVR
Create an unfitted copy with the same hyperparameters.

ml-svm.tsTypeScript

import { LinearSVC, OneClassSVM, SVC, SVR } from "deepbox/ml";import { tensor } from "deepbox/ndarray"; const X = tensor([[1, 2], [2, 1], [3, 3], [4, 4]]);const y = tensor([0, 0, 1, 1]); console.log(new LinearSVC().fit(X, y).predict(X).toString());console.log(new SVC({ kernel: "rbf", C: 1 }).fit(X, y).predict(X).toString());console.log(new SVR({ kernel: "rbf" }).fit(X, tensor([1, 1.5, 3, 4])).predict(X).toString());console.log(new OneClassSVM({ nu: 0.25 }).fit(X).predict(X).toString()); // 1 inlier, -1 outlier

Found a mistake? Open an issue.