All Examples
Example 29
advanced
29
Tensors
Neural Networks

Attention & Transformer Layers

Demonstrates MultiheadAttention and TransformerEncoderLayer for sequence-to-sequence modeling. This example uses deepbox/ndarray, deepbox/nn and focuses on tensor, GradTensor; MultiheadAttention, TransformerEncoderLayer.

Deepbox Modules Used

deepbox/ndarraydeepbox/nn

What You Will Learn

  • Use deepbox/ndarray for tensor, GradTensor.
  • Use deepbox/nn for MultiheadAttention, TransformerEncoderLayer.
  • Demonstrates MultiheadAttention and TransformerEncoderLayer for sequence-to-sequence modeling.

Source Files

index.ts
1/**2 * Example 29: Attention & Transformer Layers3 *4 * Demonstrates MultiheadAttention and TransformerEncoderLayer.5 * Attention mechanisms allow models to focus on relevant parts of the input sequence.6 */78import { GradTensor, tensor } from "deepbox/ndarray";9import { MultiheadAttention, TransformerEncoderLayer } from "deepbox/nn";1011console.log("=== Attention & Transformer Layers ===\n");1213// ---------------------------------------------------------------------------14// Part 1: Multi-Head Attention15// ---------------------------------------------------------------------------16console.log("--- Part 1: Multi-Head Attention ---");1718// MultiheadAttention(embedDim, numHeads)19// embedDim must be divisible by numHeads20const mha = new MultiheadAttention(8, 2);21console.log("MultiheadAttention(embedDim=8, numHeads=2)");22console.log("  Each head has dimension 8/2 = 4\n");2324// Input: (batch, seqLen, embedDim)25// Self-attention: query = key = value = same input26const seqData = tensor([27  [28    [1, 0, 1, 0, 1, 0, 1, 0],29    [0, 1, 0, 1, 0, 1, 0, 1],30    [1, 1, 0, 0, 1, 1, 0, 0],31  ],32]);33console.log(`Input shape: [${seqData.shape.join(", ")}]  (batch=1, seq=3, embed=8)`);3435// Self-attention: Q=K=V=input36const attnOut = mha.forward(seqData, seqData, seqData);37const attnShape = attnOut instanceof GradTensor ? attnOut.tensor.shape : attnOut.shape;38console.log(`Output shape: [${attnShape.join(", ")}]`);39console.log("  Each position attends to all other positions\n");4041// ---------------------------------------------------------------------------42// Part 2: TransformerEncoderLayer43// ---------------------------------------------------------------------------44console.log("--- Part 2: TransformerEncoderLayer ---");4546// TransformerEncoderLayer combines:47//   MultiheadAttention + FeedForward + LayerNorm + Dropout48const encoderLayer = new TransformerEncoderLayer(8, 2, 16);49console.log("TransformerEncoderLayer(dModel=8, nHead=2, dimFeedforward=16)");50console.log(`Input shape: [${seqData.shape.join(", ")}]`);5152const encoderOut = encoderLayer.forward(seqData);53const encShape = encoderOut instanceof GradTensor ? encoderOut.tensor.shape : encoderOut.shape;54console.log(`Output shape: [${encShape.join(", ")}]`);55console.log("  Full transformer encoder block with residual connections\n");5657// ---------------------------------------------------------------------------58// Part 3: Parameter inspection59// ---------------------------------------------------------------------------60console.log("--- Part 3: Parameter Counts ---");61const mhaParams = Array.from(mha.parameters()).length;62const encParams = Array.from(encoderLayer.parameters()).length;63console.log(`MultiheadAttention params: ${mhaParams}`);64console.log(`TransformerEncoderLayer params: ${encParams}`);65console.log("  Encoder layer includes attention + feedforward + normalization");6667console.log("\n=== Attention & Transformer Complete ===");68

Console Output

$ npx tsx 29-attention-transformer/index.ts
Console output showing self-attention and transformer encoder operations
PreviousNext