torch.nn.CTCLoss

class CTCLoss extends Module

new CTCLoss(options?: { blank?: number; reduction?: Reduction; zero_infinity?: boolean })

readonlyblank(number)
readonlyreduction(Reduction)
readonlyzero_infinity(boolean)

Connectionist Temporal Classification (CTC) Loss: for sequence-to-sequence with unknown alignment.

Computes loss for tasks where input and output sequences have different lengths and alignment is unknown. The model outputs probabilities over all possible labels at each time step, and CTC marginalizes over all valid alignments. Essential for:

Speech recognition (audio to text, no character-level alignment)
Handwriting recognition
Online handwriting recognition
Any sequence-to-sequence where alignment is latent

When to use CTCLoss:

Sequence labeling with unknown frame-to-label alignment
Speech recognition or similar audio tasks
Handwriting recognition
When output sequence is shorter than input sequence
Models like wav2vec, Whisper use CTC-like losses

Key difference from other losses:

Handles variable-length input/output sequences
Marginalizes over all valid alignments (don't know which input frame → output label)
Allows repeated labels without explicit separators (uses blank token)
Computes loss over log-probabilities of label sequences

Algorithm: For a sequence (x, y) where x is input sequence of length T and y is target sequence of length S:

Model outputs log-probabilities P[t, c] for each time step t and character c
CTC sums over all valid alignments A from P to y, marginalizing over alignment uncertainty
Loss = -log(sum over valid alignments of product of probabilities)
Blank token allows collapsing repeated labels and unused frames

\begin{aligned} \text{loss} = -\log\left(\sum_{\text{alignments}} \prod_{(t,l) \in \text{alignment}} \log p_t(l)\right) \\ \text{where alignments map input sequence to target with blank tokens (B = blank symbol)} \end{aligned}

Unknown alignment: Automatically finds best frame-to-label alignment
Blank token critical: Essential for separating labels and ignoring frames
Log-probabilities: Input must be log_softmax output, not raw logits
Variable length: Handles sequences of different lengths via length tensors
Computational cost: More expensive than frame-level classification
Modern use: Speech models often use CTC loss or variants (Conformer, Whisper)
Decoding: At inference, use beam search or greedy decoding with blank removal

Input must be log-probabilities (from log_softmax), not raw logits
Blank index must be unique and not used for actual labels
Input and target lengths must match actual sequence lengths

Examples

// Speech recognition: audio frames to text
const ctc_loss = new torch.nn.CTCLoss({ blank: 0 });

// Model outputs probabilities over vocabulary at each time step
const log_probs = torch.log_softmax(torch.randn([100, 32, 28]), 2);
// [time_steps=100, batch_size=32, num_classes=28]

// Target sequences (variable length, padded)
const targets = torch.tensor([
  [1, 2, 3, 0, 0],  // "abc" (padded)
  [4, 5, 0, 0, 0],  // "de" (padded)
  // ... more samples
]);

// Actual lengths
const input_lengths = torch.tensor([100, 100, 95, ...]);  // Some shorter audio
const target_lengths = torch.tensor([3, 2, 4, ...]);

const loss = ctc_loss.forward(log_probs, targets, input_lengths, target_lengths);
// CTC finds best alignment automatically

// End-to-end speech recognition model
class SpeechRecognizer extends torch.nn.Module {
  conv: torch.nn.Conv1d;
  lstm: torch.nn.LSTM;
  fc: torch.nn.Linear;

  constructor(num_classes: number) {
    super();
    this.conv = new torch.nn.Conv1d(80, 256, 5);  // MFCC features
    this.lstm = new torch.nn.LSTM(256, 256, 2);
    this.fc = new torch.nn.Linear(256, num_classes);  // Output logits
  }

  forward(x: torch.Tensor, lengths: torch.Tensor): torch.Tensor {
    // x: [batch, time, features]
    x = this.conv(x);
    x = this.lstm(x);
    x = this.fc(x);  // [batch, time, num_classes]
    return torch.log_softmax(x, 2);  // Return log-probs
  }
}

const model = new SpeechRecognizer(28);  // 26 letters + space + blank
const ctc = new torch.nn.CTCLoss({ blank: 27 });

const audio_features = torch.randn([32, 500, 80]);  // Audio batch
const input_lengths = torch.full([32], 500);

// Targets: character sequences
const texts = torch.tensor([
  [1, 2, 3, ...],  // "abc..."
  [4, 5, ...],
]);
const target_lengths = torch.tensor([10, 8, ...]);

const log_probs = model.forward(audio_features, input_lengths);
const loss = ctc.forward(log_probs, texts, input_lengths, target_lengths);

// Handwriting recognition: image sequences to characters
const ctc_loss = new torch.nn.CTCLoss({ blank: 0 });

// Model processes handwriting features at each pixel position
const image_batch = torch.randn([32, 200, 64]);  // 200 positions, 64 features
const log_probs = torch.log_softmax(torch.randn([200, 32, 62]), 2);  // 62 chars + blank

// Variable-length text labels
const labels = torch.tensor([
  [1, 2, 3, 4, 0, 0, 0],      // "abcd"
  [5, 6, 7, 0, 0, 0, 0],      // "cde"
]);
const input_lens = torch.tensor([200, 200]);
const target_lens = torch.tensor([4, 3]);

const loss = ctc_loss.forward(log_probs, labels, input_lens, target_lens);

torch.nn.CTCLoss

Examples

See Also

torch.nn.CTCLoss

Examples

See Also