torch.nn.SoftshrinkOptions
Softshrink activation function (Soft shrinkage / soft thresholding).
Softshrink is soft shrinkage (soft thresholding) that zeros out small magnitude activations and shrinks larger ones. It applies Softshrink(x) = (x - λ) if x > λ, (x + λ) if x < -λ, 0 otherwise. Unlike hard thresholding (Hardshrink) which preserves large values unchanged, soft shrinkage reduces their magnitude. Softshrink is rarely used in standard deep learning but appears in sparse coding, denoising, and specific signal processing architectures.
Core idea: Softshrink(x) applies a soft threshold at ±λ. For |x| ≤ λ, output is zero. For |x| > λ, output is x - λ*sign(x) (shrinks toward zero). This creates continuous sparse representation compared to Hardshrink's binary sparsity.
When to use Softshrink:
- Sparse representation learning with smooth shrinkage
- Denoising autoencoders (soft shrinkage for noise removal)
- Iterative shrinkage algorithms
- Sparse coding and dictionary learning
- Rarely: standard deep networks don't use shrinkage functions
Trade-offs vs Hardshrink:
- Sparsity: Softshrink shrinks large values (continuous) vs Hardshrink preserves them (binary)
- Smoothness: Softshrink has no gradient discontinuity (smooth) vs Hardshrink's kink
- Interpretation: Softshrink continuous shrinkage vs Hardshrink's binary selection
- Effectiveness: Similar for sparse coding; choice depends on task
- Gradient: Softshrink has gradient everywhere (smooth) vs Hardshrink's sparse gradient
Trade-offs vs Softplus:
- Range: Softshrink zeros small values, shrinks large; Softplus is bounded smooth
- Sparsity: Softshrink explicitly sparse; Softplus non-sparse smooth approximation
- Use case: Softshrink for sparse coding; Softplus for general smooth activation
Algorithm: Forward: Softshrink(x) = x - λ if x > λ, x + λ if x < -λ, 0 if |x| ≤ λ Backward: ∂/∂x = 1 if |x| > λ, 0 if |x| ≤ λ (like Hardshrink) The soft shrinkage creates a dead zone [−λ, λ] where output is exactly zero.
Definition
export interface SoftshrinkOptions {
/** Threshold value for shrinkage (default: 0.5) */
lambd?: number;
}lambd(number)optional- – Threshold value for shrinkage (default: 0.5)
Examples
// Soft shrinkage for sparse coding
class SparseCodeAutoencoder extends torch.nn.Module {
private encode: torch.nn.Linear;
private softshrink: torch.nn.Softshrink;
private decode: torch.nn.Linear;
constructor() {
super();
this.encode = new torch.nn.Linear(100, 50);
this.softshrink = new torch.nn.Softshrink(0.1); // λ = 0.1
this.decode = new torch.nn.Linear(50, 100);
}
forward(x: torch.Tensor): torch.Tensor {
x = this.encode.forward(x);
x = this.softshrink.forward(x); // Soft threshold for sparse codes
return this.decode.forward(x);
}
}// Comparing hard vs soft shrinkage
const x = torch.linspace(-2, 2, [1000]);
const hardshrink = new torch.nn.Hardshrink(0.5);
const softshrink = new torch.nn.Softshrink(0.5);
const y_hard = hardshrink.forward(x); // Exactly zero in [-0.5, 0.5], unchanged outside
const y_soft = softshrink.forward(x); // Exactly zero in [-0.5, 0.5], shrunk outside
// Hard: binary selection, Soft: continuous shrinkage