torch.nn.Softshrink

class Softshrink extends Module

new Softshrink(options?: SoftshrinkOptions)

readonlylambd(number)

Softshrink activation function (Soft shrinkage / soft thresholding).

Softshrink is soft shrinkage (soft thresholding) that zeros out small magnitude activations and shrinks larger ones. It applies Softshrink(x) = (x - λ) if x > λ, (x + λ) if x < -λ, 0 otherwise. Unlike hard thresholding (Hardshrink) which preserves large values unchanged, soft shrinkage reduces their magnitude. Softshrink is rarely used in standard deep learning but appears in sparse coding, denoising, and specific signal processing architectures.

Core idea: Softshrink(x) applies a soft threshold at ±λ. For |x| ≤ λ, output is zero. For |x| > λ, output is x - λ*sign(x) (shrinks toward zero). This creates continuous sparse representation compared to Hardshrink's binary sparsity.

When to use Softshrink:

Sparse representation learning with smooth shrinkage
Denoising autoencoders (soft shrinkage for noise removal)
Iterative shrinkage algorithms
Sparse coding and dictionary learning
Rarely: standard deep networks don't use shrinkage functions

Trade-offs vs Hardshrink:

Sparsity: Softshrink shrinks large values (continuous) vs Hardshrink preserves them (binary)
Smoothness: Softshrink has no gradient discontinuity (smooth) vs Hardshrink's kink
Interpretation: Softshrink continuous shrinkage vs Hardshrink's binary selection
Effectiveness: Similar for sparse coding; choice depends on task
Gradient: Softshrink has gradient everywhere (smooth) vs Hardshrink's sparse gradient

Trade-offs vs Softplus:

Range: Softshrink zeros small values, shrinks large; Softplus is bounded smooth
Sparsity: Softshrink explicitly sparse; Softplus non-sparse smooth approximation
Use case: Softshrink for sparse coding; Softplus for general smooth activation

Algorithm: Forward: Softshrink(x) = x - λ if x > λ, x + λ if x < -λ, 0 if |x| ≤ λ Backward: ∂/∂x = 1 if |x| > λ, 0 if |x| ≤ λ (like Hardshrink) The soft shrinkage creates a dead zone [−λ, λ] where output is exactly zero.

\begin{aligned} Softshrink(x) = {x - λ if x > λ, 0 if |x| ≤ λ, x + λ if x < -λ} \\ where λ > 0 is the shrinkage parameter (default: 0.5) \\ Equivalent to: sign(x) * max(|x| - λ, 0) \end{aligned}

Soft shrinkage: Shrinks large values by λ, creating continuous sparse representation.
Dead zone: All values in [-λ, λ] become exactly zero (threshold).
Smooth gradients: Gradient is 1 or 0, no discontinuity (unlike Hardshrink's kink).
Sign preservation: Preserves sign of input while shrinking magnitude.
Lambda parameter: Controls dead zone width; larger λ → more zeros, smaller λ → less sparse.
Rarely used: Standard activations better for general deep learning.

Examples

// Soft shrinkage for sparse coding
class SparseCodeAutoencoder extends torch.nn.Module {
  private encode: torch.nn.Linear;
  private softshrink: torch.nn.Softshrink;
  private decode: torch.nn.Linear;

  constructor() {
    super();
    this.encode = new torch.nn.Linear(100, 50);
    this.softshrink = new torch.nn.Softshrink(0.1);  // λ = 0.1
    this.decode = new torch.nn.Linear(50, 100);
  }

  forward(x: torch.Tensor): torch.Tensor {
    x = this.encode.forward(x);
    x = this.softshrink.forward(x);  // Soft threshold for sparse codes
    return this.decode.forward(x);
  }
}

// Comparing hard vs soft shrinkage
const x = torch.linspace(-2, 2, [1000]);
const hardshrink = new torch.nn.Hardshrink(0.5);
const softshrink = new torch.nn.Softshrink(0.5);

const y_hard = hardshrink.forward(x);  // Exactly zero in [-0.5, 0.5], unchanged outside
const y_soft = softshrink.forward(x);  // Exactly zero in [-0.5, 0.5], shrunk outside
// Hard: binary selection, Soft: continuous shrinkage

torch.nn.Softshrink

Examples

See Also

torch.nn.Softshrink

Examples

See Also