torch.nn.FeatureAlphaDropout
class FeatureAlphaDropout extends Modulenew FeatureAlphaDropout(options?: DropoutOptions)
- readonly
p(number)
FeatureAlphaDropout: channel-wise alpha dropout for 2D/3D SNNs.
A variant of AlphaDropout designed for 2D and 3D convolutional self-normalizing neural networks. Instead of dropping individual elements, FeatureAlphaDropout drops entire feature maps (channels) while preserving mean and variance. This is appropriate for CNNs using SELU, providing both spatial coherence (like Dropout2d) and self-normalizing properties (like AlphaDropout). Essential for:
- 2D/3D convolutional SNNs with SELU
- Preserving feature map coherence in self-normalizing nets
- Maintaining self-normalization through conv layers
- Deep convolutional SNNs without batch normalization
- Image/video processing with self-normalizing properties
FeatureAlphaDropout combines the spatial structure preservation of channel dropout (Dropout1d/2d/3d) with the mean/variance preservation of alpha dropout (AlphaDropout). This makes it ideal for convolutional layers in SELU-based networks.
When to use FeatureAlphaDropout:
- 2D convolutional SNNs (images)
- 3D convolutional SNNs (volumes, medical imaging)
- Networks using SELU activation with convolutions
- Deep CNN architectures without batch normalization
- Preserving feature channel structure in self-normalizing networks
Comparison of dropout variants:
- Dropout: Element-wise, breaks structure and normalization
- Dropout1d/2d/3d: Channel-wise, preserves structure but not normalization
- AlphaDropout: Element-wise, preserves normalization but not structure
- FeatureAlphaDropout: Channel-wise, preserves BOTH structure and normalization
Trade-offs:
- vs Dropout2d + BatchNorm: Similar effect, but doesn't need batch statistics
- vs AlphaDropout: FeatureAlphaDropout preserves spatial structure within channels
- vs standard Dropout2d: Preserves mean/variance for SELU networks
- Computational cost: Slightly higher than standard Dropout2d
- Applicability: Most effective with SELU activation
Input shape expectations:
- Works with feature map outputs: (batch, channels, ...)
- Automatically detects dimensionality (2D, 3D, or higher)
FeatureAlphaDropout mechanics: For input with shape (N, C, ...) where C is channels:
- Create channel mask M ~ Bernoulli(1-p) of shape (N, C, 1, 1, ...)
- For kept channels: y[:, c, ...] = x[:, c, ...] (unchanged)
- For dropped channels: Replace with values maintaining mean/variance
- Result: Entire feature maps replaced, spatial structure intact, E[y]=E[x], Var[y]=Var[x]
- Channel-wise: Entire feature maps dropped together, preserving spatial structure
- Mean preservation: Output mean equals input mean (E[y] = E[x])
- Variance preservation: Output variance equals input variance (Var[y] = Var[x])
- Self-normalizing: Maintains self-normalizing properties through conv layers
- Dimensionality agnostic: Works with 2D, 3D, or higher dimensional features
- SELU compatible: Best used with SELU activation function
- SELU activation: Most effective with SELU, less useful with ReLU/others
- Training/inference: Must call .train()/.eval() to control behavior
- Initialization critical: SELU SNNs require proper weight initialization
- Different masks per batch: Each batch item gets independent channel mask
Examples
// Convolutional SNN with FeatureAlphaDropout
class ConvSNN extends torch.nn.Module {
conv1: torch.nn.Conv2d;
fad1: torch.nn.FeatureAlphaDropout;
conv2: torch.nn.Conv2d;
fad2: torch.nn.FeatureAlphaDropout;
fc: torch.nn.Linear;
constructor() {
super();
this.conv1 = new torch.nn.Conv2d(1, 64, 3, { padding: 1 });
this.fad1 = new torch.nn.FeatureAlphaDropout(0.1);
this.conv2 = new torch.nn.Conv2d(64, 128, 3, { padding: 1 });
this.fad2 = new torch.nn.FeatureAlphaDropout(0.1);
this.fc = new torch.nn.Linear(128 * 28 * 28, 10);
}
forward(x: torch.Tensor): torch.Tensor {
// SELU + FeatureAlphaDropout preserves self-normalization
x = torch.selu(this.conv1.forward(x));
x = this.fad1.forward(x);
x = torch.selu(this.conv2.forward(x));
x = this.fad2.forward(x);
x = x.view([x.shape[0], -1]);
return this.fc.forward(x);
}
}// Medical imaging SNN with 3D convolutions
class MedicalSNN extends torch.nn.Module {
conv3d: torch.nn.Conv3d;
fad: torch.nn.FeatureAlphaDropout;
constructor() {
super();
this.conv3d = new torch.nn.Conv3d(1, 64, 3, { padding: 1 });
this.fad = new torch.nn.FeatureAlphaDropout(0.1);
}
forward(x: torch.Tensor): torch.Tensor {
x = torch.selu(this.conv3d.forward(x));
x = this.fad.forward(x); // Preserves 3D structure and normalization
return x;
}
}// Direct dropout comparison
const feature_alpha = new torch.nn.FeatureAlphaDropout(0.1);
const feature_regular = new torch.nn.Dropout2d(0.1);
const x = torch.randn([32, 64, 224, 224]);
// Both preserve spatial structure, but FeatureAlphaDropout also preserves normalization
feature_alpha.train();
const out_alpha = feature_alpha.forward(x); // E[out] = E[x], Var[out] = Var[x]
feature_regular.train();
const out_regular = feature_regular.forward(x); // E[out] ≠ E[x] (affected by dropout)