torch.leaky_relu
function leaky_relu<S extends Shape, D extends DType = DType, Dev extends DeviceType = DeviceType>(input: Tensor<S, D, Dev>, negative_slope?: number): Tensor<S, D, Dev>Applies the Leaky ReLU function element-wise.
A ReLU variant that allows a small negative slope instead of hard zero, addressing the "dying ReLU" problem where neurons can become inactive. Popular in GANs, adversarial training, and models where keeping negative information is important.
- Dying ReLU fix: Allows gradients to flow even for negative inputs
- Typical slopes: 0.01 (default, minimal), 0.1-0.3 (GANs), varies by application
- Gradient: Always non-zero, never "dies" during training
- Trade-off: More computation than ReLU, but enables richer representations
Parameters
inputTensor<S, D, Dev>- The input tensor
negative_slopenumberoptional- The slope for negative values (default: 0.01). Typical range: 0.01-0.3
Returns
Tensor<S, D, Dev>– A new tensor with Leaky ReLU applied element-wiseExamples
// Basic usage with default slope
const x = torch.tensor([-2, -1, 0, 1, 2]);
torch.leaky_relu(x); // [-0.02, -0.01, 0, 1, 2]
// With larger slope for GANs (more information preserved)
torch.leaky_relu(x, 0.2); // [-0.4, -0.2, 0, 1, 2]
// In discriminator network
class Discriminator extends torch.nn.Module {
forward(x: Tensor): Tensor {
x = torch.leaky_relu(this.conv1(x), 0.2);
x = torch.leaky_relu(this.conv2(x), 0.2);
return this.dense(x);
}
}
// Comparison with ReLU
const relu_out = torch.relu(x); // [0, 0, 0, 1, 2]
const leaky_out = torch.leaky_relu(x, 0.1); // [-0.2, -0.1, 0, 1, 2]
// Leaky allows information to flow through negative branchSee Also
- PyTorch torch.nn.functional.leaky_relu()
- relu - Simpler variant without negative slope
- elu - Smoother alternative with exponential negative region
- prelu - PReLU learns the slope per channel