Dropout

Definition

Dropout is a regularization method that randomly disables some activations during training.

Simple form:

during training:
some units are temporarily ignored

The model must learn to work without relying on every pathway being present.

Dropout discourages fragile dependence on specific units.

---

Why Dropout Helps

Without dropout, a model may depend too strongly on particular internal pathways.

Dropout adds noise during training.

This forces the network to distribute useful information more broadly.

pathway missing temporarily -> model must still learn useful behavior

Dropout encourages redundancy and robustness.

---

Training vs Inference

Dropout is usually active during training.

It is usually turned off during inference.

training:
randomly drop activations

inference:
use the full network

The training process teaches the model not to depend too narrowly on one route.

Dropout changes training behavior, not normal inference behavior.

---

Dropout As Regularization

Dropout makes fitting the training data harder in a controlled way.

That can reduce overfitting.

Instead of memorizing with fragile internal pathways, the model is pushed toward more stable patterns.

Dropout regularizes by injecting controlled uncertainty during training.

---

Tradeoff

Too little dropout may have little effect.

Too much dropout may prevent the model from learning enough.

moderate dropout -> better robustness
excessive dropout -> underfitting

The dropout rate is a design choice.

Dropout must be strong enough to regularize but not so strong that it destroys signal.

---

Final Summary

• Dropout randomly disables activations during training.
• It discourages dependence on specific internal pathways.
• It is usually turned off during inference.
• It can reduce overfitting.
• Too much dropout can cause underfitting.