Underfitting

Underfitting happens when a model fails to capture enough structure even in the training examples.

~/writing/math-foundations-for-reasoning-models $ cat topic.md

Definition

Underfitting happens when a model fails to capture enough of the pattern.

It performs poorly even on the training examples.

Example:

training examples -> high loss
new examples -> high error

Underfitting means the model is not learning enough structure.

Underfitting can happen for several reasons.

model is too simple
features are not useful
optimization has not run long enough
learning rate is poorly chosen
objective does not match the task

The shared problem is that the system does not reach a useful representation of the pattern.

Underfitting means the current setup cannot fit the needed structure well enough.

Suppose the true pattern is curved.

But the model can only represent a straight line.

true pattern: curved
model form: straight line

No parameter setting can fit the pattern well.

The model underfits because its available transformations are too limited.

A model can underfit when its structure is too weak for the pattern.

Underfitting is not always caused by bad optimization.

Sometimes the optimizer cannot find good parameters because good parameters do not exist inside the current parameter space.

Example:

best possible setting under current model -> still high loss

In other cases, the model could fit better, but optimization has not reached a good region yet.

Underfitting can come from model limits, optimization limits, or both.

Underfitting and overfitting are different failures.

underfitting:
bad on training examples
bad on new examples

overfitting:
good on training examples
bad on new examples

Underfitting means too little useful structure was captured.

Overfitting means too much training-specific detail was captured.

Underfitting is not enough fit; overfitting is too narrow a fit.

Underfitting means the model fails to capture enough structure.
It usually produces high training loss.
It can happen when the model is too simple.
It can also happen when optimization is poor or incomplete.
Underfitting differs from overfitting.
The model needs enough capacity and optimization progress to capture the useful pattern.

~/writing/math-foundations-for-reasoning-models $ open

Regularization