Generalization

Generalization is the ability of a learned system to work on new examples, not only the examples used for training.

~/writing/math-foundations-for-reasoning-models $ cat topic.md

Definition

Generalization is the ability to perform well on new examples.

The model does not just repeat the training examples.

It captures structure that applies beyond them.

Example:

training examples -> low loss
new examples -> also good behavior

Generalization means learned behavior transfers.

Training loss measures performance on examples used during training.

But the real test is often new cases.

Example:

training loss: low
new-example error: high

This means the model optimized the training objective but did not learn the right reusable structure.

Low training loss is useful only if it leads to good behavior beyond training.

Memorization reproduces seen examples.

Generalization handles unseen examples.

memorization:
works because the example was already seen

generalization:
works because the underlying pattern was learned

The difference matters because most useful systems are expected to respond to new inputs.

Generalization is pattern transfer, not example recall.

Optimization directly reduces an objective.

Generalization asks whether reducing that objective produced useful behavior.

Example:

objective: reduce training loss
desired result: low loss on new examples too

If optimization reduces training loss but harms new-example behavior, the optimization target is incomplete or too narrow.

Generalization connects optimization to real usefulness.

Regularization helps generalization by shaping the solution.

It discourages parameter settings that fit training details too specifically.

loss -> fit the data
regularization -> prefer less brittle solutions

This can make the learned pattern more likely to transfer.

Regularization is one tool for making optimization support generalization.

Generalization means working on new examples.
Training loss alone does not guarantee generalization.
Memorization repeats seen examples.
Generalization captures reusable structure.
Optimization reduces an objective, but generalization measures broader usefulness.
Regularization can help optimization find solutions that transfer better.

~/writing/math-foundations-for-reasoning-models $ open

Bridge to Neural Networks