Model Capacity

Definition

Model capacity is the ability of a model to represent different patterns.

Simple idea:

more capacity -> can represent more complex functions
less capacity -> can represent fewer patterns

Capacity depends on model structure and number of parameters.

Model capacity is representational ability.

---

Why Capacity Matters

A model needs enough capacity to capture the task pattern.

If capacity is too low, the model may underfit.

If capacity is very high, the model may overfit if not controlled.

too little capacity -> weak fit
too much uncontrolled capacity -> training-specific fit

Capacity affects both fitting and generalization.

---

Capacity Is Not Automatically Bad

High capacity does not automatically mean poor generalization.

Modern models can be very large and still generalize well when trained with enough data, useful objectives, and proper methods.

The issue is not size alone.

The issue is how capacity is shaped by:

data
architecture
regularization
optimization
objective

Capacity needs guidance, not just reduction.

---

Capacity And Function Space

A model with higher capacity can represent more possible input-output mappings.

Simple view:

model class = set of functions the model can express
capacity = richness of that set

This connects model capacity to earlier topics about functions and transformations.

Capacity describes the range of transformations a model can learn.

---

Practical Signal

A practical way to reason about capacity:

low training performance -> maybe too little capacity or poor optimization
high training performance but low validation performance -> maybe overfitting

Capacity is interpreted through training and validation behavior.

Capacity should be judged by how it affects fit and generalization.

---

Final Summary

• Model capacity is representational ability.
• Low capacity can cause underfitting.
• High capacity can overfit if poorly constrained.
• High capacity is not automatically bad.
• Capacity is shaped by data, architecture, objective, and regularization.