~/writing/math-foundations-for-reasoning-models $ ls
Embeddings and Representation
Representation as Compression
Representation as compression means keeping task-useful information while discarding details that are less useful.
~/writing/math-foundations-for-reasoning-models $ cat topic.md
Representation As Compression
Definition
Representation as compression means encoding information in a smaller or more useful form.
Simple idea:
raw detail -> compact useful representation
In embeddings, the model cannot store every possible fact directly.
It learns a representation that preserves patterns useful for computation.
A representation compresses information into a usable form.
Compression Does Not Mean File Compression
This is not the same as a zip file.
A zip file tries to preserve the original data exactly.
Model representation is usually lossy.
That means some details are kept, and others are not.
kept:
task-useful patterns
lost:
details not preserved by the representation
Neural representation is useful compression, not exact reconstruction.
Why Compression Is Needed
The model needs internal states that are manageable.
It cannot treat every input as an unrelated isolated object.
Embeddings compress identity and usage patterns into a vector.
This lets the model reuse structure.
Compression helps turn many discrete cases into shared representation patterns.
What Gets Preserved
Training decides what the representation tends to preserve.
Examples:
patterns that reduce loss
relationships useful for prediction
features useful for downstream transformation
What is preserved depends on the data, model, and objective.
Representation preserves what training makes useful.
Compression And Generalization
Compression supports generalization because similar cases can share representation structure.
Example:
new input resembles learned pattern
model uses shared representation structure
The model does not need a separate rule for every possible input.
Generalization depends on reusable compressed structure.
Final Summary
- Representation can be understood as useful compression.
- It keeps some information and loses other information.
- It is not exact file compression.
- Training shapes what gets preserved.
- Compression helps models reuse patterns and generalize.
~/writing/math-foundations-for-reasoning-models $ open
Continue the series.
Bridge to Generalization