~/writing/math-foundations-for-reasoning-models $ ls
Information Theory and Loss
Learning as Reducing Uncertainty
Learning can be viewed as using data to reduce uncertainty about predictions, parameters, and patterns.
~/writing/math-foundations-for-reasoning-models $ cat topic.md
Learning As Reducing Uncertainty
Definition
Learning can be viewed as using data to reduce uncertainty.
Uncertainty about what?
which label is correct
which pattern matters
which parameters fit
which output should be produced
Learning uses information to reduce uncertainty.
Before Learning
Before learning, the system may be uncertain.
Example:
bug: 0.25
billing: 0.25
feature: 0.25
question: 0.25
The distribution is spread out.
The model does not strongly prefer one answer.
High uncertainty means many outcomes remain plausible.
After Learning
After seeing useful data, the model may assign probability more sharply.
Example:
bug: 0.85
billing: 0.05
feature: 0.05
question: 0.05
If this aligns with reality, uncertainty has been reduced in a useful way.
Learning should reduce uncertainty in the right direction.
Data As Information
Data is useful when it changes uncertainty.
Example:
message contains stack trace
This may increase probability of:
bug report
The data helps distinguish one outcome from alternatives.
Useful data carries information about the target.
Reducing Uncertainty Is Not Always Good
A model can become confidently wrong.
Example:
true label: billing
model: bug 0.99
Uncertainty is low, but the prediction is bad.
So learning is not merely about reducing entropy.
It is about reducing uncertainty while matching reality.
Confidence must be aligned with truth.
Loss Guides The Reduction
Loss tells the model whether its uncertainty was useful.
If the model assigns high probability to the true outcome:
low loss
If it assigns low probability:
high loss
Loss distinguishes useful confidence from wrong confidence.
Learning As Distribution Shaping
A model can be understood as transforming inputs into output distributions.
Learning changes parameters so those distributions improve.
Basic idea:
input -> model -> predicted distribution
Training tries to make the predicted distribution place probability where reality places it.
Learning shapes probability distributions toward better alignment.
Final Summary
- Learning can be viewed as reducing uncertainty.
- Data is useful when it carries information about the target.
- Lower uncertainty alone is not enough.
- The model must become confident in the right outcomes.
- Loss guides whether uncertainty reduction is useful.
- Learning can be seen as shaping output distributions.
~/writing/math-foundations-for-reasoning-models $ open
Continue the series.
Bridge to Optimization