~/writing/math-foundations-for-reasoning-models $ ls
Geometry of Data
Normalization
Normalization rescales values so they can be compared more consistently.
~/writing/math-foundations-for-reasoning-models $ cat topic.md
Normalization
Definition
Normalization is the process of rescaling values into a more consistent form.
For vectors, a common form is to rescale a vector so its length becomes 1.
Example:
v = [3, 4]
Its length is:
5
Normalized vector:
[3 / 5, 4 / 5] = [0.6, 0.8]
Normalization changes scale while preserving structure.
Why Normalize
Raw values can have different sizes.
Example:
rooms = 3
size = 1200
The size component is much larger than the rooms component.
If we compare raw numbers directly, larger-scale components can dominate.
Normalization helps make comparison less controlled by raw scale.
Vector Length Normalization
To normalize a vector by length:
- Measure its norm.
- Divide each component by that norm.
Example:
v = [3, 4]
norm(v) = 5
Divide each component:
[3 / 5, 4 / 5] = [0.6, 0.8]
The new vector has length 1.
Length normalization turns a vector into a unit-length vector.
What Normalization Preserves
Length normalization preserves direction.
Example:
[3, 4]
[0.6, 0.8]
These point in the same direction.
The second vector is just a smaller version of the first.
Normalization can preserve direction while changing magnitude.
What Normalization Changes
Normalization changes size.
Example:
[3, 4] has length 5
[0.6, 0.8] has length 1
So after normalization, magnitude no longer carries the original scale.
That can be useful or harmful depending on the question.
Normalization removes some scale information.
Zero Vector Problem
The zero vector cannot be length-normalized.
Example:
v = [0, 0]
norm(v) = 0
To normalize by length, we would divide by 0.
That is undefined.
You cannot create a unit direction from a vector with zero length.
Normalization In Data
Normalization is useful when comparing patterns instead of raw size.
Example:
A = [3, 4]
B = [30, 40]
These have the same direction but different size.
After length normalization:
A -> [0.6, 0.8]
B -> [0.6, 0.8]
They become identical in direction.
Normalization can make direction more important than magnitude.
Final Summary
- Normalization rescales values.
- Vector length normalization divides by the vector norm.
- It can turn a vector into length
1. - It can preserve direction while changing magnitude.
- It removes some scale information.
- The zero vector cannot be length-normalized.
~/writing/math-foundations-for-reasoning-models $ open
Continue the series.
Unit Vector