~/writing/math-foundations-for-reasoning-models $ ls
Attention and Transformers
Long-Range Dependency
A long-range dependency is a relationship between distant positions that must be connected for the representation to work.
~/writing/math-foundations-for-reasoning-models $ cat topic.md
Long-Range Dependency
Definition
A long-range dependency is a relationship between positions that are far apart in a sequence.
Simple form:
early position influences later position
even though many positions are between them
The dependency matters because the later representation needs information from the earlier one.
A long-range dependency is a meaningful relation across distance in a sequence.
Why Distance Can Be A Problem
Some architectures move information through short paths.
Example:
step 1 -> step 2 -> step 3 -> step 4
If information must travel through many steps, it can become harder to preserve or use.
This was one motivation for attention.
Long-range dependencies are hard when information must pass through many intermediate transformations.
Attention And Direct Interaction
Self-attention lets one position directly use information from another position.
Example:
position 20 can attend to position 2
It does not need to move only through every intermediate position.
This makes long-distance interaction structurally easier.
Attention shortens the path between distant positions.
Long-Range Does Not Mean Always Important
Distance alone does not decide relevance.
A far position may matter a lot.
A nearby position may matter little.
Attention allows the model to learn this distinction.
near does not always mean relevant
far does not always mean irrelevant
Attention separates relevance from physical sequence distance.
Example
Consider a sentence where an early subject affects a later verb.
The model may need to connect:
early noun phrase -> later verb form
If those positions are far apart, the dependency is long-range.
Attention gives the later position a way to directly receive signal from the earlier position.
Long-range dependencies require information to remain usable across distance.
Final Summary
- A long-range dependency connects distant positions.
- Distance can make information harder to preserve in step-by-step architectures.
- Attention allows more direct interaction between distant positions.
- Relevance is not the same as closeness.
- Attention helps models use faraway information when it matters.
~/writing/math-foundations-for-reasoning-models $ open
Continue the series.
Attention as Learned Alignment