~/writing/math-foundations-for-reasoning-models $ ls
Deep Representations
Skip Connections
Skip connections let information bypass one or more layers, giving deeper networks shorter paths for signals and gradients.
~/writing/math-foundations-for-reasoning-models $ cat topic.md
Skip Connections
Definition
A skip connection sends information around one or more layers.
Instead of only:
input -> layer -> layer -> output
a skip connection allows:
input -> layer -> layer -> output
input -----------------------> output
A skip connection creates a shortcut path through the network.
Why Shortcuts Help
In deep networks, signals and gradients must travel through many layers.
Long paths can make training harder.
Skip connections create shorter paths.
long path:
through many layers
skip path:
around some layers
This can help information and gradients move more reliably.
Skip connections improve signal flow in deep networks.
Not Skipping Learning
A skip connection does not mean the skipped layers are useless.
The network can still learn transformations inside those layers.
The shortcut simply gives another route for information.
transformed path + shortcut path
The model can use both.
Skip connections add paths; they do not remove learning.
Relation To Depth
Depth gives more transformation stages.
But more stages can make training difficult.
Skip connections help deeper networks remain trainable by reducing the difficulty of moving information across many layers.
Skip connections make depth easier to use.
Final Summary
- A skip connection creates a shortcut around layers.
- It gives information another path through the network.
- It can help gradients flow backward.
- It does not make skipped layers useless.
- It supports training deeper networks.
- Skip connections are a structural tool for stable deep learning.
~/writing/math-foundations-for-reasoning-models $ open
Continue the series.
Residual Learning