Contextual Embedding

A contextual embedding is a representation whose meaning changes based on surrounding tokens.

~/writing/math-foundations-for-reasoning-models $ cat topic.md

Definition

A contextual embedding is a representation of a token after context has been mixed in.

Simple form:

token embedding + surrounding information -> contextual embedding

The same token can have different contextual embeddings in different sequences.

A contextual embedding represents a token in context.

The same token can be used in different ways.

Example:

"bank" in "river bank"
"bank" in "bank account"

The token may start with the same base embedding.

After attention and layer transformations, its representation can become different.

Context changes the representation of the token.

Base token embedding:

same starting representation for the token ID

Contextual embedding:

updated representation after sequence information is mixed in

This distinction is important.

The base embedding identifies a token's starting representation.

The contextual embedding reflects the token's role in the current sequence.

Base embedding is context-free; contextual embedding is sequence-dependent.

In transformer models, attention helps mix context.

Simple flow:

token embeddings
  -> attention
  -> feedforward transformations
  -> contextual representations

Each layer can update the representation based on other positions.

Contextual embeddings are produced by representation transformations across layers.

Contextual embeddings let a model represent flexible use.

The same token does not have to mean exactly one thing everywhere.

Example:

same token
different surroundings
different representation

Contextual representation is one reason modern models handle language better than fixed word embeddings.

~/writing/math-foundations-for-reasoning-models $ open

Embedding Space