Non-Convex Optimization

Non-convex optimization deals with loss landscapes that can contain many valleys, flat regions, and unstable points.

~/writing/math-foundations-for-reasoning-models $ cat topic.md

Definition

Non-convex optimization means optimizing an objective whose loss landscape is not shaped like one simple bowl.

A simple bowl has one clear direction toward the lowest point.

A non-convex landscape can have:

many valleys
many hills
flat regions
local minima
saddle-like regions

Non-convex optimization is optimization in a complicated loss landscape.

The shape of the loss landscape affects how optimization behaves.

In a simple landscape:

move downhill -> keep approaching the best point

In a non-convex landscape:

move downhill -> may reach a local minimum
move downhill -> may enter a flat region
move downhill -> may not reveal the best overall direction

The optimizer only sees local change, but the landscape may be globally complicated.

Gradient descent uses nearby information.

It asks:

how does loss change around the current parameters?

That is useful, but it does not show the whole landscape.

Example:

current region looks low nearby
another faraway region is much lower

The optimizer may not know about the faraway region.

Local improvement does not guarantee global best behavior.

Non-convex optimization is harder to reason about, but it can still work.

The optimizer may still find useful low-loss regions.

The goal is often practical:

find parameters that work well enough

not always:

prove the absolute lowest possible loss

Non-convex optimization can be useful even when global guarantees are difficult.

Learning systems often have many parameters.

Many interacting parameters can create complicated landscapes.

This means training is not simply:

follow one straight path to the best answer

It is closer to:

use local signals to search a large landscape

Learning often depends on navigating a complex parameter space.

Non-convex optimization deals with complicated loss landscapes.
These landscapes can contain many valleys, hills, and flat regions.
Gradient descent uses local information, not the whole landscape.
Local improvement does not guarantee a global minimum.
Non-convex does not mean useless or impossible.
Learning often works by finding useful low-loss regions, not by proving perfect global optimality.

~/writing/math-foundations-for-reasoning-models $ open

Overfitting