AI Systems · Core
Learning Rate Schedule
A learning-rate schedule is the planned path of η_t — warmup, cosine, step decay — that often matters more than the architecture headline on a given run.
Definition
Too high η diverges; too low wastes compute. Warmup avoids early large steps on a random net. Cosine or linear decay anneals to a small η so you settle. One-cycle and cosine-with-restarts are variants. AdamW still needs a schedule.
Why it matters
Most failed trains are schedule bugs labeled as ‘the model is bad.’
Case
LLM pretrains with 2k warmup + cosine to 10% of peak vs a constant 3e-4: the constant either spikes or plateaus early.
How to read it
Plot η, grad norm, and loss on one clock. If loss is flat and η is already tiny, you annealed too soon.
No query has been run yet. Which is tragically normal for most knowledge systems, but we are trying to evolve past decorative databases.