Scaling Laws, in the Arithmetic You Will Actually Use

Scaling Laws, in the Arithmetic You Will Actually Use

"We have a budget. Should we train a bigger model or train longer?" is an arithmetic question, and the ability to do that arithmetic out loud separates candidates who have read about scaling from candidates who have used it.

Start with the compute identity

Training compute is approximately six times parameters times tokens. Six, because each token costs roughly two FLOPs per parameter forward and four backward. That single relation lets you sanity-check any published run in about thirty seconds, and interviewers notice when you do.

Compute-optimal is a ratio, not a size

The Chinchilla result is that for a fixed compute budget, parameters and tokens should scale together (roughly twenty tokens per parameter) and that the field had been training models that were too large on too little data. The number matters less than the shape of the claim.

Compute-optimal is usually not what you want

Chinchilla optimizes training cost. You are probably paying for inference for years afterwards, and a smaller model trained past its compute-optimal point is often the better economic choice by a wide margin. Saying this unprompted is a strong signal in a design interview.

Where the picture is contested

  • Data quality and deduplication move the curves enough to change conclusions
  • Mixture-of-experts models break the simple parameter count
  • Repeated epochs on limited data do not behave like fresh tokens
  • Downstream task performance does not track loss as tightly as one would like

Knowing the disagreements is what distinguishes someone who has followed the literature from someone who has read a summary of it.

Keep reading

The ML Interview Loop, Round by Round
The ML Interview Loop, Round by Round

Five rounds, five different things being graded. Most candidates prepare for two of them and are surprised by the rest.

Your RAG System Has a Retrieval Problem
Your RAG System Has a Retrieval Problem

When the answer is wrong, teams reach for a better model. The document usually was never retrieved in the first place.

Read the Loss Curve: A Field Guide to Failed Runs
Read the Loss Curve: A Field Guide to Failed Runs

Loss flat, loss spiking, training down and validation up. Each shape has a short list of causes, and guessing is not one of the steps.