How Models Are Trained

A transformer fresh off the assembly line has hundreds of billions of parameters set to random noise. Ask it anything and you get gibberish. Everything that makes a model useful — its knowledge, its tone, its ability to follow instructions and refuse harmful requests — comes from training.

Modern training happens in three big stages, and they answer three different questions:

1. Pretraining — learn the world.
2. Fine-tuning — learn the job.
3. Alignment (RLHF and friends) — learn good judgment.

Let's walk through each, because the differences between them explain a lot about how models behave.

Stage 1: Pretraining — learning to predict everything

Pretraining is where the model reads a huge slice of human writing and learns the one skill from post #2: predict the next token. The setup is elegantly simple, and it needs no human labelers, because the text labels itself:

   Take a sentence:   "The mitochondria is the powerhouse of the cell."
   Hide the next token at each point and ask the model to guess it:

       "The"                         → predict "mitochondria"
       "The mitochondria"            → predict "is"
       "The mitochondria is the"     → predict "powerhouse"
       ...

For every guess, we compare the model's prediction to the actual next token, measure how wrong it was, and nudge all those billions of parameters a tiny bit to make the right answer more likely next time. That nudging mechanism is called gradient descent, and "backpropagation" is the bookkeeping that figures out which parameters to nudge and in which direction.

Do this trillions of times, across a meaningful fraction of the public internet, books, and code, and the patterns of language — and with them, a rough model of the world — get compressed into the weights.

A few things to sit with:

• This stage is astronomically expensive. It's months of computation on tens of thousands of specialized chips, costing tens or hundreds of millions of dollars. This is the part only a handful of organizations can afford, and it's where most of the model's raw capability comes from.
• The result is a "base model." It's brilliant at completing text but socially clueless. Ask a base model "What is the capital of France?" and it might reply with more questions — because in its training data, questions are often followed by lists of similar questions, like a quiz. It has knowledge but no sense that it's supposed to help you.
• The knowledge cutoff lives here. A base model only knows what was in its training data, frozen at the moment training stopped. Anything after that — last week's news, your private documents — is invisible unless you feed it in later (post #14).

Stage 2: Fine-tuning — teaching it the job

To turn that knowledgeable-but-clueless base model into a helpful assistant, we continue training it on a much smaller, carefully curated set of examples that demonstrate the behavior we want. This is supervised fine-tuning (SFT).

The data here looks like conversations:

   User:      What is the capital of France?
   Assistant: The capital of France is Paris.

   User:      Write a haiku about debugging.
   Assistant: Code refuses to run /
              one missing semicolon /
              three hours of my life.

Tens of thousands to millions of these high-quality examples, often written or vetted by humans, teach the model the format and posture of being an assistant: when someone asks a question, answer it; follow instructions; produce the requested format. The knowledge was already there from pretraining; fine-tuning shapes how it's delivered.

This stage is dramatically cheaper than pretraining — it builds on all that prior work. It's also where specialization happens: you can fine-tune the same base model into a coding assistant, a medical-notes summarizer, or a customer-support bot by choosing different examples.

Stage 3: Alignment — learning judgment from feedback

SFT gets you a helpful assistant, but "helpful" has a lot of subtle dimensions that are hard to capture by demonstration alone. Of two correct answers, which is clearer? More appropriately cautious? Better structured? And how should the model behave when a request is harmful?

The breakthrough technique here is Reinforcement Learning from Human Feedback (RLHF). The trick is that judging is easier than demonstrating: it's hard to write the perfect answer, but easy to look at two answers and say which is better.

It works in two moves:

   1. Collect preferences:
      For a prompt, generate two responses (A and B).
      A human picks the better one.   →  thousands of these comparisons.

   2. Train a "reward model" to predict those human preferences,
      then use it to score the main model's outputs and nudge it
      toward higher-scoring (more preferred) responses.

In effect, we distill fuzzy human taste into a scorer, then let the model practice against that scorer at scale. This is the stage that gives a model its characteristic "personality," its instinct to be helpful and honest, and its willingness to refuse genuinely harmful requests.

Newer variants streamline this — some skip the separate reward model, some use AI feedback instead of (or alongside) human feedback to scale it up, and some use reinforcement learning against automatically checkable rewards (did the code pass the tests? is the math answer correct?) to sharpen reasoning. But the core idea is constant: improve the model using a signal about which outputs are better.

Putting the stages together

   ┌───────────────┐     ┌──────────────┐     ┌─────────────────┐
   │  PRETRAINING  │ ──► │ FINE-TUNING  │ ──► │   ALIGNMENT      │
   │  learn the    │     │ (SFT)        │     │   (RLHF, etc.)   │
   │  world        │     │ learn the job│     │  learn judgment  │
   ├───────────────┤     ├──────────────┤     ├─────────────────┤
   │ trillions of  │     │ thousands–   │     │ thousands of     │
   │ tokens, $$  │     │ millions of  │     │ preference       │
   │ no labels     │     │ examples     │     │ comparisons      │
   │ → base model  │     │ → assistant  │     │ → aligned model  │
   └───────────────┘     └──────────────┘     └─────────────────┘
        capability            behavior              judgment

This three-stage view explains a lot of model behavior:

• Hallucinations trace back to pretraining: the model learned to produce fluent, plausible text, and fluency doesn't guarantee truth. Later stages reduce it but can't fully erase it.
• Tone and refusals come from fine-tuning and alignment. When two models with similar raw ability "feel" very different, the difference is usually here.
• The knowledge cutoff is set in pretraining; that's why even a freshly released model can be unaware of recent events.
• "Reasoning" models that think step by step before answering are largely a product of the alignment stage being pushed hard on problems with checkable answers.

We now have a fully trained model. The next question is the one you actually hit every time you use it: what happens, mechanically, when you press enter and it starts writing back?