How LLMs work

Start with a small scene. You type: “The cat sat on the.” A large language model does not “know cats” the way a person does. It has seen billions of similar fragments during training, and it answers a narrower question: given everything so far, what token is most likely next?

It might put high probability on “mat”, lower on “rug”, and a little on “roof”. You sample one token, append it, and ask again. That loop — predict, sample, append — is the entire runtime story. Essays, code, and multi-step plans are long chains of that same move.

Tokens, not words

Models do not read characters or whole English words by default. Text is cut into tokens: common words stay whole; rarer ones split into pieces. Limits and pricing are in tokens, not characters. Code with dense punctuation burns tokens differently than prose. Truncating the wrong end of a prompt can delete the instruction that mattered.

The forward pass in plain English

Modern LLMs are transformers. You do not need every matrix name, but you do need the plot:

Tokens become vectors (embeddings) in a high-dimensional space.
Attention lets each position look at other positions and decide what matters — “it” finds its noun; a function call finds its arguments.
Layers of attention and feed-forward networks refine those vectors.
The final layer scores every vocabulary token; softmax turns scores into probabilities.

Training taught those weights by predicting the next token on huge corpora, then usually instruction and preference tuning so the model follows requests instead of only continuing internet prose.

Sampling is a design choice

Generation controls you should explain
Knob	What it changes	When to turn it
Temperature	How peaky vs flat the next-token distribution is	Low for code/JSON; higher for brainstorming
Top-p	Sample only from a probable nucleus	Stabilize creative tasks
Max tokens	Hard stop on generation length	Always set in production
Stop sequences	End early on a marker	Tool protocols and field delimiters

Context window: the whole stage

Everything the model can see for this call — system instructions, tools, retrieved docs, chat history, the user message — shares one context window. There is no hidden long-term mind unless you build it. When the window fills, something must go. That constraint is why RAG, memory, and agents exist: they decide what earns a seat on stage.

What “reasoning” actually is

When a model works a problem step by step, it is not running a separate logic engine. It is generating intermediate tokens that make the eventual answer more likely — a scratchpad in the same stream. Asking for chain of thought often helps because those tokens reshape later predictions.

PYTHON

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    temperature=0.2,
    messages=[
        {"role": "system", "content": "You are a precise interview coach. Prefer short, correct answers."},
        {"role": "user", "content": "In one paragraph, what is an LLM doing at inference time?"},
    ],
)
print(response.choices[0].message.content)

Failure modes interviewers love

Hallucination — fluent next tokens that are not grounded in evidence.
Context overflow — silently dropping early instructions when you stuff the window.
Prompt injection — untrusted retrieved text that tries to override system policy.
Distribution shift — the model never saw your jargon; without RAG it will guess.

Once you see the model as a conditioned token engine with a finite stage, the rest of this part becomes inevitable engineering: put the right tokens on stage, constrain the outputs, and verify against tools and documents.

Pretraining, then alignment

Pretraining is reading the internet (and more) to learn statistics of language. Alignment stages afterward — supervised instruction data, preference tuning, safety policies — bend the same engine toward being helpful and constrained. In system design talks, separate “what the weights know” from “what we put in the prompt today.” One changes rarely and costs a fortune; the other changes every deploy.

Multimodal note

Some models also take images or audio by mapping those inputs into the same kind of token stream. The story does not change: everything becomes conditioning for the next token. When you design products, ask whether vision is required or whether a captioning tool plus text RAG is enough.

Interview drill — LLMs in production

Production LLM questions: context, decoding, structure, cost, eval.

More drills in the Interview Lab.

Q1. Context window overflow

Conversation exceeds model context.

Asked at: All AI product interviews · Difficulty: Medium · Pattern: Context management

Answer

Summarize older turns; pin entities into working state.
Retrieve long-term memory instead of full history.
Truncate least relevant; larger windows are only one lever.