THE DAILY DRAM
Small sips. Big ideas.
DRAM 000 · TOPIC · 2 MIN READ
You upload a document and ask: “Summarize this in five bullet points.”
The LLM processes your request and generates a summary. The process of using a trained AI model to produce that response is called inference.
Every time you ask an LLM to explain something, write code, or summarize a document, you are using inference.
What Happens During Inference?
Several steps happen between your request and the answer appearing on screen.

1. Tokenization — break the text into pieces
Your instructions and the document’s text are broken into tokens. A token can be a word, part of a word, or punctuation. These tokens are represented as numbers the model can process.
2. Forward pass — process the input
The model processes the input through its internal layers and calculates possibilities for the next token. This computation is called a forward pass.
The initial processing of your input is often called prefill.
3. Token selection — choose the next piece
The system selects a token from those possibilities. Settings such as temperature influence how varied that selection can be.
4. Generation — build the response
The selected token is added to the response. The model then calculates what could come next, and the process repeats until it finishes or reaches a limit.
This stage is called decoding. The summary is built one token at a time.
5. Detokenization — turn tokens into readable text
The generated tokens are converted into the words you see. When an answer streams onto your screen, this happens as the response is generated.
Key Inference Concepts in GenAI
Context window: how much the model can work with
The context window limits how much information a model can handle in a request. Instructions, document content, conversation history, and the generated response all need to fit within the supported limits.
Latency: how long you wait
Time to first token measures how long it takes for the answer to start appearing. Total response time measures how long it takes to finish.
A model can start answering quickly but still take time to complete a long response.
Throughput: how much work gets done
Throughput measures how much a system processes over time—for example, tokens per second or requests per minute. It matters when many people use an AI application simultaneously.
KV cache: reuse earlier calculations
While generating an answer, the model stores certain intermediate calculations about earlier tokens in a key-value cache, or KV cache.
This avoids recalculating that information for every new token and makes generation faster. The trade-off is memory: longer contexts and more simultaneous requests require more cache space.
The KV cache is working memory for generation, not permanent learning.
Inference Cost vs. Training Cost
Training cost is the cost of teaching or updating a model. Inference cost is the cost of running it to produce responses.
If your team uses an existing model through an API, inference is usually the ongoing model expense.
For our document-summary example, both the document sent to the model and the summary it generates contribute to usage. API pricing commonly measures these as input tokens and output tokens.
Longer documents, longer answers, and repeated calls can increase the cost.
Where Does an AI Harness Fit?
Inference is the model producing an output. A harness is the surrounding software that manages how the model is used.
It can assemble context, make tools available, execute permitted tool calls, apply controls, and handle retries.
For example, summarizing an uploaded document may need one inference call. An agent that finds the document, summarizes it, checks the result, and saves it may need several calls coordinated by a harness.
One user task does not always mean one inference call.
The Last Sip 🥃
When you ask AI to summarize a document, the answer is built through inference: process the input, select the next token, and repeat.
Understanding that process helps explain three everyday questions: Why does the answer take time? Why does it cost money? And what software is needed around the model to get the job done?
THE DAILY DRAM
Small sips. Big ideas.
One concept. Two minutes. A little wiser.
Enjoyed this Dram? Get the next one in your inbox.