How Do Large Language Models Work? Tokens, Embeddings and Attention Explained for 2026
Large language models work by converting your text into tokens, turning each token into a vector, and running those vectors through stacked attention layers that produce a probability for every possible next token. The model then samples one token, appends it to the input, and repeats the entire pass until it emits a stop token.
- One token at a time. No paragraph-level planning step: the full network re-runs for every token you watch appear.
- Tokens are the unit of your bill. OpenAI's cookbook puts English near four characters per token, so a 3,500 word policy document lands around 4,500 tokens.
- Attention is a weighted lookup, not comprehension. Each token asks every earlier token how relevant it is, then blends their values.
- The context window is a wall, not a suggestion. Claude Opus 5 ships 1M tokens of context and 128K maximum output, and attention cost grows with the square of what you put in it.
- The sampling knobs are going away. Anthropic's docs state temperature, top_p and top_k are unsupported on Claude 4.7 and later, and a non-default value returns a 400 error.
- Start at the tokenizer, not the transformer paper. Almost every bug you ship is a tokenizer, chunking or context bug.
Your document assistant answers policy questions perfectly across twenty PDFs. Point it at four thousand and answers get vaguer, latency triples, and the bill arrives at a number nobody wants to defend in a review meeting. Nothing is broken. You have walked into how large language models work underneath the chat box, and each symptom has a cause you can name.
How Do Large Language Models Work? Six Stages, One Token at a Time
Strip away the product surface and a model is a function that takes numbers and returns a score for every word piece it knows. Everything else is plumbing.
One prompt, one token, one full pass
The model re-runs in full for every token it produces.
Mechanism as described in vendor inference documentation, checked 20 September 2026.
| Stage | What goes in | What comes out | Where it costs you |
|---|---|---|---|
| 1. Tokenize | Your raw string | Integer token IDs | Every ID is a billed token; odd strings split badly |
| 2. Embed | Token IDs | One vector per token | Fixed cost, rarely your problem |
| 3. Add position | Vectors | Vectors that know their order | Long inputs stretch the scheme and recall degrades |
| 4. Attention layers | The full sequence | Context-aware vectors | Compute grows with the square of sequence length |
| 5. Logits | Final layer output | A score for every token in the vocabulary | Nothing directly, but truth is decided here |
| 6. Sample | Scores | Exactly one token | Output tokens cost several times input |
Every chatbot reply you have read was assembled one token at a time, with the entire network re-run for each one.
What Is a Token in an LLM, and Why Your Bill Is Written in Them
A token is a chunk of characters the model treats as one unit. Not a word, not a letter. Common English words map to one token; rare strings get shredded. OpenAI's token-counting cookbook puts English at roughly four characters per token, and its cl100k_base encoding carries about 100,257 entries. That is why prose survives intact and your reference codes do not.
Count them rather than estimating:
pip install tiktoken
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
text = "Policy HDFC-2024-ML-0917 lapsed on 14 March."
ids = enc.encode(text)
print(len(ids)) # how many tokens you pay for
print([enc.decode([i]) for i in ids]) # the actual pieces
Read the second line. Lapsed and March come back whole. The policy number comes back as fragments, because the tokenizer has never seen that string and falls back to smaller pieces until it can represent it. A CSV of account numbers quietly eats your context.
Count against the model you will actually call
Tokenizers differ between vendors, so a tiktoken number is an estimate for a Claude model, not an answer. Anthropic exposes POST /v1/messages/count_tokens, which takes the same shape as a real message, tools and files included:
import anthropic
client = anthropic.Anthropic()
resp = client.messages.count_tokens(
model="claude-opus-5",
system="You are a claims assistant.",
messages=[{"role": "user", "content": open("policy.txt").read()}],
)
print(resp.input_tokens)
The Claude documentation calls this an estimate: a real request may differ slightly because the platform adds tokens for its own optimisations, and states you are not billed for those. A percent or two is all a budget needs. Working fluently against the Messages API is the practical core of 360DT's CCDV-F developer prep course.
Embeddings: How Tokens Become Directions in Meaning Space
Each token ID is looked up in a large table and comes back as a vector of floats: its starting position in a space where distance approximates relatedness. The open model all-MiniLM-L6-v2 uses 384 dimensions; frontier models use thousands.
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("all-MiniLM-L6-v2")
v = model.encode(["policy lapsed", "premium not paid", "car insurance claim"])
def cos(a, b):
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
print(cos(v[0], v[1])) # high: same idea, different words
print(cos(v[0], v[2])) # lower: same domain, different idea
Run it and you get the most useful intuition in applied AI: policy lapsed and premium not paid land close together despite sharing no words, while car insurance claim sits further away despite sharing the topic. Every vector database is that one comparison. Also read: Retrieval Augmented Generation Explained, Step by Step.
One detail trips people up. The embedding table knows nothing about order, so claim denied and denied claim are identical bags of vectors. Position is added separately, which is why models get less reliable at recalling a fact buried mid-document. Vector indexes on Azure AI Search are a large part of the AI-103 syllabus in the Generative AI Developer course.
Transformer Architecture Explained: What Attention Actually Computes
The mechanism, without the matrix notation. For every token the model computes three vectors: a query (what am I looking for), a key (what do I offer) and a value (what I would contribute if chosen).
Take the sentence the claim was rejected because it was filed late. Processing it, that token's query is compared against the key of every earlier token. Claim scores high; because scores low. Those scores become weights, and the new representation of it is a blend of the value vectors tilted heavily toward claim.
Three consequences follow, and they explain most of what you will see in production:
- It is a lookup, not understanding. Attention scores are dot products between learned projections. Nothing checks whether the result is true.
- It is quadratic. Every token compares against every other, so doubling your input roughly quadruples the attention work. Your latency curve is not a line.
- It is repeated, in width and depth. Multi-head attention runs many lookups side by side, each learning a different relationship, and stacked layers build on earlier ones.
That stack is the transformer. Tool use, agent loops and structured output sit on top without changing it: the model still only predicts the next token, and a function call is just a token the surrounding software knows how to intercept. Building that software is what 360DT's live AI Engineer course spends its middle weeks on.
How LLMs Generate Text: Logits, Temperature and the Sampling Step
The final layer returns one raw score, a logit, for every token in the vocabulary. Softmax turns those into probabilities, and temperature divides the logits first, sharpening or flattening the distribution. The model has written the premium is and gets these four scores, everything else far below:
| Candidate token | Logit | P at T = 0.5 | P at T = 1.0 | P at T = 2.0 |
|---|---|---|---|---|
| " due" | 8.2 | 86.6% | 61.9% | 43.1% |
| " paid" | 7.1 | 9.6% | 20.6% | 24.9% |
| " calculated" | 6.5 | 2.9% | 11.3% | 18.4% |
| " overdue" | 5.9 | 0.9% | 6.2% | 13.6% |
Nothing mystical happened. At temperature 1.0 the model picks due about six times in ten; at 0.5, nearly nine times in ten, which reads as consistency. At 2.0, overdue climbs from one chance in sixteen to better than one in eight, which reads as creativity right up until it reads as a wrong answer about somebody's insurance.
This one bites everybody once. You set temperature=0.2 because a tutorial told you to, it works, then you move to a newer model and every request comes back a 400. The Claude documentation is blunt: temperature, top_p and top_k are not supported on Claude 4.7 and later, and a non-default value is an error, not a warning. Strip the knobs out and put the determinism into the prompt and the output schema.
Why the LLM Context Window Decides How Large Language Models Work at Scale
The context window is the maximum number of tokens the model can attend over in one pass: system prompt, conversation, retrieved documents, tool definitions and the reply itself. Claude Opus 5 ships a 1M token window with 128K maximum output, checked on 20 September 2026.
A window that large tempts you into a bad habit, and the arithmetic is worth doing once. Take a two person data team at a mid sized Indian insurer running a policy assistant: 2,000 questions a day on Opus 5, at the published 5 dollars per million input tokens and 25 per million output.
| Approach | Input tokens per question | Cost per question | Cost per day at 2,000 questions |
|---|---|---|---|
| Retrieve 8 relevant chunks | ~7,000 | ~$0.045 | ~$90 |
| Paste the whole handbook every time | ~60,000 | ~$0.31 | ~$620 |
| Whole handbook, cached as a stable prefix | ~60,000 read from cache | ~$0.04 | ~$80 |
Anthropic publishes cache reads for Opus 5 at 0.50 dollars per million tokens, which is what makes the third row possible. The catch is real: cache writes cost more than ordinary input, the prefix must be identical, and it expires, so the saving lands only when many requests share an opening. That 60,000 token version also stays slower and less accurate, because attention over a wall of irrelevant text dilutes the tokens that matter.
Underneath, the constraint is memory. To avoid recomputing attention over the whole history at every step, serving systems cache the key and value vectors for every previous token, and that cache grows linearly with context length inside GPU memory. Published 2026 inference cost analyses put the KV cache for one 128,000 token request to Llama 3 70B at roughly 40 GB, on top of around 140 GB for the FP16 weights. That is the honest reason self-hosting gets expensive fast (Also read: How to Run an LLM Locally in 2026), and sizing that layer is the day job behind the AI-300 objectives in the MLOps Engineer course.
Back at the insurer
None of their three problems was a model quality problem. The vague answers came from chunking PDFs on a fixed 1,000 character boundary, which cuts tables in half and orphans a clause number from its clause. The tripled latency came from raising retrieval from 4 chunks to 12: recall rose a few points, response time rose by a multiple. The bill was row two. Fixing the chunking is pipeline work, which is what the Microsoft Fabric data engineering course builds.
Go from knowing how LLMs work to shipping systems built on them
Build agents that plan, use tools and act, certified on both Microsoft Copilot Studio and Claude Code. 100+ hours of live instruction over 16 weeks, with RAG and agent tool loops built in session, not on slides.
Explore the course
How Do Large Language Models Work in Practice? Where to Put Your First 20 Hours
Most people start in the wrong place. They open the 2017 attention paper, lose an evening to matrix dimensions, and come away able to recite query, key, value without explaining why their retrieval returns garbage. Unless you want a research role, do not build a transformer from scratch. That returns intellectual satisfaction; tokenizers and context economics return a system that works.
A 20 hour split that actually pays off
Weighted toward the layers you will debug, not the ones most fun to read about.
A recommended allocation, not a measured statistic.
Over four weekends: tokenize five documents from your own work and find the one that splits worst; embed forty sentences from your domain and check the neighbours make sense; cut one long prompt in half without losing answer quality; then read the attention theory with that in hand.
Now the caveat the explainers skip. None of this makes you better at prompting. The mechanism explains failures after they happen and rarely predicts which wording will work, because the weights that decide are not inspectable from outside. If your goal this quarter is a working assistant, spend the hours on retrieval quality and evaluation. Also read: What Is Context Engineering.
For architectural framing rather than code, the CCAR-F foundations prep course covers context design, tool boundaries and guardrails; the full certifications overview shows how the Microsoft and Anthropic tracks fit together.
Related guides
- What Is Claude Opus 5? puts the context and pricing numbers here against one frontier model.
- RAG vs Fine-Tuning in 2026 is the decision you face once you know context is expensive.
- LLM Evaluation in 2026 shows how to prove a change helped once you are shipping.
- AI Engineer Roadmap 2026 turns this into an ordered plan for a first role in India.
Frequently asked questions
How do large language models work in simple terms?
Your text is split into tokens, each becomes a vector, and those vectors pass through stacked attention layers where every token absorbs context from the others. The last layer scores every possible next token, the sampler picks one, and the whole pass repeats.
What is a token in an LLM and how many tokens is one word?
A token is a chunk of characters the model treats as one unit. OpenAI's cookbook puts English near four characters per token, roughly 1.3 tokens per word. Code, non-English scripts and identifiers such as policy numbers cost far more.
Do LLMs actually understand what they are saying?
The mechanism is a weighted lookup, and nothing in it verifies truth. But autocomplete undersells what the layers build: attention resolves references and tracks entities across paragraphs. It models token relationships extremely well and cannot check correctness.
What happens when you exceed the LLM context window?
The API returns an error rather than truncating silently, so you can catch it. The subtler failure comes earlier: recall of a fact buried mid-document degrades as input grows, and attention cost rises with the square of the sequence.
Why do LLMs make things up?
Because the sampling step always returns a token. When the correct continuation is not strongly represented, the next most plausible one is picked and still sounds fluent. Grounding answers in retrieved text reduces it; nothing eliminates it, which is why evaluation matters.
Is temperature still worth tuning in 2026?
Less than it used to be. Anthropic's documentation states temperature, top_p and top_k are not supported on Claude 4.7 and later, and a non-default value returns a 400 error. Where still accepted, adjust temperature or top_p, not both.
How long does it take to learn this properly?
Around 20 focused hours gets you to where you can debug a retrieval system and reason about a token bill. Production systems with agents and evaluation are a different scale: 360DT's AI Engineer course runs 100+ hours over 16 weeks.
If you take one thing from this page, make it the tokenizer exercise. Watching your own documents get chopped into pieces you did not expect is the fastest correction to a wobbly mental model, and everything else here follows from it, including the bill. Run it this week on five real files, then decide whether to keep assembling this from articles or learn it with an instructor. A free webinar or a demo class answers that faster than another blog post.
About this guide. 360 Digital Transformation is an Authorized Training Partner of Anthropic and Microsoft. Other certification bodies, vendors and employers named here are not affiliated with us. Tools and versions change quickly; commands and figures cited were checked on 20 September 2026.
Python
Claude API
FAISS
ChromaDB
LangChain
LangGraph
CrewAI
AutoGen (AG2)
