Tokens, Vectors, and Attention: How an LLM Actually Reads Your Prompt

Type "G'day mate, can you explain tokenisation?" into an LLM and something answers you back in about a second. Nowhere in that process does the model read your sentence the way you just did. It never sees the word "mate." It never sees a sentence at all. What it sees is a list of numbers, and everything interesting happens to those numbers before anything resembling an answer comes back out the other side.

This post follows that one prompt through the machine, start to finish. I'll be upfront about the trade-off: this is a genuinely deep field, and I'm covering it at a level that gets you a solid, accurate mental model, not a research paper. Tokenization alone has entire papers written about it. Attention has entire courses built around it. Any section here could become its own post if there's appetite for it. What follows is the map, not the territory.

What's actually inside a "model"

LLM stands for Large Language Model. Model, in this context, means the same thing it means in statistics: a mathematical function that takes an input and produces an output, tuned against real examples until it gets good at the job. A large language model is a model built for one specific job: given some text, predict what text comes next. Everything else it appears to do, answering questions, writing code, holding a conversation, falls out of doing that one job extremely well.

Structurally, it's a neural network: a stack of layers, each made of units loosely called neurons. Every neuron takes in a set of numbers, multiplies each one by a weight, adds them up, and runs the total through a small filtering step that decides how strongly the neuron should fire before passing that value on. The weights are the entire model. A frontier LLM has hundreds of billions of them. Training doesn't write code or rules, it just adjusts those weights until the network's predictions get closer to correct.

Networks are typically described in three parts: an input layer (where your data goes in), one or more hidden layers (where the actual computation happens), and an output layer (where the prediction comes out). "Deep learning" just means a network with a meaningful number of hidden layers stacked between the two.

Why depth matters

A single layer of neurons, on its own, can only draw a straight line through its inputs. That's a real limitation: plenty of real-world patterns, including basic ones like "true when exactly one of two things is true," can't be separated by a straight line no matter how you tune the weights. Stack a second layer on top of the first, though, and the network can bend that line into curves and combine simple patterns into more complex ones. Stack dozens or hundreds of layers, each building on the patterns the last one found, and you get a network that can represent something as messy as human language.

Here's a tiny version of that structure. Press play and watch a signal move layer by layer through the weighted connections:

Getting the weights right is training, and the mechanism behind it is backpropagation. The short version: the network makes a prediction, you compare it to the correct answer and compute how wrong it was (the "loss"), and then you work backward through every layer figuring out how much each individual weight contributed to that error. Each weight gets nudged a tiny amount in the direction that would have reduced the error. Repeat that process billions of times over billions of examples, and the weights slowly settle into values that make good predictions. No human decides what any individual weight should be. The training process finds them.

The prompt gets chopped into tokens

Before any of that machinery runs, your prompt has to become numbers. The first step is tokenization, and it doesn't work the way most people assume.

The model doesn't split your sentence into words. It doesn't split it into individual characters either. It splits it into tokens, chunks somewhere in between, decided ahead of time by the tokenizer's own training process rather than by grammar. A token is the smallest unit the model ever operates on. It cannot see anything smaller.

Common, short words are usually a single token. "Mate" is one token. "The" is one token. But a longer or less common word like "tokenisation" typically gets split into several: something like token, is, and ation. Spaces matter too, most tokenizers bundle a leading space into the following token, so " mate" and "mate" at the start of a line can genuinely be different tokens. Line breaks, punctuation, and code syntax all get tokenized the same way as ordinary words. None of it is special-cased.

The most common method behind this is byte-pair encoding (BPE). It's simpler than it sounds. Start with a vocabulary of individual characters (or bytes). Scan an enormous body of text and find the pair of tokens that shows up next to each other most often. Merge that pair into a single new token. Repeat, thousands of times, and common chunks like ing, tion, and whole short words gradually earn their own single-token entry, while rare or unusual words stay broken into smaller, more common pieces. It's a frequency-driven compression trick, run once during the tokenizer's own setup, long before the model itself is trained.

Our example sentence, "G'day mate, can you explain tokenisation?", might come out something like:

["G", "'day", " mate", ",", " can", " you", " explain", " token", "isation", "?"]

Ten tokens for six words. That gap between "word" and "token" is one of the most useful facts to hold onto when reasoning about how an LLM behaves.

Here's that split happening one piece at a time, watch each chunk turn into a token and then into an ID:

Every token is just a number

The tokenizer isn't inventing new tokens on the fly. It's built once, against a fixed vocabulary, typically somewhere between 30,000 and 200,000 entries depending on the model. Every possible token already has a permanent slot in that vocabulary, and each slot has an index: a plain integer.

So " mate" doesn't get passed into the network as text. It gets looked up and replaced with something like 13592. The entire prompt becomes a short list of integers before the model does a single scrap of "thinking." This is also why an LLM has a hard limit on how much text it can take in at once, its context window is measured in tokens, not words or characters.

Embeddings: you shall know a word by the company it keeps

An integer index tells the network which token it's looking at, but an index carries no meaning by itself. Token 13592 being one more than token 13591 says nothing about whether they're related. So the model looks each ID up in an embedding table and gets back a vector, a list of numbers, in return.

A vector, if the term's unfamiliar, is just an ordered list of numbers treated as a single point (or direction) in space. Two numbers give you a point on a page. Three give you a point in a room. An embedding vector might have a few hundred or few thousand numbers, describing a point in a space with that many dimensions, more than anyone can actually picture, but the maths works the same regardless of how many numbers are in the list.

Here's the part that matters: those numbers aren't arbitrary. They're learned during training, and they're learned from context. This is an old idea in linguistics, usually credited to J.R. Firth's line: "you shall know a word by the company it keeps." During training, the network is rewarded for placing tokens that tend to appear in similar surroundings closer together in this vector space, and tokens that never appear in similar surroundings further apart.

Nobody hand-labels "happy" and "joyful" as synonyms. The model never sees a dictionary. It simply notices, across a training set of trillions of words, that both tokens keep turning up around the same kinds of sentences, and the training process pushes their vectors closer together as a side effect of getting better at prediction. This is called distributional semantics, and it's the reason concepts like gender, tense, sentiment, and even rough analogies (king is to queen as man is to woman) emerge in the vector space without anyone designing them in.

It's worth being precise about what happens when, though. This embedding lookup gives each token a fixed, general-purpose vector, the same one every time that token appears, anywhere. It doesn't yet know it's sitting next to "tokenisation" in this specific sentence. That contextual awareness is what the transformer layers add next.

Measuring closeness: cosine similarity

If meaning lives in the geometry of this space, the model needs a way to measure how close two vectors actually are. The standard tool for that is cosine similarity, and it deliberately ignores how long a vector is and looks only at the angle between it and another vector.

Point two vectors from the same origin. If they point in exactly the same direction, the angle between them is zero and cosine similarity gives you 1. If they point in completely unrelated directions (at right angles to each other), similarity sits around 0. If they point in opposite directions, similarity heads toward -1, though in practice most everyday token pairs land somewhere between 0 and 1. Fully opposite vectors are rare. "Happy" and "joyful" end up pointing in a very similar direction and score close to 1. "Happy" and "refrigerator" point in largely unrelated directions and score close to 0.

This is why vector search, recommendation systems, and a fair amount of what happens inside a transformer all lean on the same trick: turn "how related are these two things" into "what's the angle between their vectors," a calculation a computer can do instantly across billions of comparisons.

Here's what that looks like with real coordinates: six example tokens placed by direction, then the actual angle and score worked out between two of them:

The transformer: attention and feed-forward

The embeddings are the raw material. The transformer is the machinery that actually processes them, and every modern LLM is built from a stack of near-identical transformer blocks, each one containing two sub-layers: attention, then a feed-forward network.

Attention is the part that lets a token's vector shift based on the other tokens around it in this specific prompt, the piece the embedding step deliberately left out.

For every token, the network generates three vectors from its embedding, using three separate learned weight matrices: a query, a key, and a value. A useful way to picture it, borrowed from AI researcher and educator Jay Alammar: the query is a sticky note describing what this token is looking for, the keys are the labels on every folder in a filing cabinet, and the value is what's actually inside the folder you end up pulling.

Each token's query gets compared against the key of every token it's allowed to see, itself included, to produce a relevance score. Those scores get turned into weights, and the token's new vector becomes a weighted blend of all those value vectors, leaning heavily on whichever tokens scored highest.

Time for a different example sentence, one built specifically to make this payoff obvious: "the trophy didn't fit in the suitcase because it was too big." Nothing about the word "it" tells you, on its own, whether it means the trophy or the suitcase. Attention is what pulls that answer in from context.

Here's that resolution actually happening. Watch the attention weights settle mostly onto "trophy":

Right after attention comes the feed-forward network, a smaller, ordinary stack of layers applied to each token's vector independently. Where attention mixes information across tokens, the feed-forward layer does further processing on each token's own vector in isolation. A lot of what interpretability researchers currently believe about where an LLM stores learned facts points to these feed-forward layers specifically, though that's still an active area of research rather than settled fact.

Real transformer blocks have more plumbing than that (residual connections and normalisation steps that keep training stable across very deep stacks), which is exactly the kind of detail this post is deliberately skipping to keep the shape clear.

Emergent, not designed

Nothing above describes anyone writing a grammar rule, a fact, or a reasoning strategy directly into the model. Nobody coded "trophy refers back three words" or "Paris is the capital of France" into a weight. Those weights were found the same way described earlier: backpropagation, paired with an optimisation method called gradient descent, nudging billions of numbers over and over to reduce prediction error across an enormous amount of text. The behaviours we can observe (grammar, facts, some reasoning, some of the model's failures too) are a side effect of that process, not a specification anyone wrote.

That distinction matters more than it sounds like it should. It's why nobody, including the labs building these models, can point to the exact weight responsible for a specific fact or behaviour. It's why LLMs can be startlingly capable in some areas and confidently wrong in others, with no clean line separating the two. And it's why an entire field, mechanistic interpretability, exists purely to reverse-engineer what training actually produced, after the fact, the same way a biologist studies an organism that evolved rather than one that was drafted on a whiteboard.

Stacking it all up

One transformer block, attention followed by feed-forward, refines every token's vector once. Modern LLMs chain dozens of these blocks, sometimes over a hundred in the largest models, and the output of one block becomes the input to the next. Early blocks tend to pick up on simple, local patterns. Later blocks build increasingly abstract representations on top of what earlier blocks already found, the same way stacking simple layers earlier let a network bend a straight line into a curve. By the final block, the vector sitting at the last token's position is carrying a highly compressed, context-loaded summary of everything relevant that came before it in the prompt.

Training the base model

Everything described so far is architecture, the shape of the network and how information flows through it. None of it does anything useful until the weights are trained, and that's pre-training: a lab takes a raw, randomly-initialised network and trains it on a next-token-prediction task across a vast corpus of text, sometimes trillions of tokens scraped and licensed from the web, books, and code.

The task itself is deceptively simple: given the tokens so far, predict the next one, check the answer against the real next token, and backpropagate the error. No human labels any of it. The correct answer is just whatever token actually came next in the source text. Run that loop across enough data and enough compute (frontier pre-training runs use tens of thousands of specialised chips for months), and a model that started out predicting gibberish ends up predicting fluent, coherent, often accurate text. Worth flagging clearly: that's only the first stage. Turning a raw base model into the kind of assistant you'd actually want to talk to takes further stages after pre-training, like instruction tuning and reinforcement learning from human feedback, which are entirely their own topic.

From vectors back to your answer

So far this whole post has been the input side. The output side runs the same machinery in reverse, one token at a time.

After the last transformer block, the vector at the final position gets compared against the whole vocabulary one more time. That produces one raw score, called a logit, for every single token in the vocabulary, tens of thousands of numbers in total. A function called softmax turns that pile of scores into a proper probability distribution: every token gets a probability between 0 and 1, and the whole set adds up to 1. A sampling step then picks one token from that distribution, sometimes just the single highest-probability token, sometimes with a controlled bit of randomness mixed in across the top candidates, depending on the settings in play.

Whichever token gets picked, it's converted straight back from its ID into text using the tokenizer's reverse lookup, and appended to the sequence. Then the entire process runs again: the full sequence, prompt plus everything generated so far, goes back through tokenization (already done), embeddings, and every transformer block, to predict the next token after that. One token out, per full pass through the network. That's why longer replies visibly take longer: the model runs a fresh pass through every layer for every single token it writes (real systems cache some of the repeated work rather than starting from zero each time, but a full layer-by-layer pass still happens per token).

The whole trip, in one breath

Your prompt becomes tokens. Tokens become vocabulary IDs. IDs become vectors. A stack of transformer blocks, attention pulling in context, feed-forward refining each token further, turns those vectors into an increasingly meaning-loaded representation. The last block's output becomes a probability across the whole vocabulary. One token gets picked, gets decoded back into text, and gets fed straight back in to predict the next one. Repeat until the model stops. That's the entire path, from your keyboard, through a few hundred billion weighted numbers, and back out as an answer.