From Tokens to Chips

Before your prompt reaches the model's numerical computation, text is split into token IDs. Those IDs are the interface between human text and numerical computation.

File paths, stack traces, logs, and code snippets all become tokens before the model can use them as context.

Token count shapes context length, latency, and cost.

Do not think of tokens as exactly words. Code and logs can tokenise in surprisingly dense ways.

OpenAI token docs.

Parameters are model weights learned during training and reused during inference to transform token representations.

When the assistant matches a common retry-loop bug pattern, that behaviour comes from patterns distributed across many learned weights.

Model size is partly a question of how many learned numbers must be stored, moved, and used.

A parameter is not a fact in a database. It is one number in a learned computation.

Standard neural-network background; Karpathy's Neural Networks: Zero to Hero is the intuition source. Reiner and Chinchilla become relevant later for model size, active parameters, and training trade-offs.

The token ID is mapped to a dense vector, usually floating-point values, placing text into a mathematical space.

The tokens for billing, retry, timeout, and failed test become vectors the model can compare and transform.

Embeddings are where language first becomes geometry.

An embedding is not a sentence summary by itself. It is a numerical representation used by later computation.

OpenAI embeddings guide.

During inference, the model applies its layers to token representations and produces probabilities for next tokens.

The assistant processes the prompt and produces the next suggested word, code token, or explanation token through repeated forward passes.

Serving an AI coding tool is mostly about doing forward passes quickly and economically.

Inference is not training. The weights are used, not broadly rewritten.

Standard LLM serving background and Lecture 1 framing.

Model layers store and transform vectors using large matrices of learned weights and activations.

The assistant's internal representation of the failing test is repeatedly reshaped by matrices.

Once language becomes numbers, much of the work becomes matrix work.

The matrix is not the model's memory in a human sense. It is the shape of a computation.

Standard linear algebra background.

Transformer computation involves many large matrix and tensor operations, including projections, feed-forward layers, and attention-related operations.

Your prompt becomes batches of numerical operations that accelerators can perform in parallel.

This is why AI hardware is optimised around high-throughput numerical operations.

The model is not doing symbolic code review like a compiler. It is doing learned numerical transformations.

NVIDIA matrix multiplication guide.

MAC operations are the tiny arithmetic core behind dot products and matrix multiplication.

A code explanation that feels conversational is built from vast numbers of these tiny arithmetic steps.

Chip design often starts with making this repeated operation fast and efficient.

The operation is simple; the scale, data movement, and scheduling are the hard parts.

Lecture 2 chip-design path from small operations upward.

Compared with CPUs, GPUs trade broad flexibility for massive parallel throughput and specialised matrix units.

Serving many coding-assistant requests means packing similar work onto accelerators efficiently.

LLM economics depends on how well model work maps to accelerator hardware.

A GPU is not magically fast for everything. It shines when the work can be parallelised and fed with data.

NVIDIA performance sources and Lecture 2 hardware discussion.

Even when arithmetic is fast, model serving can be limited by moving weights, activations, and cache data.

A long log pasted into the assistant is not just text. It is data that must be carried through memory systems.

Speed and cost are often bottlenecked by movement, not only arithmetic.

Do not assume more compute always fixes latency. The model may be waiting on memory.

NVIDIA inference optimisation and Lecture 1 roofline framing.

Training optimises weights using large datasets. Inference runs forward passes with fixed weights for user requests.

Your coding assistant is usually doing inference when it proposes a patch.

Training and serving have different cost profiles and hardware pressures.

A model responding to you is not usually learning from that one interaction by changing its core weights.

Standard technical background and Lecture 1 serving focus.

Serving systems may combine multiple requests or token operations to improve throughput, with latency tradeoffs.

Your small code-fix request may share accelerator time with other users' requests.

Batching shapes cost, speed, and how products feel under load.

Bigger batches can improve throughput without always improving your individual wait time.

Grounded in Reiner Pope's Lecture 1 and NVIDIA inference material.

Prefill processes the input context in parallel. Decode repeatedly uses prior context to produce the next token.

Reading the issue, files, and logs is prefill-like. Writing the patch explanation is decode-like.

The two phases stress hardware differently and explain why long prompts and long answers feel different.

A model does not generate the whole response in one instant. Decode is sequential at the token level.

NVIDIA inference optimisation and Lecture 1.

Transformers compute query, key, and value representations so token positions can weight and combine information from other positions.

The assistant can connect the failing assertion to a billing-worker function shown many lines earlier.

Attention is central to why long prompts can influence later generated tokens.

Attention is not human attention. It is a learned numerical mechanism.

Attention Is All You Need, 3Blue1Brown, and Jay Alammar transformer explainers.

During decode, cached keys and values from earlier context are reused when producing later tokens.

If your prompt includes a stack trace, the model carries useful key/value state from that context while drafting the fix.

KV cache makes generation practical but turns context into a memory bill.

Cache helps avoid recomputation, but storing and reading it still costs memory.

Grounded in Reiner Pope's Lecture 1 and NVIDIA inference material.

More context increases the amount of token state that must be processed, stored, and attended over.

Pasting an entire repository gives the assistant more evidence, but also a larger computation and memory problem.

Long context is useful, but it is not free.

The product problem is not "include everything"; it is "include the right context at the right time." Irrelevant logs can dilute focus and add cost.

Standard background; see the token, KV cache, and inference sources below.

HBM, caches, scratchpads, registers, and off-chip memory differ in capacity, bandwidth, latency, and control.

The assistant's useful context has to live somewhere while the model generates; where it lives affects speed.

AI chips are shaped as much by memory movement as by arithmetic.

Do not think of memory as one uniform pool.

Lecture 2 cache/scratchpad path and NVIDIA performance material.

An MoE model may route a token through only some expert subnetworks. These experts are learned components, not necessarily human-readable specialists like code, SQL, or testing.

The routing choice is part of the learned computation, so only some internal pathways may be active for a given token.

MoE changes the relationship between model size, memory placement, communication, and work per token.

Total parameters and active parameters are not the same thing.

Lecture 1 model layout and active-parameter discussion.

Different parts of a model can run on different accelerators, with data passed between stages.

A frontier model may be too large or too busy for one chip, so serving becomes a distributed systems problem.

Scaling AI is not just a single-chip problem. It is scheduling, networking, and utilisation.

Parallelism adds communication costs and coordination complexity.

Lecture 1 discussion of splitting work across racks.

They are empirical relationships that guide training strategy and resource allocation.

They help answer whether to train a bigger model, use more data, or spend compute differently.

They turn frontier AI into a planning and economics question, not only an algorithm question.

A scaling law is not a guarantee for every architecture, dataset, or product use case.

Kaplan et al. for broader neural language model scaling laws; Chinchilla and Lecture 1 for compute-optimal model-size-versus-training-token trade-offs.

The Chinchilla work argued many large models were undertrained relative to their parameter count.

It reframes the question from 'how big is the model?' to 'how was compute allocated?'

Training strategy affects the models that later become practical to serve.

Bigger is not automatically the best use of compute.

Training Compute-Optimal Large Language Models.

Chip design builds upward from transistors and gates to arithmetic units, memory, and dataflow.

The cloud service answering your prompt ultimately depends on physical circuits switching.

Lecture 2 grounds AI in physics and hardware, not abstraction alone.

The cloud is not weightless. It is machines, power, heat, and silicon.

Lecture 2 chip design from the bottom up.

It choreographs data movement so multiply-accumulate work can happen with high reuse and throughput.

Instead of sending every number back and forth to distant memory, the chip moves data through a local pattern.

It is a concrete example of hardware taking the shape of matrix multiplication.

It is not just more cores. It is a dataflow design.

Lecture 2 and matrix multiplication sources.

Both put data close to compute, but they differ in who controls placement and movement.

Accelerator efficiency depends on keeping the right model data close at the right time.

This distinction explains why chip design and compiler/runtime design are linked.

Nearby memory is scarce, so placement choices matter.

Lecture 2 memory hierarchy discussion.

Area, power, memory bandwidth, interconnect, arithmetic units, and programmability trade off against each other.

AI accelerators look the way they do because matrix work and data movement dominate the economics.

Understanding chips helps explain what future AI coding tools can practically do.

No chip is best at everything. Specialization buys performance by giving something up.

Lecture 2 overall frame.