LLM Jargon, Decoded: Weights, KV Cache, Quantization and Inference

A primer published by Tom's Hardware, written by contributor Bruno Ferreira, breaks down the vocabulary now attached to every AI hardware launch — weights, quantization, KV cache, mixture-of-experts, and the two-phase mechanics of inference. The trigger is concrete: hardware for running local models, such as Nvidia's RTX Spark, is pushing a wave of new acronyms into consumer coverage. The article's central caveat is worth stating up front. It is a terminology guide, not a product review, and it promises no benchmark results.

What is a model, and what are weights?

The primer defines a model as the large language model itself, computed from training sets of text on the scale of terabytes to petabytes. Sources include the internet, books, and video and audio transcripts. A model is roughly a stack of statistical layers that analyze relationships between tokens — words, parts of words, code, and symbols.

Weights determine how strongly those tokens connect. The article offers an example: the weights linking "na" to "naval", "nachos", or "naruto" all differ, and a higher weight means the word appeared more often in training data. Weights cannot change without retraining. Parameter counts such as "4B" or "70B" measure weights — billions for consumer models, trillions for frontier models like GPT, Claude, and Kimi.

The guide is blunt about the marketing angle. More weights do not guarantee a smarter model, it says, comparing the metric to CPU clock speeds. Bigger is usually better, but the weights must be meaningful.

Open-weight models — Llama, DeepSeek, Kimi — publish weights for free download. The primer adds a caveat many vendors skip: downloading weights reveals nothing about how the model was trained. It likens this to a free customizable application, not open-source software.

Why does quantization decide what runs on your GPU?

Data types such as FP16, INT8, and FP4 define the width and precision of numbers in a program. Memory is finite, the article notes, and VRAM-limited GPUs reward the smallest usable unit. Training labs use high precision for weights; the final model cannot, or it would be prohibitively large and slow.

Quantization is the fix, described plainly as "clever rounding." Several post-training passes convert precise weights to smaller formats, and each lab guards its own method as a critical trade secret. Frontier models keep BF16 weights to preserve reasoning quality — the guide compares this to choosing whether a GPS location is accurate to miles or inches. Consumer models drop to smaller types so they fit on a Mac Studio, an RTX Spark, or a graphics card. Llama 3.3 70B, for example, runs in full BF16 or quantized INT4 versions, and Hugging Face listings often offer several quantization levels.

Context windows: bigger is not always better

The context window, measured in tokens, is the model's thinking space — analogous to RAM, with one million tokens common among frontier models. It holds everything: questions, replies, system prompts, uploaded files, and fetched links. For the model, these remain one large pile of tokens, which is why separating "data" from "code" is technically impossible; modern models only approximate the distinction.

The primer stresses that LLMs do not think. They re-run statistics over all tokens and predict the next few. The caveat follows: even at three million tokens, prediction can break down before the window fills. This is the "lost in the middle" effect — models attend to the start and end of a conversation and forget the middle. Each model is tuned to its window size.

What is the KV cache, and why does it eat VRAM?

The KV cache stores the key-value vectors that record token relations within a session, making them reusable between queries. Since generation is linear, each new token needs every preceding KV. In a 300,000-token conversation, recalculating all of them from scratch would be slow, and disproportionately slower as interactions grow. The cache is also quantized, trading minor accuracy loss for lower VRAM use — and with large contexts, the article warns, the cache can consume more memory than the model and weights themselves.

Mixture-of-experts and RAG

MoE embeds sub-models within the main model. Each starts with the same data but slightly randomized weights; training's "rich-get-richer" reinforcement converges related tokens into experts. The primer corrects a common misconception: experts relate to token patterns, not topics. Good answers about landscape painting are a happy accident. A router selects experts per token, so only part of the model runs — faster and more accurate in places, at the cost of higher VRAM usage.

Retrieval-augmented generation feeds external material — web searches, documents, spreadsheets — into the context window. Perplexity, built around RAG, is described as more context-aware despite using the same models as rivals. The cost is compute: a 10,000-token exchange can pull in hundreds of thousands of tokens of retrieved webpages. Search queries themselves are usually generated by a smaller, separate model.

Prefill, decode, and the tricks that speed them up

Inference splits into two phases. Prefill tokenizes and processes all input in parallel, and it dominates "time to first token," a common benchmark reported alongside prefill tokens per second. Decode then generates one token at a time, sequentially, and is limited by memory bandwidth rather than raw compute — memory speeds matter more than FLOPS here. Decode speed is quoted as tokens per second.

Speculative decode adds a drafting mechanism that guesses upcoming tokens; the model verifies them in a parallel pass, rejects bad guesses, and keeps its own output. Multi-token prediction, often built in during training, can serve as that drafting mechanism.

The guide is explicitly non-exhaustive and will be updated. For hardware buyers, its most practical lesson is arithmetic, not hype: model size, quantization level, KV cache, and context window together determine whether a model fits — and runs — on consumer silicon.