Running an LLM Locally Is a Memory Problem: The Arithmetic of 4-Bit Quantization

Barakaeli Lawuo, Sep 29, 2026

In 2022 Tim Dettmers and colleagues tried to store large language models in 8 bits instead of 16, and found that it stopped working at a specific size. Up to about 2.7 billion parameters, a careful 8-bit scheme kept the model's accuracy. Once models reached about 6.7 billion parameters, standard 8-bit methods failed.¹ The cause was a small set of unusually large numbers inside the model. At the 6.7B scale they counted about 150,000 of these outliers in a single 2,048-token sequence, and yet they sat in only 6 of the model's feature dimensions.¹ When the authors set those few dimensions to zero, validation perplexity got 600 to 1000% worse. Zeroing the same number of random dimensions made it about 0.1% worse.¹

That finding matters to anyone who wants to run a model on their own machine, because on a laptop the limiting resource is memory, and quantization is how a model is made to fit. This post starts from the byte count, shows why the obvious way of shrinking it breaks, and then walks through how five papers each got around the problem in a different way.

Fourteen gigabytes before the first token

A model is mostly a large collection of learned numbers called weights (or parameters). A "7B" model has about 7 billion of them. Each weight is stored with some number of bits, called its bit width. The standard format for inference is 16-bit floating point, which takes 2 bytes per weight. So the memory for the weights alone is just the parameter count times the bytes per weight:

Plug in a round 7 billion parameters and three common bit widths:

The papers make the same point at their own scales. GPT-3's 175 billion parameters take 326 GB in 16-bit format (counting a GB as 1024³ bytes), which is more than any single GPU holds.³ The AWQ authors ran a 13B model on a laptop GPU with 8 GB of memory at 33 tokens per second after 4-bit quantization, and note that the 16-bit version could not even fit a 7B model on that card.⁴ The 14 GB line in the chart explains why.

Weights are not the whole bill. While a model generates text it also keeps a cache of past keys and values. For the 175B OPT model, the GPTQ authors estimate that cache at about 9 GB for 2,048 tokens, on top of about 63 GB for the 3-bit weights.³ Still, weights are the largest single item, and they are the part quantization shrinks.

Rounding to 256 levels, and the value that ruins it

Quantization means storing each number on a coarse grid of allowed values instead of as a full-precision float. An 8-bit integer has 256 possible values, and a 4-bit integer has only 16. The simplest scheme, which LLM.int8() calls absmax quantization, finds the largest absolute value in a tensor, scales everything so that value lands on 127, and rounds to the nearest integer:¹

The weakness is that one number, the maximum, sets the step size for everyone. Here is a made-up row to show it. Take the values 0.1, −0.2, 0.15, and 60. The scale is 127 / 60, about 2.1, so 0.1 becomes 0.21, which rounds to 0. So do the other two small values. Only the 60 survives. Remove the 60 and the scale becomes 127 / 0.2 = 635, and all three small values keep distinct integers. LLM.int8() describes exactly this failure in real models: once the outliers appear, "most quantization bins are empty and small quantization values are quantized to zero, essentially extinguishing information."¹

Two kinds of numbers flow through a model, and they behave differently. Weights are fixed after training, so you can study them ahead of time. Activations are the intermediate values computed from your input as it passes through each layer, so they change with every prompt. The outliers Dettmers found are in the activations: particular hidden dimensions that carry magnitudes up to 20 times larger than the rest.¹ The SmoothQuant paper, from MIT and NVIDIA, puts it directly: weights are "quite uniform and flat," while activation outliers are about 100 times larger than most activation values, which, with one scale for the whole tensor, leaves ordinary channels with only 2 or 3 effective quantization levels.² It also found the outliers stay in the same channels from token to token.²

The usual way to measure the damage is perplexity. The model reads held-out text, you take the log probability it gave to each correct next token, average them, and exponentiate.³ Lower is better. A perplexity of 10 roughly means the model was as unsure as if it were choosing among 10 equally likely tokens. The LLM.int8() paper shows how fast naive rounding degrades it. On C4 text, a 13B model scores 12.45 in 32-bit floats and 19.08 with plain 8-bit absmax. That is worse than the 8-bit 6.7B model at 14.59.¹

Every method below has to deal with that one problem. Where they differ is in what they choose to protect and when they do the work.

LLM.int8(): give the outlier columns their own lane

The first fix is the bluntest. LLM.int8() splits each matrix multiplication in two. Any hidden dimension containing a value with magnitude of at least 6.0 is pulled out and multiplied in 16-bit. Everything else, more than 99.9% of the values, is multiplied in 8-bit.¹ It also gives each row of the input and each column of the weights its own scaling constant, instead of one per tensor.¹ That alone is not enough, because an outlier dimension shows up in almost every row, which is why the 16-bit split is needed.¹

This works because the outliers are so concentrated. For models up to 13B parameters the paper found at most 7 outlier dimensions, so keeping them in 16-bit costs only about 0.1% extra memory.¹ With this split, perplexity for the 13B model returns to 12.45, the same as full precision, and a 175B model runs without measured degradation.¹ The cost is that it stays at 8 bits. It saves about half the memory, a 1.96× reduction for BLOOM-176B, which by the arithmetic above puts a 7B model at roughly 7 GB.¹

SmoothQuant: hand the outliers to the weights

SmoothQuant wanted both weights and activations in 8-bit integers (called W8A8), so that the hardware's fast integer matrix units could do all of the work. Its trick depends on one piece of algebra. You can divide an activation channel by some factor and multiply the matching row of weights by the same factor, and the layer's output does not change:²

Each $s_j$ is computed once from a small calibration sample, and the division can usually be folded into the previous layer's parameters offline, so it adds no extra kernel call.² The exponent $\alpha$ controls how much of the difficulty moves over. For OPT and BLOOM models, the authors found $\alpha = 0.5$, an even split, worked well.² The weights become a bit spikier. The activations become far flatter.

The paper reports up to 1.56× speedup and 2× memory reduction with negligible accuracy loss.² SmoothQuant still stores weights in 8 bits, which for a 7B model means about 7 GB. Getting under that requires pushing the weights themselves below 8 bits, and the next three papers all target that.

GPTQ: round one column, repair the rest

Going to 4 bits leaves only 16 grid points, so each rounding error is bigger. GPTQ, from IST Austria and ETH Zurich, stops treating each weight's rounding as an independent decision. It quantizes a layer column by column. After each column is rounded, it adjusts the weights that have not been quantized yet to cancel out the error, using second-order information computed from sample inputs.³ The goal for each layer is to keep the layer's output close to the original:

Notice that $X$, the activations, appears in the objective. GPTQ only quantizes weights, but it judges each weight by how much error it causes on real inputs. Its calibration set is small: 128 random 2,048-token segments of web text.³ It quantized the 175B models in about four GPU hours.³

The difference from plain rounding (which the paper calls round-to-nearest, or RTN) is large. On OPT-175B at 4 bits, GPTQ lost 0.03 perplexity. RTN lost 2.2, enough to score worse than the full-precision 13B model, which is about ten times smaller.³ At 3 bits, RTN collapsed completely while GPTQ still held up.³ For the 6.7B OPT model on WikiText2, full precision scores 10.86, 4-bit RTN 12.10, and 4-bit GPTQ 11.39.³

AWQ: find the 1% that matter by watching activations

The AWQ team at MIT, several of whom also wrote SmoothQuant, started from an experiment. They quantized OPT-6.7B to 3 bits, in groups of 128 weights, but kept a small slice of weight channels in 16-bit, and compared three ways of choosing that slice. Keeping 1% of channels chosen by large weight magnitude barely helped: perplexity went from 23.54 to 22.37. Keeping 1% chosen at random did nothing useful. Keeping 1% chosen by large activation magnitude brought perplexity down to 11.39, close to the full-precision 10.86.⁴ The weights that matter are the ones that get multiplied by the big activations.

Mixed precision is awkward for hardware, so AWQ protects those channels by scaling instead. Quantization rounds $w / \Delta$, where the step size $\Delta$ is set by the group's largest weight. If you multiply one salient weight by $s > 1$ and divide its input by $s$, the rounding error stays about the same size in absolute terms. Relative to that weight, though, it shrinks by roughly a factor of $s$, because scaling a single weight usually does not change the group's maximum.⁴

Scaling too far backfires, because $\Delta$ grows and the other weights in the group lose precision. On OPT-6.7B the best single value was $s = 2$, and AWQ searches for the best scale per channel automatically.⁴ It needs no backpropagation and no reconstruction. The authors argue this makes it less likely than GPTQ to overfit the calibration set.⁴ The paper also lists llama.cpp among the systems that already used group-wise 4-bit quantization, where a small group of weights shares one scale.⁴

NF4: a 4-bit grid shaped like the weights

All the methods above use evenly spaced grid points. The QLoRA paper asked whether the 16 points of a 4-bit format could be placed better. Trained weights are usually shaped like a zero-centered normal distribution, so most values sit near zero and few sit far out.⁵ NormalFloat4 (NF4) places its 16 values at quantiles of a normal distribution, so each grid point receives about the same number of weights. The authors describe it as information-theoretically optimal for normally distributed data.⁵ Averaged over OPT, BLOOM, LLaMA, and Pythia models from 125M to 13B, NF4 with double quantization gave a perplexity of 27.41, against 34.34 for plain 4-bit integers and 31.07 and 29.48 for two 4-bit float layouts.⁵

QLoRA also accounts for a cost the headline "4-bit" number hides. Every block of weights needs its own scaling constant. With a 32-bit constant for every 64 weights, that adds half a bit per parameter. Double quantization stores those constants in 8 bits, in blocks of 256, and brings the overhead down:⁵

GPTQ reports a similar overhead for its grouped variants, about 0.15 extra bits for groups of 128 weights.³ So a realistic 4-bit 7B model is closer to 3.6 to 3.9 GB than to 3.5 GB. QLoRA's main result is about fine-tuning. Training small adapter weights on top of a frozen NF4 model matched 16-bit fine-tuning on the paper's benchmarks, which let the authors fine-tune a 65B model on one 48 GB GPU.⁵

What the papers measured at 4 bits

The cleanest comparison for a 7B model is in the AWQ paper, which ran plain rounding, GPTQ, GPTQ with a reordering trick, and AWQ on Llama-2-7B under the same settings: groups of 128 weights, measured by WikiText-2 perplexity. Full precision scores 5.47.⁴ The chart shows how much each method adds.

Two things stand out. At 4 bits with small groups, even plain rounding costs only 0.26 perplexity on this model. The AWQ authors note that round-to-nearest "is actually quite strong" when the group size is as small as 128.⁴ Part of the reason is that these are weight-only methods: activations stay in 16-bit (the W4A16 setting), so the activation outliers that broke 8-bit rounding are never rounded at all.⁴ The second point is that the gap between methods widens at 3 bits. There, AWQ adds 0.77 and plain rounding adds 1.19.⁴ Clever methods matter most when you push past the point where simple rounding still works.

The scale trend runs in the model's favor. GPTQ found that larger models generally seemed easier to quantize, OPT-66B being an exception.³ LLM.int8() showed that perplexity, more than parameter count, tracks when outliers appear.¹ One caveat applies to all of these numbers. They are mostly perplexity on held-out text plus a set of zero-shot benchmarks. They show the model predicts text almost as well as before. They do not tell you how a quantized model will do on your particular task.

So the path from 14 GB to under 4 GB for a 7B model rests on three findings. The dangerous values are a few activation channels, not the weights in general.^1,2 Weights should be judged by the activations they multiply.^3,4 The rounding grid and its scaling constants can be shaped to fit the weights' real distribution.⁵ At 4 bits, with those ideas combined, the measured cost on a 7B model is a perplexity increase of roughly a tenth to a quarter of a point.⁴