Quantization: How Large AI Models Fit on a Phone

Large AI models have one problem that never shows up in demos: their size. A model with 70 billion parameters needs roughly 140 GB of memory just for its weights, at FP16 precision. That is more than the RAM in an ordinary laptop, and far more than the VRAM in even the most expensive consumer graphics card.

Quantization is the answer to that problem. It converts the numbers inside a model from high precision to low precision, so the size and the cost of running it drop sharply. The interesting question is not whether the model gets smaller, but how much dumber it gets, and why sometimes it does not get dumber at all.

In 2023, Tim Dettmers and his team at the University of Washington released QLoRA, which made it possible to train a 65 billion parameter model on a single 48 GB GPU. Before that it took eight cards. The jump came not from a new architecture, but from a more frugal way of representing numbers.

What Is Actually Stored Inside a Model

A language model is a pile of numbers. Each parameter is one number that sets how strongly one neuron connects to another. A 70 billion parameter model means 70 billion numbers.

At FP16 precision, each number takes 2 bytes. So 70 billion times 2 bytes equals 140 GB. That is just the weights, not the activations, not the key-value cache, not the runtime overhead. Total memory during inference can be two to three times that.

The analogy is simple. FP16 is like storing GPS coordinates to six decimal places. For everyday navigation, three decimals is plenty. The last three only matter if you want to know a car's position down to the centimeter. AI models also carry centimeters that rarely get used.

From FP32 to INT4: The Precision Ladder

Precision in an AI model follows a clear ladder. Each step down halves the size, but also halves the resolution of the numbers.

FP32 is almost never used for inference. FP16 and BF16 are the training standard. INT8 and INT4 are the targets of quantization.

BF16 (Brain Float 16) deserves a special mention. It has the same exponent range as FP32, but a shorter mantissa. That means it can hold very large and very small numbers without overflow, at coarser resolution. Google has used BF16 since 2018 for TPUs, and now almost all large-model training uses it.

Two Ways to Quantize

There are two main approaches, and they have different consequences.

Post-Training Quantization (PTQ) happens after the model is trained. An existing model has its precision changed without retraining. This is the most common route, because it is cheap and fast. GPTQ, AWQ, and GGUF are all PTQ variants.

Quantization-Aware Training (QAT) injects simulated quantization during training. The model learns to live with low precision from the start. The result is better, but it requires retraining infrastructure. Most large labs use QAT for models they sell; most users use PTQ for models they download.

The analogy: PTQ is like compressing a photo to JPEG after taking it. QAT is like shooting directly in a smaller format, so the camera knows which details to keep.

GPTQ, AWQ, and GGUF: Three Dialects of the Same Thing

These three names show up together, and all three do the same thing: turn FP16 weights into INT4.

GPTQ (Frantar et al., 2022) works layer by layer. It measures the error that quantization introduces, then adjusts neighboring weights to compensate. The process needs calibration on a small amount of data, usually a few hundred text examples. The result is INT4 with quality close enough to FP16.

AWQ (Lin et al., 2023) starts from the observation that not all weights matter equally. A small fraction, around 1 percent, holds most of the information. AWQ protects those weights with higher precision, while the rest is quantized aggressively. The result is often better than GPTQ at the same size.

GGUF is a file format, not an algorithm. It was created by Georgi Gerganov for the llama.cpp project. It stores already-quantized weights with metadata, and supports several schemes (Q4_0, Q4_K_M, Q5_K_M, and others). GGUF is the reason models can run on an ordinary CPU, with no GPU.

Why the Model Does Not Always Get Dumber

The obvious question: if the numbers are simplified, do the answers become wrong?

It depends on what gets thrown away. Two things make quantization workable.

Redundancy. Large models have more parameters than they strictly need for a task. Some weights store the same information as other weights. Removing resolution from redundant weights does not change the output, because the same information survives elsewhere.

Distribution. Model weights are not spread evenly. Most weights are small, close to zero. A small fraction are large, and those large weights hold the important information. Good quantization keeps the large weights precise and simplifies the small ones.

This is why a 7B model quantized to INT4 often loses more quality than a 70B model quantized to INT4. The smaller model has less redundancy, so every bit removed is felt.

The KV Cache: The Hidden Cost That Is Bigger

Quantizing weights alone is not enough. When a model answers, it stores the key and value for every token it has already processed, so it does not recompute them. This cache, called the KV cache, grows with conversation length.

For a 70B model with a 32K token context window, the KV cache can reach tens of GB, sometimes larger than the model weights themselves. Quantizing the KV cache to FP8 or INT8 can cut that cost in half, with a small quality drop.

This is why modern AI servers often use two kinds of quantization at once: INT4 for weights, INT8 for the KV cache. That combination is what lets large models run on smaller hardware.

What Is Still Unsolved

Quantization is not a magic fix. Three things remain its limits.

Activation outliers. A small fraction of activations, the outputs between layers, have values far larger than the rest. Uniform quantization damages those values. Solutions like LLM.int8() from Dettmers et al. (2022) separate outliers into high precision, but that adds complexity.

Sensitive tasks. Math, code, and multi-step reasoning are more sensitive to quantization than ordinary conversation. A model that is good at chat can become bad at coding after INT4 quantization.

Honest evaluation. Standard benchmarks such as MMLU and GSM8K often miss quality drops that are real. A model that drops 2 points on MMLU can be much worse on real tasks. Good evaluation needs task-specific tests, not aggregate scores.

Analysis: Quantization Is Not Just Compression

Quantization is often read as a way to make a model smaller. That is true, but it is not the most important part. What matters more is what quantization reveals about the model itself. There are three things.

• Model size is not capability size. A 70 billion parameter model does not use all its weights equally. Most weights store redundant information that can be simplified without losing capability.
• Precision exposes training quality. Models trained on high-quality data survive quantization better, because their information is spread evenly. A model that breaks under quantization may simply be undertrained.
• Quantization changes who can run AI. Before quantization, running a large model meant renting expensive servers. After it, the same model runs on cheaper hardware.

What often gets missed: quantization makes models easier to run, but not smarter. A model quantized to INT4 is still the same model, with the same knowledge and the same biases. If the original model has a bias toward certain languages, quantization does not remove it. It just makes that bias cheaper to run. That is exactly why a model should be evaluated in Indonesian on its own, rather than trusted on an English benchmark score alone.

Numbers That Can and Cannot Be Quoted

Some of the numbers above come from original papers, some from community measurements.

• 140 GB for a 70B model at FP16: a calculation, not a measurement.
• QLoRA fitting 65B on one 48 GB GPU: claimed by Dettmers et al., 2023, and replicated by many.
• INT4 cutting size 4x: math, not a claim.
• 1-3 percent quality drop at INT4: depends on model and task, no standard figure.

What has no standard figure: how much quality drops on a specific task, and how much electricity a production server actually saves. Both depend on hardware, batch size, and workload.

What This Changes for Readers in Indonesia

For small developers and universities in Indonesia, quantization changes one thing: models that previously required a paid API can now run on your own server.

A 7B model quantized to INT4 needs about 4 GB of VRAM, which fits in a mid-range consumer graphics card. A 70B model quantized to INT4 needs about 35 GB, which can still run on one workstation with two 24 GB cards.

But there is a condition. A quantized model still needs good data. A 7B model trained on low-quality data will not get smart just by running at full precision. Conversely, a well-trained 7B model quantized to INT4 can be enough for everyday office tasks.

The question to answer before choosing: what task is actually being done, and is a small self-hosted model enough, or does the task need a large model only reachable through an API.