Quantization Explained: Run Open Source LLMs on Small Hardware

The problem quantization solves

A language model is basically a huge pile of numbers, weights, that get multiplied together billions of times per generated word. By default those numbers are stored as 16-bit floats. That's useful during training, but for actually running the model, it's overkill. The difference between a weight of 0.4127 and 0.4125 almost never changes what comes out.

That extra precision costs you twice.

Memory footprint. A 35-billion-parameter model at 16-bit needs roughly 70GB just to hold the weights, before you even load a conversation into context.

Speed. For most real-world inference, the GPU isn't limited by how fast it can do math. It's limited by how fast it can read the weights from memory on every single token. Smaller weights mean less to read, which means faster generation, not just a smaller file.

Quantization shrinks each weight down to fewer bits, 8, 4, sometimes lower, while keeping the model behaving the same way. Done well, you get a model 2 to 4x smaller and noticeably faster for a barely there quality cost. Done badly, the model gets visibly dumber. Almost all the engineering effort in this space goes into the "done well" part.

How it actually works, briefly

The naive approach, round every weight to the nearest value on a small grid, breaks quickly because weight distributions have outliers: a handful of unusually large values sitting among mostly small ones. One shared scale for everything gets stretched by those outliers, and all the ordinary small weights get crushed toward zero.

Modern quantization gets around this with a few tricks:

• Smaller scaling blocks. Instead of one scale per tensor, split it into small chunks and give each its own scale. An outlier in one chunk no longer wrecks its neighbors.
• Floating point grids over integer grids. A floating point low bit format spaces its values unevenly, finer near zero, coarser at the extremes, which matches how weights actually cluster, instead of an evenly spaced integer grid.
• Calibration. Methods like GPTQ or AWQ don't round weights blindly. They use a small sample of real data to quantize in a way that minimizes the actual error the model will see, and can leave a few especially sensitive weights at higher precision.
• No retraining needed. This all typically happens after training, as a one time pass over finished weights. That's why a quantized version of a new model shows up on Hugging Face within hours of release.

Quick refresher: bits and bytes

A bit is the smallest unit of storage there is, a single 0 or 1. A byte is 8 bits stacked together. More bits per value means more possible values it can represent, and more storage space it takes up.

A 16 bit number uses 16 of those 0s and 1s, which is 2 bytes. A 4 bit number uses only 4 of them, which is half a byte, since a byte doesn't split evenly in two for anything smaller than 4 bits either way. With 4 bits you can only represent 16 distinct values in total (2 to the power of 4), which is exactly why quantization needs a scale: 16 possible values isn't enough to represent a weight directly, but it's enough to represent how many scale-steps away from zero that weight is.

A worked example

Take a small block of four real weight values: [0.02, 1.35, 0.88, -0.15].

At 16 bit, each value costs 2 bytes, so this block costs 8 bytes total, no matter how small or large the numbers are.

To quantize this block to 4 bit, we first find a scale. Take the largest absolute value, 1.35, and divide it by 7 (the top of the 4 bit range): scale = 1.35 / 7 ≈ 0.193. Every value in the block gets divided by that scale and rounded to the nearest whole number:

• 0.02 / 0.193 ≈ 0.10 → rounds to 0
• 1.35 / 0.193 ≈ 7.00 → rounds to 7
• 0.88 / 0.193 ≈ 4.56 → rounds to 5
• -0.15 / 0.193 ≈ -0.78 → rounds to -1

Those four whole numbers, 0, 7, 5, -1, are what actually get stored in memory, each fitting in 4 bits instead of the original 16. That's where the saving comes from: not the number of values (still four), but how much room each one takes up.

When the model actually uses this weight to compute something, that small number gets temporarily multiplied back by the scale, only in that moment, not as something written back to storage: 0 → 0.000, 7 → 1.350, 5 → 0.965, -1 → -0.193.

Compare those recovered values to the originals: 0.02 became 0.000, 1.35 became 1.350, 0.88 became 0.965, -0.15 became -0.193. Small drift, nowhere near enough to change what the model does with these numbers. This row shows accuracy, not the storage saving.

The storage math shows the actual saving: four values at 4 bit is 2 bytes, plus one small scale number to remember (2 bytes), for 4 bytes total against the original 8 bytes (four values at 16 bit). That's already a 2x saving on a tiny block of just four numbers. Real blocks hold 16 to 32 values, so that one shared scale number costs almost nothing per value, and the savings get even better.

This also shows why block size matters: the scale of 0.193 was set by the single largest value, 1.35. If that block had instead been 32 values where only one was large and the rest were tiny, the same scale would have crushed most of those tiny values down to 0. Keeping blocks small is what keeps that outlier's damage contained to its own neighborhood instead of the whole tensor.

NVFP4: the format DGX Spark is built around

DGX Spark's GB10 chip has Tensor Cores with native support for 4 bit floating point, in NVIDIA's own flavor called NVFP4. A few things make it a genuinely good implementation rather than just "4 bit and hope":

• Small 16 value scaling blocks, so scale factors track the data closely
• Two layers of scaling (a per block scale plus a per tensor scale) for both local precision and overall range
• A floating point grid rather than a uniform one
• Native execution on the Tensor Cores, no decompressing back to higher precision before every matrix multiply

Versus 8 bit, NVFP4 roughly doubles throughput and cuts memory by another 45%, typically for less than 1% accuracy loss on major models. That's the combination that makes FP4 the headline feature of Blackwell class hardware rather than a party trick.

Why this matters specifically on DGX Spark

DGX Spark packs a 20 core Arm CPU and a Blackwell GPU sharing 128GB of unified memory into a desktop box, with up to 1 petaFLOP of FP4 compute. NVIDIA advertises it as capable of running models up to 200B parameters, a claim that only holds up because of quantization. At 16 bit, a 200B model needs about 400GB, more than three times what the box has. At 4 bit, it needs roughly a quarter of that, and comfortably fits.

Quantization here isn't an optimization you bolt on afterward. On unified memory, bandwidth limited hardware like this, it's the thing that makes running a serious model locally possible at all. That's exactly the pitch behind UPPR's data sovereignty angle: you don't need a data center to run real open source models entirely on premise, if the model is quantized properly for the hardware underneath it.

Putting it to the test: Qwen3.6-35B-A3B on our DGX Spark

To make this concrete, here's an example running on our own hardware. We run Qwen3.6-35B-A3B on our DGX Spark through vLLM, and it's a good pick to illustrate the general point.

It's a sparse Mixture of Experts model: 35B total parameters, but only about 3B active per token (8 routed experts plus 1 shared, out of 256). Inference cost tracks the active parameters, not the full model, so it's already efficient before quantization even enters the picture.

It also uses hybrid attention, mostly linear attention (Gated DeltaNet) with regular attention layers mixed in, which keeps long context cheap, and supports a native 262K token context. On top of that it posts strong agentic and coding benchmark scores, relevant to the kind of agent work UPPR builds for clients.

Running the NVFP4 build instead of the full precision one drops weight memory from roughly 70GB down to around 19 to 20GB. That's the difference between "barely fits, no room for context" and "comfortable, with headroom for long conversations and multiple users at once."

Our actual numbers

This isn't theoretical. It's our live setup, benchmarked on our own DGX Spark.

111.1 tok/s output, on nvidia/Qwen3.6-35B-A3B-NVFP4

Prefill: 5,000.3 tok/s · Time to first token: ~500ms · Peak VRAM: 79.4GB of 128GB

Getting there took a specific configuration, not just a default vllm serve. Marlin as the MoE backend, FlashInfer for attention, native MTP speculative decoding (3 speculative tokens, Triton backend for the MoE part of that path), chunked prefill, prefix caching, and async scheduling all switched on, with GPU memory utilization capped at 65% to leave headroom.

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --host 0.0.0.0 --port 8000 \
  --served-model-name Qwen3.6-35B-A3B \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 4 \
  --moe-backend marlin \
  --trust-remote-code \
  --language-model-only \
  --gpu-memory-utilization 0.65 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --dtype auto \
  --quantization modelopt \
  --kv-cache-dtype fp8 \
  --load-format instanttensor \
  --attention-backend flashinfer \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  -tp 1 -pp 1 \
  --default-chat-template-kwargs '{"preserve_thinking":true}' \
  --generation-config vllm \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

111 tok/s output on a desktop box, on a model that would need a real GPU server at full precision. That's the whole argument for quantization made concrete.

What actually broke, and what that teaches

Getting a clean deployment running on brand new hardware rarely goes in a straight line, and it's worth being honest about that instead of pretending it just worked.

• OOM freezes. Unified memory means CPU and GPU share one pool. A runaway allocation can lock up the whole box, not just slow it down.
• CUDA driver mismatches. Blackwell support is new enough that driver and toolkit versions need to line up carefully.
• Stale kernel cache. vLLM compiles custom kernels for your specific quantization setup. An old cache can silently serve the wrong one.
• Disk exhaustion from Docker images. Easy to overlook on a "desktop supercomputer," but the drive fills up fast once you're pulling multiple model variants and container layers.

None of these are bugs in quantization itself. They're the normal growing pains of running a bleeding edge stack on bleeding edge hardware. But they're the real cost behind "run a 200B capable model on your desk," and worth budgeting time for.

The takeaway for SMBs evaluating local AI

The headline claim, a 200B parameter model running on a desktop, is true because of quantization, not despite it. That distinction matters if you''re deciding whether local, sovereign AI is realistic for your organization.

You''re giving up far less model quality than intuition suggests. Well executed 4 bit quantization costs low single digits of accuracy on most tasks, if that. The hardware bar for "good enough" local inference is lower than it looks, when the serving stack is built around the quantization format properly. And the real friction is operational: drivers, kernels, storage, memory tuning. All solvable, but real, and better planned for than discovered mid deployment.

That''s the gap UPPR closes for clients. Not just picking a model, but making the quantization, hardware, and serving stack actually hold together in production.

Part of UPPR''s ongoing series on local AI infrastructure and the DGX Spark. Questions about running open source models on premise for your organization? Get in touch.