Fitting GLM-5.3-Flash on one DGX Spark: measured bit placement

The budget. A DGX Spark has 121 GiB of memory visible to the operating system (marketed as 128 GB), shared between CPU and GPU. The box's own services take about 5 GiB and we keep 3 GiB free as a margin, so weights plus KV cache must fit in roughly 113 GiB. That puts the weights at roughly 108 GiB. Throughout, memory and GGUF sizes are in GiB; vendor figures and Hugging Face listings are in decimal GB.

The file we wanted. The uniform 3-bit mix, the Prime tier at 118 GiB, is 10 GiB over. Five attempts to load it on the Spark failed, and two of them froze the machine. In those attempts, expert tensors offloaded to the CPU side or mapped with -ngl 0 became resident rather than staying evictable, so the box ran out of memory either way.

Where the bytes go. About 95% of the parameters sit in the routed-expert tensors: three per MoE layer (gate, up, down), each 4096 × 2048 × 288 experts, 4.8 GB (4.5 GiB) apiece at BF16. Everything else, attention, the KDA linear-attention projections, the indexer, the router, the shared expert, the three dense layers and the embeddings, is about 9 GiB at 8-bit. The question is narrow: how many bits does each expert tensor get?

Stock tooling. Everything runs on llama.cpp master (commit 330f018, architecture glm5-next). No custom engine is involved and the output is ordinary GGUF.

Four steps. The FP8 checkpoint is converted to a BF16 GGUF, a 627 GB intermediate. The multi-token-prediction draft layer is dropped: the importance matrix has no statistics for it and it would cost several GB of the budget, so these files do not support MTP speculative decoding. llama-quantize writes the final file with per-tensor type overrides, using Unsloth's importance matrix as calibration data. The file is served by llama-server behind an OpenAI-compatible API with 16K context and the F16 vision projector. The Spark tier quantized in 74 minutes with 16 threads on the Spark; the uniform recipes took between 64 and 95 minutes.

How perplexity is measured. llama-perplexity on the wikitext-2 test split, at two settings: 20 windows of 2048 tokens and 30 windows of 512. The tool scores the second half of each window, so the settings score 20,480 and 7,680 tokens respectively, and the ± it reports is the standard error over scored tokens. Every figure is a single run. Two machines were used: the Spark (CUDA build) and an x86 workstation (Xeon w5-2455X, 125 GiB RAM, CPU-only build) for files that do not fit the Spark.

The low-bit types involved. Q8_0 is a block format: 32 weights, one fp16 scale, an 8-bit integer per weight. The k-quants (Q2_K, Q3_K, Q6_K) use 256-weight super-blocks with per-sub-block scales, still one scalar integer per weight. The IQ types quantize several weights jointly: IQ2_XXS and IQ2_S map each group of eight weights to a point from an E8-lattice codebook (256 and 1024 entries); IQ3_XXS maps each group of four weights to one of 256 codebook entries. Signs are stored separately, plus a 4-bit scale per 32 weights and an fp16 scale per 256. Quantizing a group of weights to a lattice point gives lower error per bit than rounding each weight to a scalar grid, which is why the IQ2 and IQ3 types outperform Q2_K and Q3_K at equal or smaller size.

Three starting points. We name each uniform recipe by its expert gate/up type. All three keep the core at Q8_0 and vary only the expert types.

The IQ3_XXS mix is the file we wanted. The IQ2_S mix fits with about 2 GiB to spare but is 0.286 of perplexity worse. The question became whether that gap could be closed inside the budget.

Method. We could not load the BF16 model to measure sensitivity directly, so we measured it between two finished quantized files. Take the IQ2_S mix, replace the gate/up expert tensors of one block of seven MoE layers with the IQ3_XXS mix's tensors (a raw tensor copy with gguf-py), and measure perplexity. Because llama-quantize quantizes each tensor independently, the spliced file is what the quantizer itself would have produced for that recipe. Each block costs 1.9 GiB, so every row buys the same bytes.

Result. The best block, layers 31–37, gained 0.100; the worst, layers 38–44, gained 0.034 for the same bytes. The gains are roughly additive: they sum to −0.363 against −0.286 for all six at once, a difference within the combined noise, with the mild sub-additivity one would expect. ±0.05 is the standard error of each absolute perplexity; the gains are paired differences on the same scored tokens and are more precise than that, though we did not compute the paired error. The ranking of the three middle blocks (0.057, 0.061, 0.069) is therefore not sharp; the two ends are.

Decision. Give 3-bit gate/up tensors to 31–37, the clear winner, and to 24–30, which measured best of the rest and is adjacent, so the upgraded region is one contiguous span of fourteen layers. Either of the other two middle blocks would have been within noise of that choice.

The shortfall. Upgrading two blocks costs about 4 GiB, and the IQ2_S mix left only about 2 GiB free.

The trade. The core, everything that is not a routed expert, was at Q8_0. Taking it to Q6_K, with the output tensor kept at Q8_0, frees about 2 GiB. The core is about 5% of the model. The cost of the change is not detectable in our measurements: the scan predicts 3.370 for the two upgraded blocks on a Q8_0 core (3.539 − 0.069 − 0.100), and the finished file with a Q6_K core measured 3.354 ±0.047.

Measured. 108.3 GiB. Perplexity 3.354 ±0.047 at 20 × 2048 and 2.765 ±0.060 at 30 × 512, about 23% above the BF16 reference at the latter setting. The Spark's CUDA build and the x86 CPU build agree within 0.012.

Served. With 16K context and the vision projector loaded, 5.0 GiB of memory was free after load, and one 300-token request decoded at about 14.6 tokens per second. One coding prompt (merge sort) and one image-description prompt were answered correctly.

One machine, one text. Every row below was measured on the x86 workstation with the same settings and 20 threads. Two llama.cpp builds were needed: commit 330f018 for all rows except Maya-M and the IQ2_S mix's 512-context figure, which were taken on master commit 186eab1 with --no-repack (section 11 says why). A cross-check of the IQ2_S mix on both builds agreed within noise. The Spark tier measured on the Spark's own GPU build gives 3.358 ±0.047 and 2.776 ±0.061.

Reading the table. The BF16 reference, 2.2474 at 30 × 512, is the figure published on the aj9o9/GLM-5.3-Flash-GGUF model card at identical settings; we could not run BF16 ourselves. Relative to the IQ2_S mix, the Spark tier closes about two-thirds of the gap to the 118 GiB file at 2048 context (0.185 of 0.286) and about three-quarters at 512, while staying 10 GiB smaller than that file.

The compared files. Maya-M, from Project Maya, is the most-downloaded GLM-5.3-Flash GGUF family on Hugging Face at the time of writing (about a million downloads on the hub counter, 6 November 2026). It uses GPTQ-style error-feedback rounding with per-expert statistics. Its gate/up expert tensors are IQ2_S in all 42 MoE layers; only the down tensors are upgraded, to IQ3_S in the first four and the last four MoE layers and IQ3_XXS elsewhere, over a Q6_K core. It also includes the MTP draft layer, which our files omit, so at the same file size it spends slightly less on the main network. GSQ-RCO uses learned quantization grids and a budget allocator, and places its extra bits in MoE layers 3–8.

What the difference is. Both are careful pieces of work aimed at different hardware, and their authors evaluate differently: zero-shot tasks, KL divergence to the original model, their own text. The rows above are therefore not a judgment on their published claims; they are what we measured on wikitext-2 with our builds. On that measurement, our scan says layers 24–37 are where the bits belong for this model, and that placement is the difference.

The idea. Keep the 118 GiB file and shrink it 8.5% losslessly by entropy-coding the codebook indices, so that it fits as it is. We measured the entropy of the IQ3_XXS expert tensors field by field.

The measurement. The indices are already close to uniform: 7.80 bits of entropy in an 8-bit field. The order-0 bound is a 3.2% saving; zlib at level 9 achieves 0.3%; a context model conditioned on the previous index, on the slot within the block or on the block scale reaches at most 2.3%. No repeated 32-weight groups exist across the 288 experts, so there is nothing to deduplicate.

Rejected. Even if the 8.5% had been available, the GPU kernels would have to decode in place. We report it because it is the obvious first idea and cheap to rule out.

• Quality is measured by wikitext-2 perplexity only, at two settings (20 × 2048 and 30 × 512), one run per figure. No coding or reasoning benchmark was run, and no KL divergence to the FP8 original was computed.
• The BF16 reference (2.2474 at 30 × 512) is taken from the aj9o9/GLM-5.3-Flash-GGUF model card at identical settings; we could not run BF16 ourselves.
• Unsloth's own quantizations at this size were not measured. The comparison covers the two community files we could run on one machine. GSQ-RCO (109.4 GiB) was measured on the workstation only and was not loaded on a Spark.
• Two llama.cpp builds were involved, as stated in section 7. Maya-M would not load on the older build.
• The ±0.05 on the block scan is the standard error of each absolute perplexity. The gains are paired differences on the same 20,480 scored tokens and are more precise than that figure suggests, but we did not compute the paired error.
• The sensitivity scan is at block granularity (seven layers) and covers the gate/up expert tensors only. Finer splits, the down tensors and the core were not scanned.
• The serving figures are from a single session: memory free after load, one 300-token decode run for the tokens-per-second figure, one coding prompt (merge sort) and one image-description prompt as functional checks.

1. Built from Z.ai's official weights. No pruning, no abliteration, no expert removal.
2. Bit placement chosen by direct measurement on this model, block by block, not by a rule of thumb.
3. Every compared file re-measured on one machine and one text, and published with error bars.
4. A negative result, reported.
5. Plain GGUF: runs in llama.cpp as tested, and in any runtime whose bundled llama.cpp has glm5-next support. Vision included.

Scope. The files, SHA-256 checksums and this recipe are on Hugging Face. The commands below are the ones that ran, with paths generalised. They assume a llama.cpp master build at commit 330f018 or later and the Unsloth importance matrix linked above.

The Maya-M build caveat. Maya-M quantizes the three-dimensional MLA attn_k_b tensor to Q6_K. On an AMX-capable CPU, llama.cpp's AMX extra buffer type claims Q6_K weights at load but cannot run that matrix multiplication, so the CPU build aborts at graph reserve. This is a limitation of the CPU backend, not a defect of the file; our files keep that tensor in BF16 and never hit it. --no-repack disables the extra buffer types, and Maya-M was measured that way on master 186eab1.

# FP8 safetensors -> BF16 GGUF (627 GB intermediate). The MTP draft layer is dropped.
python3 convert_hf_to_gguf.py GLM-5.3-Flash --outtype bf16 --no-mtp \
  --outfile GLM-5.3-Flash-BF16.gguf

# Spark tier, as run (74 min, 16 threads). Rules are regexes on lowercase tensor
# names and the FIRST match wins, so the layers 24-37 rule precedes the general one.
# The trailing IQ2_XXS is the default for anything unmatched; nothing falls through,
# but it is what the file's general.file_type metadata will report.
llama-quantize --imatrix imatrix_unsloth.gguf_file \
  --tensor-type "blk\.[0-2]\.ffn_=q6_k" \
  --tensor-type "blk\.(2[4-9]|3[0-7])\.ffn_(gate|up)_exps=iq3_xxs" \
  --tensor-type "ffn_(gate|up)_exps=iq2_s" \
  --tensor-type "ffn_down_exps=iq3_xxs" \
  --tensor-type "_shexp=q6_k" --tensor-type "attn_=q6_k" --tensor-type "ssm_=q6_k" \
  --tensor-type "indexer=q6_k" --tensor-type "ffn_gate_inp=q6_k" \
  --token-embedding-type q6_k --output-tensor-type q8_0 \
  GLM-5.3-Flash-BF16.gguf YogiQ-GLM-5.3-Flash-Spark.gguf IQ2_XXS 16

# Prime tier: the same command with every expert rule replaced by
#   --tensor-type "ffn_(gate|up|down)_exps=iq3_xxs"
# and no q6_k rules, so the core stays Q8_0.

# The published files are the quantizer's output split into 45 GB shards. The
# type token in the name is the base type of the mix (the hub reads it from the
# file name); the actual per-tensor types are the recipe above.
llama-gguf-split --split-max-size 45G YogiQ-GLM-5.3-Flash-Spark.gguf \
  Spark/YogiQ-GLM-5.3-Flash-Spark-IQ2_S

# wikitext-2 raw test split, the same two settings for every file in Table 3.
# llama-perplexity scores the second half of each window: 20 x 2048 scores
# 20,480 tokens, 30 x 512 scores 7,680. At 512 context pass -b 512: the default
# batch of 2048 packs four sequences per batch, and that path gives wrong numbers
# for this architecture on these builds. On the Spark GPU:
llama-perplexity -m MODEL.gguf -ngl 99 -fa auto -c 2048 -b 2048 --chunks 20 -f wiki.test.raw --no-warmup
llama-perplexity -m MODEL.gguf -ngl 99 -fa auto -c 512 -b 512 --chunks 30 -f wiki.test.raw --no-warmup

# On the x86 workstation (the rows of Table 3): the same without -ngl 99 -fa auto,
# with -t 20. Maya-M: master 186eab1 with --no-repack (see section 11).

# OpenAI-compatible API with the F16 vision projector, 16K context. Point
# llama-server at the first shard; it loads the remaining shards itself.
# After load on the Spark: 5.0 GiB of 121 GiB free, ~14.6 tokens/s decode.
llama-server -m Spark/YogiQ-GLM-5.3-Flash-Spark-IQ2_S-00001-of-00003.gguf \
  --mmproj vision/mmproj-GLM-5.3-Flash-F16.gguf \
  -ngl 99 -fa auto -c 16384 -np 1 --jinja --host 0.0.0.0 --port 8090 --alias glm-5.3-flash

Both files are published by YogiQ AI Labs Pvt Ltd, derived from Z.ai's GLM-5.3-Flash, and released under the same MIT license. They are quantizations, not fine-tunes: no weights were trained, pruned or edited, apart from the multi-token-prediction draft layer, which is not included (section 2). Work done 5–6 November 2026; published 6 November 2026.

Credits

• Z.ai

  GLM-5.3-Flash, the model, released under the MIT license.

• Unsloth

  The importance matrix used as calibration data.

• llama.cpp and ggml contributors

  The GGUF format, the quantizers, the kernels and the glm5-next architecture support.

• peasantsmith / Project Maya

  Maya-M, one of the two community files we compared against.

• pfeifferj / GSQ-RCO

  The GSQ-RCO 3.0-bit file, the other comparison.

• ISTA-DASLab

  GPTQ, the error-feedback rounding method Maya builds on.