◆ PLONKLAB plonklab.com ● NODE 1/1 · ONLINE
PLONKLAB
◂ back to the log
MSG #0008 2026-09-12 [B] Linux 9 min Linux & the Shell

LLMs and Sharks

The Shark

So I picked up a BLEShark from Infishark. Nifty little device. Off the bat I got to say the UI feels silky smooth. I like the fact you can connect via serial or use cli commands for basic functions. The SDK is for Rust which I am not familiar with, so I am working on a Python library for it. I'll toss it up on github once I feel it is working well enough. I think I'm going to order a three pack next payday and experiment with the Shiver mesh functionality. Oh and it has breakout. Totally worth it.

LLM

So I've been dinking around with local LLMs the past month or so. Been pretty successful with 27B-25B models using MoE... But man, I just got Qwen3.8-Flash-Next running. Fair warning, The guide below was written by my LLM.

Running Qwen3.8-Flash-Next (180B MoE) on a 16 GB GPU via CPU expert offload

A technical writeup on serving a 180-billion-parameter Mixture-of-Experts model on a single 16 GB consumer GPU using llama.cpp, by offloading the expert tensors to system RAM. The method yields ~18 tokens/second decode on the reference hardware. This document covers the mechanism, the exact configuration, a performance characterization, and the tuning that does and does not matter.

Summary of results

Model:                      Qwen3.8-Flash-Next  (180B total, ~6B active/token, MoE)
Quant:                      unsloth UD-IQ3_XXS  (82 GB on disk, 3 shards)
Decode throughput (warm):   ~18 tok/s  (DDR4-2933)   |  ~16 tok/s (DDR4-2133)
Prefill throughput:         ~77 tok/s
VRAM used:                  ~7 GB
RAM resident:               ~40-42 GB  (this is a hard floor, see section 7)

The workload is memory-bandwidth-bound, not compute-bound. Every conclusion below follows from that.

1. Model

Qwen3.8-Flash-Next is a first-party Qwen Mixture-of-Experts model, architecture qwen4exp:

  • 180B total parameters, of which only ~6B are active per token (a "180B-A6B" model).
  • MoE: the feed-forward layers are split into a large number of experts; a router selects a small subset per token.
  • Multimodal.

The serving quant used here is unsloth/Qwen3.8-Flash-Next-GGUF, tag UD-IQ3_XXS — an Unsloth dynamic quant, ~82 GB across three GGUF shards.

2. Method: MoE expert offload

An MoE model's parameters divide into two classes with very different access patterns:

  • Non-expert tensors (attention, embeddings, shared layers): used on every token, latency-sensitive, comparatively small. These belong on the GPU.
  • Expert tensors: enormous in aggregate, but only a few experts fire per token. These can reside in system RAM and be computed on the CPU.

llama.cpp exposes this partition with --n-cpu-moe N, which keeps the expert tensors of up to N layers on the CPU while -ngl offloads the remainder to the GPU. On the reference system this reduces GPU residency to ~7 GB of VRAM (hence the 16 GB card is not the constraint), while the expert weights occupy ~40 GB of RAM.

The trade being made is explicit: GPU memory capacity is exchanged for CPU memory bandwidth. The performance characterization in section 7 is entirely a consequence of that trade.

3. Reference system and requirements

Reference system:

  • GPU: NVIDIA RTX 5060 Ti, 16 GB
  • CPU: Intel i9-10900, 10 cores / 20 threads (Comet Lake)
  • RAM: 64 GB DDR4 (2×32 GB), running 2933 MT/s with XMP enabled

Minimum practical requirements:

  • GPU: ~8 GB VRAM or more (offloaded footprint is ~7 GB).
  • RAM: ~48 GB free minimum, 64 GB recommended — the resident footprint is ~40 GB and it cannot be reduced (section 7).
  • Disk: ~85 GB for the quant.
  • RAM speed: run it at its rated clock; decode throughput scales with memory bandwidth (section 7).

4. Build llama.cpp with CUDA

The qwen4exp architecture is supported by current llama.cpp. Build with CUDA:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

For a very recent GPU, set -DCMAKE_CUDA_ARCHITECTURES=<arch> to match the card.

5. Acquire the quant

hf download unsloth/Qwen3.8-Flash-Next-GGUF \
  --include "UD-IQ3_XXS/*" \
  --local-dir ./qwen3.8-flash-next

Set a token first (hf auth login); large multi-file pulls are rate-limited without one. Point llama.cpp at the first shard (...-00001-of-00003.gguf) — it resolves the full shard set automatically.

6. Runtime configuration

./build/bin/llama-server \
  -m ./qwen3.8-flash-next/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
  -ngl 99 \
  --n-cpu-moe 99 \
  --load-mode mmap \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  -c 16384 \
  -np 1 \
  -b 2048 -ub 512 \
  -t 10 \
  --jinja \
  --host 127.0.0.1 --port 8080

Flag reference (load-bearing flags only):

  • -ngl 99 — offload all layers to GPU; in combination with the next flag this means all non-expert tensors.
  • --n-cpu-moe 99 — the key flag: keep expert tensors on the CPU/RAM. This is what brings VRAM to ~7 GB.
  • --load-mode mmap — memory-map the weights; expert weights live as page cache in RAM.
  • -fa on — FlashAttention for the GPU-resident attention.
  • -ctk q8_0 -ctv q8_0 — 8-bit KV cache, halving KV memory.
  • -c 16384 — context length; raise it if RAM/VRAM allows (KV grows with it).
  • -np 1 — single sequence slot (protects the prompt cache from eviction; see 7.3).
  • -b 2048 -ub 512 — batch / micro-batch sizes for prefill.
  • -t 10 — CPU threads; set to the physical core count (section 8).
  • --jinja — use the model's chat template.

The server exposes an OpenAI-compatible API on the chosen port.

7. Performance characterization

7.1 Decode is memory-bandwidth-bound

Decode throughput approximates active_parameters / memory_bandwidth. The active experts are streamed from RAM on every token, so the memory bus is the limiting resource — not the CPU cores, not the GPU.

Evidence 1 — thread count has no effect past bus saturation. Sweeping -t:

-t 6     13-16 tok/s
-t 10    13-16 tok/s   (physical core count)
-t 20    13-16 tok/s

A flat curve is the signature of a bandwidth-bound workload: once enough threads exist to saturate the memory bus (~8 on this 10-core part), additional threads do nothing. Reports of large speedups from raising thread count come from compute-starved systems (few cores), where threads were the bottleneck; that result does not transfer to a system with spare cores.

Evidence 2 — throughput scales with RAM clock. The reference RAM defaulted to JEDEC 2133 MT/s. Enabling XMP (platform-clamped to 2933 MT/s, the i9-10900 memory-controller ceiling):

DDR4-2133   ~16 tok/s
DDR4-2933   ~18.3 tok/s     (+37.5% bandwidth -> +14% decode)

Bandwidth does not convert 1:1 to throughput (a compute/latency floor does not scale), but this is the single highest-leverage tuning available, and it costs only a BIOS setting. Verify RAM runs at its rated speed.

7.2 Memory footprint and the RAM floor

The resident footprint is ~40-42 GB and cannot be reduced without collapsing throughput. Constraining it forces the hot expert set out of RAM, causing per-token page faults from disk:

uncapped    ~18 tok/s     resident ~42 GB
24 GB cap   ~1.3 tok/s    page-fault thrash (hot set cannot stay resident)

(The 24 GB figure is included because some guides suggest pinning this class of model to 24 GB; on this quant that is not viable.)

Accounting note: under mmap, the ~40 GB appears as buff/cache (page cache), not as process "used" memory. free -h will misleadingly show low "used". Track available, not used. Operationally, do not plan to run other memory-hungry workloads (e.g. a diffusion pipeline) concurrently — they will contend for page cache and both will thrash.

7.3 Prefill cost and prompt caching

Expert offload slows prompt processing as well as decode: prefill measures ~77 tok/s. A large context (15-20K tokens) therefore incurs several minutes of prefill before the first token. Mitigations:

  • Prompt caching: turn 1 pays full prefill; subsequent turns reuse the cached KV and process only the delta, so they are fast. -np 1 keeps a single slot so the cache is not evicted.
  • If driving from an agent framework, raise the client timeout to survive the first turn.

Net: strong as an interactive chat model; weak as a large-context agent model, because the prefill tax is paid on every fresh context.

8. Tuning guidance

  • Threads: set -t to the physical core count. More threads do not help (7.1); fewer may under-saturate the bus.
  • RAM clock: enable the rated XMP/DOCP profile. Highest-leverage single change (7.1).
  • Context: size -c to available RAM/VRAM headroom; the 8-bit KV cache (-ctk/-ctv q8_0) roughly doubles the context you can afford.
  • Single slot: keep -np 1 for interactive use to preserve the prompt cache (7.3).

9. Approaches evaluated and rejected

The following were tested on this hardware and did not improve throughput. They are documented so others can skip them.

  • VRAM-resident expert cache (keeping hot experts on the GPU): ~0 gain. This model has 512 small experts, so a modest cache covers too little of the expert traffic to matter; the cache demonstrably loaded (~7.4 GB) but produced no speedup.
  • Speculative decoding / MTP draft: also a wash despite 92.9% draft acceptance. On a bandwidth-bound MoE, verifying two tokens costs ~2.2× a single token, because consecutive tokens route to different experts and batching cannot amortize the per-token expert bandwidth. Speculative decode only helps when experts are already VRAM-resident (i.e. on a much larger card).
  • Optimization forks: the fork whose published numbers motivated this test lists this model's own stock baseline at 16.6 tok/s — matching the mainline result here. The fork's gains do not apply to RAM-streamed experts on this hardware.

Conclusion: stock, mainline llama.cpp with the section 6 configuration achieves the full throughput on this hardware.

10. Operational notes

  • The method generalizes to other MoE GGUFs: the governing principle is that experts are large but rarely-used (→ RAM) while attention is small but always-used (→ GPU). --n-cpu-moe implements that split.
  • Throughput is set primarily by memory bandwidth and active-parameter count, not by GPU class. A faster GPU does not materially help while the experts are RAM-resident; more/faster system RAM does.
  • The model cannot be made to run lean. Provision RAM for the full ~40 GB resident set plus headroom.

Reference system: RTX 5060 Ti (16 GB), Intel i9-10900 (10c/20t), 64 GB DDR4-2933.

◂ back to the log