Qwen3.5-9B

GGUF Q4_K_M · llama.cpp · 64K context · Radeon RX 6800 XT 16 GB

Alibaba's 9B model from February 2026: small enough for an 8 GB card, with a 64K–128K window. It thinks, calls tools and stays quick on modest hardware.

67 tok/sdecode

– tok/sprefill

64Kcontext window

Earlier checkSep 10, 2026

Run it

With Omarchy Local AI it is one button. By hand, it is three steps: the weights, the config, the container. The server then answers on http://localhost:8000/v1.

1 Download the weights

hf download unsloth/Qwen3.5-9B-GGUF \
  --revision 3885219b6810b007914f3a7950a8d1b469d598a5 \
  --local-dir ~/models/Qwen3.5-9B-GGUF-3885219b

2 Start the server

docker run --rm \
  --device /dev/kfd --device /dev/dri \
  -p 8000:8080 \
  --shm-size 8g \
  -v ~/models/Qwen3.5-9B-GGUF-3885219b:/models:ro \
  ghcr.io/ggml-org/llama.cpp@sha256:9e4536dfc7ec47fb9be2db36b25cca1eab45667509e3a1938356268d10d8d2cc \
  -m /models/Qwen3.5-9B-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --ctx-size 65536 \
  --n-gpu-layers 999 \
  --flash-attn auto \
  --jinja \
  --reasoning-format deepseek \
  --parallel 1

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Passed the older check (loads and chats); a full six-check run is pending.

Details

Weights: unsloth/Qwen3.5-9B-GGUF @ 3885219b68 ›
Image: ghcr.io/ggml-org/llama.cpp@sha256:9e4536dfc7ec47fb9be2db36b25cca1eab45667509e3a1938356268d10d8d2cc
Engine profile: llama.cpp-qwen3.5-9b-q4-k-m-64k ›
Recipe file: registry/recipes/amd/rx-6800-xt-16gb/qwen3.5-9b.llama.cpp.64k.json ›
Model released: Feb 26, 2026
Tested: earlier acceptance