Instructions to use yacht/Qwen3.6-35B-A3B-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yacht/Qwen3.6-35B-A3B-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="yacht/Qwen3.6-35B-A3B-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("yacht/Qwen3.6-35B-A3B-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("yacht/Qwen3.6-35B-A3B-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yacht/Qwen3.6-35B-A3B-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yacht/Qwen3.6-35B-A3B-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yacht/Qwen3.6-35B-A3B-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/yacht/Qwen3.6-35B-A3B-W4A16

SGLang

How to use yacht/Qwen3.6-35B-A3B-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yacht/Qwen3.6-35B-A3B-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yacht/Qwen3.6-35B-A3B-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yacht/Qwen3.6-35B-A3B-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yacht/Qwen3.6-35B-A3B-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use yacht/Qwen3.6-35B-A3B-W4A16 with Docker Model Runner:
```
docker model run hf.co/yacht/Qwen3.6-35B-A3B-W4A16
```

Qwen3.6-35B-A3B-W4A16

INT4 (W4A16, group-128) GPTQ quantization of Qwen/Qwen3.6-35B-A3B, calibrated on Thai and checked for Thai text integrity at production context length. 67 GB → 21 GB.

Purpose

Run Qwen3.6-35B-A3B on a single 24 GB GPU (RTX 4090) without losing Thai quality.

Quantized MoE models tend to degrade low-resource languages before English shows any damage. Thai is an abugida — vowels and tone marks are combining codepoints — so damage appears as detached, doubled, or misordered marks while English still looks perfect. This build was calibrated and tested against that specific failure mode.

Calibration

GPTQ Hessian error-compensation genuinely reshapes weights at 4 bits, so calibration content matters here (unlike FP8, which is data-free). 497 documents / 1,131,013 tokens, 964,929 Thai (85.3%), all from sources permitting commercial use:

source	licence	share
Wikipedia (th)	CC-BY-SA-3.0	45%
C4 (th)	ODC-BY	35%
Thai stress shard — combining marks, tone marks, thanthakhat, sara am, Thai numerals ๐–๙	authored for this build	5%
Wikipedia (en) — control	CC-BY-SA-3.0	15%

moe_calibrate_all_experts=True, so all 256 experts see every calibration token — no expert is left with degenerate statistics. dampening_frac=0.01, max_seq_length=2048.

Attribution: C4 — Raffel et al., JMLR 2020, ODC-BY. Wikipedia — © Wikipedia contributors, CC-BY-SA 3.0.

Testing

Structural metrics over the Thai Unicode block (U+0E00–U+0E7F): orphan combining marks, marks opening a word, doubled tone marks, tone-before-vowel ordering, orphan leading vowels (เ แ โ ใ ไ), U+FFFD, lone surrogates. Scored against the BF16 original on identical prompts, with an English control so any damage must be shown Thai-specific.

suite	BF16	W4A16
thai_general	0	0
thai_stress	1	0
thai_domain	0	0
english_control	0	0

Long-context test — 8,685-token retrieval-style Thai prompt, the regime where corruption actually shows up:

arm	corrupted
greedy	0 / 24
sampled (t=0.7, top_p=0.8)	0 / 24
short-context control	0 / 24

Base-model behaviour at default sampling

At this model's own generation_config defaults (temperature 1.0, top_k 20, top_p 0.95), Thai responses contain a duplicated combining mark in roughly 3% of generations — inherited from the base model, not introduced by quantization. 90 generations per model, identical prompts:

model	corrupted
BF16 (unquantized)	3/90 — 3.3%
FP8	3/90 — 3.3%
W4A16 (this model)	4/90 — 4.4%

At n=90 these are statistically indistinguishable. At temperature 0.7, top_p 0.8 all three score 0/24. If you need structurally reliable Thai, lower the temperature or mask repeated combining marks with a logits processor — a different quantization will not help.

A combining mark with no Thai on either side counts as a citation, not a defect — when explaining Thai orthography the model correctly writes marks in isolation, e.g. เครื่องหมาย (์).

Serving — important for this architecture

MODEL=<ORG>/Qwen3.6-35B-A3B-W4A16      # or a local path

vllm serve "$MODEL" --quantization compressed-tensors \
  --max-model-len 32768 --max-num-batched-tokens 32768

Keep --max-num-batched-tokens above your longest prompt. 30 of this model's 40 layers are GatedDeltaNet linear attention carrying a recurrent state across the sequence rather than a KV cache. When a prompt exceeds the batched-token budget, chunked prefill splits it and that state must be carried across the boundary. A prompt of 8,537 tokens against --max-num-batched-tokens=8192 has been observed to produce duplicated Thai tone marks (ฟื้้ณภาพ for ฟื้นฟูสมรรถภาพ). Short prompts never show it. Raising the budget above the longest prompt avoids the split.

Also: do not set --kv-cache-dtype fp8 — it corrupts hybrid GatedDeltaNet models (vllm#37554). Default BF16 KV cache is correct.

W4A16 MoE routes through Marlin, which needs Ada (SM89) or Hopper (SM90). It crashes on A100 (SM80) — use an INT8 W8A8 build there.

Details

W4A16 group-128 via llmcompressor 0.11.0 GPTQModifier, compressed-tensors / pack-quantized format. 377 minutes on one H100.

Left in BF16: lm_head, embed_tokens, norms, mlp.gate (the MoE router — quantizing it perturbs expert routing across every downstream layer), shared_expert_gate, all 30 GatedDeltaNet linear_attn.* / conv1d layers, the vision tower, and the MTP head. The 10 full-attention layers (3, 7, 11, 15, 19, 23, 27, 31, 35, 39) have q/k/v/o_proj quantized normally.

Reproducing: oneshot on this multimodal MoE checkpoint mangles keys (llm-compressor#2568, still present in 0.11.0) — the text prefix repeats and the vision tower nests inside the text model, so vLLM raises KeyError: '…experts.w2_weight'. This checkpoint has been repaired: a pure rename, tensor data untouched, key count preserved 1:1. Experts are stored per-expert 2-D rather than fused 3-D — that is correct, vLLM fuses them at load.

Limitations

Inherits base-model behaviour, including occasional non-Thai token leakage into Thai output (e.g. Chinese 追回). Present in the BF16 original; not a quantization artifact.
Vision tower is BF16, so savings apply to the text backbone only.
Not instruction-tuned beyond the base model.
Testing covers greedy and sampled generation up to ~8.7k context. Longer contexts and streaming were not evaluated.

Downloads last month: 26

Safetensors

Model size

35B params

Tensor type

I32

BF16

Model tree for yacht/Qwen3.6-35B-A3B-W4A16

Base model

Qwen/Qwen3.6-35B-A3B

Quantized

(871)

this model