Instructions to use yacht/Qwen3.6-35B-A3B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yacht/Qwen3.6-35B-A3B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="yacht/Qwen3.6-35B-A3B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("yacht/Qwen3.6-35B-A3B-FP8")
model = AutoModelForMultimodalLM.from_pretrained("yacht/Qwen3.6-35B-A3B-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yacht/Qwen3.6-35B-A3B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yacht/Qwen3.6-35B-A3B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yacht/Qwen3.6-35B-A3B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/yacht/Qwen3.6-35B-A3B-FP8

SGLang

How to use yacht/Qwen3.6-35B-A3B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yacht/Qwen3.6-35B-A3B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yacht/Qwen3.6-35B-A3B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yacht/Qwen3.6-35B-A3B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yacht/Qwen3.6-35B-A3B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use yacht/Qwen3.6-35B-A3B-FP8 with Docker Model Runner:
```
docker model run hf.co/yacht/Qwen3.6-35B-A3B-FP8
```

Qwen3.6-35B-A3B-FP8

FP8 (E4M3) quantization of Qwen/Qwen3.6-35B-A3B, checked for Thai text integrity. 67 GB → 36 GB.

Quantized MoE models can degrade low-resource languages before English shows any damage. In Thai — an abugida, where vowels and tone marks are combining codepoints — that appears as detached or misordered marks. This build was verified against that specific failure mode.

Purpose

Run Qwen3.6-35B-A3B on 1× H100 80GB, or 2× RTX 4090 24GB, without losing Thai quality.

Calibration

None — this build is data-free. FP8 derives weight scales from weight max and computes activation scales per-token at runtime, so no calibration corpus is involved and none influenced these weights. Licence is therefore Apache-2.0, inherited from the base model, with no third-party data encumbrance.

Testing

Scored against the BF16 original on identical prompts, using structural metrics over the Thai Unicode block (U+0E00–U+0E7F): orphan combining marks, marks opening a word, doubled tone marks, tone-before-vowel ordering, orphan leading vowels (เ แ โ ใ ไ), U+FFFD, lone surrogates.

Structural defects, lower is better:

suite	BF16	FP8
thai_general	0	0
thai_stress	1	0
thai_domain	0	0
english_control	0	0

No Thai degradation relative to the BF16 original. Manually confirmed: correct thanthakhat (สิทธิ์ ศักดิ์ องค์), full Thai numeral range ๐–๙ (byte-fallback in this tokenizer, the most fragile decode path), correct tone marks and combining vowels.

A combining mark with no Thai character on either side is counted as a citation, not a defect — when asked to explain Thai orthography the model correctly writes marks in isolation, e.g. เครื่องหมาย (์). Without that exemption the checker punishes correct pedagogical text and, worse, scores a healthy model as more broken simply because it explained more characters. BF16 emitted 3 such citations here, FP8 one.

Long-context test — 8,685-token retrieval-style Thai prompt: 0/24 corrupted greedy, 0/24 sampled (t=0.7, top_p=0.8), 0/24 short-context control. BF16 scores identically.

Base-model behaviour at default sampling

At this model's own generation_config defaults (temperature 1.0, top_k 20, top_p 0.95), Thai responses contain a duplicated combining mark in roughly 3% of generations — and this is inherited from the base model, not introduced by quantization. 90 generations per model, identical prompts:

model	corrupted
BF16 (unquantized)	3/90 — 3.3%
FP8	3/90 — 3.3%
W4A16	4/90 — 4.4%

At temperature 0.7, top_p 0.8 all three score 0/24. If you need Thai output to be structurally reliable, lower the temperature or mask repeated combining marks with a logits processor — do not expect a different quantization to help.

Usage

MODEL=<ORG>/Qwen3.6-35B-A3B-FP8      # or a local path to this checkpoint

# 1× H100 80GB
vllm serve "$MODEL" --quantization compressed-tensors --max-model-len 40960

# 2× RTX 4090 24GB
vllm serve "$MODEL" --quantization compressed-tensors \
  --tensor-parallel-size 2 --max-model-len 16384

from vllm import LLM

llm = LLM(model="<ORG>/Qwen3.6-35B-A3B-FP8", quantization="compressed-tensors", dtype="bfloat16")

Needs Ada (SM89) or Hopper (SM90). On Ampere (A100) there are no FP8 tensor cores — vLLM dequantizes to BF16, giving quantization error with no speedup; use INT8 W8A8 there. Do not set --kv-cache-dtype fp8 — it corrupts hybrid GatedDeltaNet models (vllm#37554).

Details

FP8_BLOCK scheme via llmcompressor 0.11.0 — E4M3, block-[128,128] weights, dynamic per-token activations, compressed-tensors / float-quantized format.

Left in BF16: lm_head, embed_tokens, norms, mlp.gate (the MoE router — quantizing it perturbs expert routing across every downstream layer), shared_expert_gate, all 30 GatedDeltaNet linear_attn.* / conv1d layers, the vision tower (333 tensors), and the MTP head. The 10 full-attention layers (3, 7, 11, 15, 19, 23, 27, 31, 35, 39) have q/k/v/o_proj quantized normally.

Reproducing: oneshot on this multimodal MoE checkpoint mangles keys (llm-compressor#2568, still present in 0.11.0) — the text prefix repeats (model.language_model.language_model.language_model.…) and the vision tower nests inside the text model, so vLLM raises KeyError: '…experts.w2_weight'. This checkpoint has been repaired: a pure rename, tensor data untouched, key count preserved 1:1. Experts are stored per-expert 2-D (61,440 tensors) rather than fused 3-D — that is correct, vLLM fuses them at load.

Limitations

Inherits base-model behaviour, including occasional non-Thai token leakage into Thai output (e.g. Chinese 追回). Present in the BF16 original; not a quantization artifact.
Vision tower is BF16, so savings apply to the text backbone only.
Not instruction-tuned beyond the base model.

Downloads last month: 25

Safetensors

Model size

35B params

Tensor type

BF16

F8_E4M3

Model tree for yacht/Qwen3.6-35B-A3B-FP8

Base model

Qwen/Qwen3.6-35B-A3B

Quantized

(863)

this model