Qwen3.5 on Dual R9700s

I benchmarked Qwen3.5-35B-A3B on shark, a local Ubuntu 24.04.4 workstation with 2x AMD Radeon AI PRO R9700 GPUs (gfx1201), using ROCm and vLLM 0.18.0+rocm700. The short version: yes, the model can be brought up and served across both GPUs, and the best warm run was actually pretty decent. The less cheerful part is that the stack is still fragile enough to ruin your mood if you mistake one clean response for stability.

The headline result is simple:

• The model can load and serve on dual R9700.
• Warm decode lands around 22.8 tok/s.
• A post-warm 1,036-token prompt returned first token in 0.422 s.
• The real blocker is not raw throughput. It is multi-process robustness on the current ROCm + vLLM tensor-parallel path.

At a glance

The chart above is the whole story in one screen: when the run is healthy, throughput is respectable. When it is unhealthy, you stop benchmarking and start doing crime-scene cleanup.

Testbed

Hardware

• Host: shark / Shark61
• OS: Ubuntu 24.04.4 LTS
• GPUs: 2x AMD Radeon AI PRO R9700
• GPU architecture: gfx1201

Software stack

• PyTorch: 2.9.1+git8907517
• HIP runtime reported by torch: 7.0.51831-a3e329ad8
• ROCm system stack: 7.1.x
• vLLM: 0.18.0+rocm700
• Model: /home/manoj/models/qwen35a3b

Launch config

• tensor_parallel_size=2
• cpu_offload_gb=20
• gpu_memory_utilization=0.90
• max_model_len=131072
• safetensors_load_strategy=eager
• enforce_eager=true
• FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
• VLLM_WORKER_MULTIPROC_METHOD=spawn

That config was not picked because it looked pretty. It was picked because the more aggressive versions tended to leave behind half-dead workers, poisoned VRAM state, and a bunch of fake optimism.

What a successful bring-up looked like

Metric	Value
Weight load time	35.24 s
Model load time	46.56 s
GPU memory used during model load	17.32 GiB
Available KV cache after load	6.62 GiB
GPU KV cache size	173,184 tokens
Claimed max concurrency at 131,072 tokens/request	5.13x

That memory split matters more than people admit. The dual-R9700 box can stand the model up, but only if you stop pretending every last GiB should be pressed into service. Leaving some allocator headroom is not cowardice; it is how you avoid turning restart loops into performance theater.

Request-level performance

The cleanest direct measurement from the rerun was a single successful post-warm streamed request:

Metric	Value
Prompt tokens	1,036
Completion tokens	128
TTFT	0.422 s
Total latency	6.028 s
Decode throughput	22.83 tok/s

Historical stable logs on the same machine told a consistent story:

• Prompt throughput bursts: roughly 4.18k to 5.05k tok/s
• Warm generation throughput: roughly 21.7 to 22.0 tok/s

So no, the hardware is not the punchline here. Once the run is healthy, the box moves tokens just fine.

Serving topology

The successful run used a pretty standard split: one vLLM API server, tensor parallel across both R9700s, plus 20 GiB of CPU offload. There is nothing exotic about the topology. The interesting part is that it works at all on this setup — and then sometimes falls over anyway.

What actually broke

This run exposed three separate failure classes.

1) Startup failure from stale VRAM

After a bad run, vLLM workers can stick around and keep VRAM reserved. In one failed restart, free memory on each GPU was only 10.11 / 29.86 GiB, while the requested utilization target wanted 29.26 GiB. That means the next startup dies before the real work even begins.

The fix is brutally simple:

• kill orphaned VLLM::Worker_TP*
• kill multiprocessing.resource_tracker leftovers
• only then relaunch

This is not a model-capacity issue. It is process hygiene.

2) Runtime tensor-parallel crash under stress

Under repeated or longer benchmark traffic, the engine died with:

• FileNotFoundError: /vllm_tp_gather_1_to_0
• followed by EngineDeadError

That points at the tensor-parallel shared-memory gather path. In plain English: the two-GPU serving path can come up, answer requests, and then still trip over its own IPC machinery.

3) Post-load worker death before readiness

A couple of relaunch attempts loaded weights, initialized caches, and then died before the server became healthy. So even the bring-up path is not yet deterministic. That is classic research-stack-pretending-to-be-a-product behavior.

If you only check whether /health returns 200, you will call this setup stable. That would be lazy and wrong.

Interpretation

My take is pretty simple:

1. The hardware is good enough.
2. The model is not the blocker.
3. The current ROCm + vLLM dual-GPU tensor-parallel path is the blocker.
4. The limiting factor right now is robustness, not throughput.

More specifically:

• Dual R9700 can host and run Qwen3.5-35B-A3B with CPU offload.
• Warm decode around 22 tok/s is already respectable for this class of model on this class of hardware.
• The operational enemy is not FLOPs. It is shared-memory coordination, worker lifecycle management, and cleanup after partial failure.

That is exactly the sort of result people miss when they benchmark only the happy path. The happy path exists. The unhappy path is just still too damn close to it.

Practical recommendations

If I had to keep using this machine tomorrow, I would do the following:

• Treat every failed run as contaminated until orphaned workers are killed.
• Use one benchmark-specific launch config instead of chasing max theoretical context every time.
• Keep gpu_memory_utilization conservative enough that restarts stay repeatable.
• Capture logs on every run, otherwise you will confuse stack fragility with model limits.
• Do not declare victory from one successful response.
• If the goal is reliable service rather than ROCm archaeology, test an alternate engine or patch the current vLLM TP gather path.

Bottom line

The lazy headline would be: “Qwen3.5-35B-A3B fails on AMD.”

That headline is garbage.

The honest headline is this:

Dual AMD Radeon AI PRO R9700 inference for Qwen3.5-35B-A3B is now real enough to benchmark, but not yet stable enough to trust under sustained long-context load.

That is still progress. A month ago the interesting question was “can this even load?” Now the interesting question is “how do we make the tensor-parallel ROCm path stop shooting itself in the foot?” That is a much better problem.