English  

كتب qwen3 8 27b nvfp4 on 4 rtx 5090 vllm inference benchmark report tensor vs data parallel mtp speculative decoding 128

اذا لم تجد ما تبحث عنه يمكنك استخدام كلمات أكثر دقة.

عرض المزيد

Qwen3.8-27B NVFP4 on 4× RTX 5090: vLLM Inference Benchmark Report (Tensor vs Data Parallel, MTP Speculative Decoding, 128K Context) (كتاب)


A complete, reproducible inference benchmark of Qwen3.8-27B (Unsloth Dynamic NVFP4) served with vLLM 0.33.0 on a single node with four NVIDIA GeForce RTX 5090 GPUs. The report compares five serving configurations behind one OpenAI-compatible API: single GPU (TP=1), tensor parallel across 2 and 4 GPUs (TP=2, TP=4), data parallel replicas (DP=4), and a hybrid DP=2 × TP=2 layout.

Measured results cover single-user decode speed, short-context throughput sweeps from 1 to 256 concurrent requests, 128K-context concurrency limits, oversubscription and queueing behaviour, time-to-first-token on 128K prompts, maximum bootable context (up to the model’s native 262,144 tokens), needle-in-a-haystack retrieval at ~120K tokens, and a GSM8K accuracy sanity check. A dedicated section analyses multi-token-prediction (MTP) speculative decoding with a depth sweep (k=0 to k=4) and an MTP on/off comparison, showing where speculative decoding helps latency and where it costs throughput.

Key findings include a 3,167 tok/s peak throughput with DP=2 × TP=2, 186.7 tok/s single-user decode with TP=4 (245 tok/s greedy), 17 simultaneous 128K requests on TP=4 with zero preemptions, and the impact of missing GPU peer-to-peer on PCIe-only RTX 5090 cards. Every launch command, server flag, KV-cache size, GPU telemetry sample (utilisation, VRAM, power, temperature), and workaround for FlashInfer FP4 autotune and OOM issues is documented, so the results can be reproduced or adapted to other Blackwell consumer-GPU deployments.

Ideal for ML engineers, MLOps teams, and self-hosters sizing local LLM inference clusters, choosing between tensor and data parallelism, or planning long-context agent workloads on consumer GPUs.