العربية  

Download Book Qwen3 8 27b Nvfp4 On 4 Rtx 5090 Vllm Inference Benchmark Report Tensor Vs Data Parallel Mtp Speculative De

Qwen3.8-27B NVFP4 on 4× RTX 5090: vLLM Inference Benchmark Report (Tensor vs Data Parallel, MTP Speculative Decoding, 128K Context)
Qr Code Qwen3.8-27B NVFP4 on 4× RTX 5090: vLLM Inference Benchmark Report (Tensor vs Data Parallel, MTP Speculative Decoding, 128K Context)

Qwen3.8-27B NVFP4 On 4× RTX 5090: VLLM Inference Benchmark Report (Tensor Vs Data Parallel, MTP Speculative Decoding, 128K Context)

Author:
Category: Artificial Intelligence And Machine Learning [Edit]
Language: English
Pages: 42
File Size: 1.01 MB
Extension: PDF
Creation Date: 03 Nov 2026
Rank: 898,216 No 1 most popular
Short link: Copy
More books like this book
Reviews ( 0 )
Quotes ( 0 )
Score ( )

Book Description

A complete, reproducible inference benchmark of Qwen3.8-27B (Unsloth Dynamic NVFP4) served with vLLM 0.33.0 on a single node with four NVIDIA GeForce RTX 5090 GPUs. The report compares five serving configurations behind one OpenAI-compatible API: single GPU (TP=1), tensor parallel across 2 and 4 GPUs (TP=2, TP=4), data parallel replicas (DP=4), and a hybrid DP=2 × TP=2 layout.

Measured results cover single-user decode speed, short-context throughput sweeps from 1 to 256 concurrent requests, 128K-context concurrency limits, oversubscription and queueing behaviour, time-to-first-token on 128K prompts, maximum bootable context (up to the model’s native 262,144 tokens), needle-in-a-haystack retrieval at ~120K tokens, and a GSM8K accuracy sanity check. A dedicated section analyses multi-token-prediction (MTP) speculative decoding with a depth sweep (k=0 to k=4) and an MTP on/off comparison, showing where speculative decoding helps latency and where it costs throughput.

Key findings include a 3,167 tok/s peak throughput with DP=2 × TP=2, 186.7 tok/s single-user decode with TP=4 (245 tok/s greedy), 17 simultaneous 128K requests on TP=4 with zero preemptions, and the impact of missing GPU peer-to-peer on PCIe-only RTX 5090 cards. Every launch command, server flag, KV-cache size, GPU telemetry sample (utilisation, VRAM, power, temperature), and workaround for FlashInfer FP4 autotune and OOM issues is documented, so the results can be reproduced or adapted to other Blackwell consumer-GPU deployments.

Ideal for ML engineers, MLOps teams, and self-hosters sizing local LLM inference clusters, choosing between tensor and data parallelism, or planning long-context agent workloads on consumer GPUs.

Book Review "Qwen3.8-27B NVFP4 on 4× RTX 5090: vLLM Inference Benchmark Report (Tensor vs Data Parallel, MTP Speculative Decoding, 128K Context)"

Book Quotes "Qwen3.8-27B NVFP4 on 4× RTX 5090: vLLM Inference Benchmark Report (Tensor vs Data Parallel, MTP Speculative Decoding, 128K Context)"

Other books like "Qwen3.8-27B NVFP4 on 4× RTX 5090: vLLM Inference Benchmark Report (Tensor vs Data Parallel, MTP Speculative Decoding, 128K Context)"

Other books for "ENG Ahmed Saeed"

Hide Intellectual property is reserved to the author of the aforementioned book
If there is a problem with the book, please report through one of the following links:
Report the book or by Contact us

E-books are complementary and supportive of paper books and never cancel it. With the click of a button, the e-book reaches anyone, anywhere in the world.
E-books may weaken your eyesight due to the glare of the screen. Support the book publisher by purchasing his original paper book. If you can access it and get it, do not hesitate to buy it.
Publish your book now for free