Serving LLMs on an RTX 5090: The Working Recipes

§1A 32 GB budget, not a 32 GB card

The model was a dense Qwen 3.8 27B checkpoint prepared in ModelOpt NVFP4 for the RTX 5090. Making the weights fit was only the first gate: the same 32 GB has to hold runtime workspaces, CUDA graphs, recurrent state, a speculative drafter when one is used, and the KV cache that grows with every token of conversation. The useful question is never "does it load?" — it is how much room remains for context and acceleration after the whole recipe loads.

The launchers also carried the model-specific reasoning and tool-call parsers. A generic template can return fluent text while quietly breaking structured tools or multi-turn agent work — so every candidate profile had to pass text generation, recall, tool calls, vision, and repeated stability checks before its speed number counted.

§2Four recipes, not one best server

vLLM and SGLang solved different parts of the problem, so the lab keeps four named profiles instead of forcing every workload through one set of flags:

§3The matched numbers

The cross-runtime comparison used the same local benchmark suite at 32,768 context, concurrency one, FP8 KV cache, and a warm loaded server. Each profile returned a real completion before the suite started:

Two honesty notes from the source dive. An older 203.1 tok/s D-Flash run was excluded from this comparison because it used a different 512-token test — real, but not matched. And a later, isolated vLLM tuning pass moved native MTP from three draft tokens to four, improving decode from 112.8 to 120.5 tok/s (+6.8%) with unchanged outputs — kept separate because it came from a later A/B, not the original cross-profile suite.

§4Context is three numbers, not one

The dashboard tracks three context figures separately: the requested context the launcher asks for, the declared model length the API reports, and the physical KV pool the loaded runtime can actually hold on this GPU. The effective limit is the smaller of the last two — an endpoint can advertise a large number while the loaded configuration has less space.

That is why the fastest recipe was not the longest: D-Flash 2 stayed the validated 32K speed profile, DSpark's lower footprint allowed 112K, and only the drafter-free vLLM profile could reserve the largest windows. Progressive three-needle recall tests recorded the real prompt length, not just the configured ceiling:

The 262K configuration's largest measured prompt was 178,014 tokens — the source dive deliberately does not call that a full 262K recall proof. What it does show is the cost curve: context that fits can still carry a substantial first-token penalty.

§5The two tuning traps

Trap one: the sensible flag that destroys decode. Raising DSpark's prefill chunk to 8,192 tokens — fewer chunks, faster prompts, obviously — silently disabled the draft model's CUDA graph. Decode collapsed from roughly 109 to 19 tok/s while speculative acceptance still looked healthy. The repair (more static memory) restored the graph but shrank the intended 114,688-token KV pool to 89,528. Both configurations were rejected; the winner was a 2,048-token chunk — in a later matched pass it cut median TTFT from 0.722 to 0.645 s on 8K prompts and 4.286 to 3.837 s on 35K prompts, lifted prefill 11.7% to 9,081 tok/s, and left decode flat.

Trap two: tokens per second is not answer speed. In chat tuning, some reasoning-heavy runs reported 240–285 decode tok/s while a roughly 115 tok/s non-thinking configuration finished the actual answer sooner. Measure prompt prefill, first visible token, decode, total time to a finished answer, acceptance, and correctness separately. Prefix caching produced a larger practical gain than most flag changes: a warm ~16K shared prefix cut TTFT from 2.891 s to 0.283 s — a 90.2% reduction — which is why stable system prompts matter; a timestamp in the prefix can invalidate the cache.

§6The KV-precision default

Every profile above runs an FP8 KV cache — in this serving work it was effectively free, and it is what makes the context math above workable on 32 GB. Going lower is a different bet: the engine's published table — measured on a larger reasoning model, not this build — shows a 4-bit KV cache halving a reasoning benchmark. Trust the direction, verify the magnitude, and treat KV precision as its own gated decision, exactly as the Apple Silicon field guide concludes from the Mac side.

§7A fast server that disappears is not a server

The next failures were operational, not computational: processes dying with their launch session, WSL terminating the runtime at idle boundaries, a GUI layer crashing in the background. The final setup used allowlisted named profiles, persistent startup, automatic tunnel reconnection, and a WSL configuration that keeps the compute instance alive.

Readiness stayed stricter than liveness — the system distinguishes configured, starting, ready, degraded, failed, and stopped, and a listening port is never enough: the gate requires model identity plus a bounded real completion. Access follows the same discipline: every inference listener stays on loopback, reached through an SSH tunnel over Tailscale, proven from an outside network at ~30 ms — no router port ever opened. The full remote-operations story is in the workstation build dive.

§8The questions people actually ask