Building My Personal AI Lab: RTX 5090, Qwen3.6 and Local Inference

For years, cloud infrastructure has been the easiest way to access serious AI compute. It is flexible, immediately available and ideal when demand changes quickly.

But experimentation feels different when every test has a running meter attached to it.

I’m now taking the next step and building a dedicated home AI lab: an environment where I can run open-weight models locally, build agentic workflows, test inference frameworks and understand the infrastructure beneath the applications.

§1The workstation

The core system is built around the following hardware:

The RTX 5090’s 32GB of VRAM will be the main inference engine. The larger system-memory pool provides room for datasets, multiple services, long-running workflows and CPU/GPU offloading when a model cannot remain entirely inside VRAM.

System memory is not a substitute for VRAM. Offloading allows larger models to run, but performance depends heavily on how much of the model stays on the GPU.

§2Why local compute?

This is not simply an attempt to replace every cloud API.

Frontier cloud models will remain extremely useful. Local infrastructure offers something complementary: control.

It gives me the ability to:

• Experiment without calculating the cost of every prompt
• Keep sensitive workflows and data inside my own network
• Run persistent agents without depending entirely on an external provider
• Compare models, quantisations and inference engines directly
• Fine-tune adapters using LoRA and QLoRA
• Test long-context and concurrent workloads
• Learn where local hardware performs well—and where the cloud still wins

Owning the hardware also changes the way I experiment. A model can remain loaded, services can run continuously, and ideas can be tested without first deciding whether the test justifies another usage bill.

§3The first major test: Qwen3.6-35B-A3B-MTP

One of the first models I plan to benchmark is Qwen3.6-35B-A3B-MTP.

It is a mixture-of-experts model with approximately 35 billion total parameters but only around 3 billion activated for each token. Instead of using every parameter during every forward pass, the router selects the experts most relevant to the current token.

That makes the architecture particularly interesting for local inference. The model retains a larger total capacity while requiring substantially less active computation per generated token than a similarly sized dense model.

The MTP version adds multi-token prediction.

Traditional autoregressive inference generates one token and then repeats the process for the next. MTP can draft multiple future tokens and verify them together. When the draft-acceptance rate is high and the inference runtime supports the implementation correctly, this can materially increase generation speed.

It is not a guaranteed multiplier. Performance depends on the model quantisation, prompt, context length, software implementation, MTP settings and how frequently the drafted tokens are accepted.

§4What I will benchmark

I intend to measure more than a single headline tokens-per-second result.

• Standard decoding versus MTP
• Prompt-processing speed
• Time to first token
• Sustained output tokens per second
• MTP draft-acceptance rate
• Q4, Q6, Q8 and FP8 configurations where supported
• Performance at 32K, 64K and 128K context
• VRAM and system-memory consumption
• Single-user versus concurrent inference
• Performance during sustained agentic workloads
• Power consumption, thermals and noise

I also want to compare the practical experience across llama.cpp, vLLM, SGLang, Ollama and LM Studio.

A benchmark that produces a high number for a short synthetic prompt may not be the best configuration for coding, tool use or a persistent agent. Reliability, prompt-processing latency, context handling and structured tool calls matter just as much as raw decoding speed. My existing local-model field notes use the same principle: compare complete jobs, not model personalities.

§5Building an agent environment

The broader objective is to use this workstation as a local intelligence layer rather than merely a chatbot server.

I plan to connect local models to Hermes and other agent frameworks, then evaluate:

• Parallel specialised agents
• Local coding and repository analysis
• Research and document-processing pipelines
• Private retrieval-augmented generation
• Automated content and operational workflows
• Local model routing based on task difficulty
• Hybrid workflows that escalate selectively to cloud models

The most interesting setup may ultimately be hybrid: fast local models handling routine work, with cloud models used only when a task genuinely benefits from them. That complements my persistent agent setup on Hetzner: the VPS supplies availability, while the workstation supplies local inference capacity.

§6What comes next

Once the system is assembled and stable, I will publish the real results—including the configurations that underperform.

Local-AI benchmarks are unusually sensitive to software versions, quantisation formats, context settings and decoding parameters. I would rather share reproducible measurements than advertise a theoretical maximum that only appears under one narrow test.

The first report will focus on Qwen3.6-35B-A3B-MTP on the RTX 5090, including actual throughput, MTP acceptance rates, memory consumption and performance inside a real agent workflow.

For anyone already operating a serious local setup: I would be interested to hear which combination of runtime, quantisation and MTP settings has produced the best practical results.