Hosted frontier models are where most of the work happens — they're the sharpest tools on the bench. But some jobs are a bad fit for sending your data to someone else's server, or for paying per token, or for a safety filter tuned so conservatively it refuses ordinary work. For those, I run open-weight models locally. This is how, and why.
§1What the local stack actually is
Nothing exotic — this runs on a laptop:
- The model: an open-weight mixture-of-experts model — roughly 35B total parameters but only ~3B active per token, so it punches well above its speed class. Quantized to 4-bit so it fits in memory.
- The runtime: MLX on Apple Silicon (the framework built for the Mac's unified memory), with a local server exposing an OpenAI-compatible endpoint — so any tool that speaks the standard API can point at it unchanged.
- The speed trick: speculative decoding — a tiny draft model proposes several tokens ahead and the big model verifies them in one pass, which meaningfully lifts tokens/second on local hardware.
- The driver: a small agent that talks to the local endpoint exactly like it would talk to a hosted one — same tool-calling, just a different
base_url.
point any OpenAI-compatible client at the local server
# a local server (LM Studio / an MLX server) exposes the standard API
BASE_URL = "http://localhost:1234/v1" # runs on your machine, offline
MODEL = "qwen-moe-35b-a3b" # the open-weight model you loaded
API_KEY = "not-needed-its-local"
# from here, the code is identical to calling a hosted model —
# same request shape, same tool-calls, just a different base_url.§2Why bother, when the cloud is smarter
The data never leaves the machine
For anything sensitive, "runs entirely offline on hardware I control" is a guarantee no cloud terms-of-service can match. No retention policy to trust because there's no transmission.
Zero marginal cost per token
Once it's on your disk, a million tokens costs electricity. For high-volume, low-stakes grinding — bulk drafting, classification, transforms — that changes what's economical to even attempt.
No rate limits, no outages
It works on a plane, in a dead zone, during a provider incident. The local tier is the one that's always up.
Refusals tuned for legitimate work
Hosted safety filters are calibrated for a global audience and sometimes refuse ordinary, lawful tasks. A locally-run model you're accountable for can be tuned to your actual, legitimate needs.
§3What "uncensored" actually means
This is the part people are curious about, so let's be precise and non-mysterious about it. An instruction-tuned model has two things bolted on after pre-training: it learned to be helpful, and it learned to refuse certain requests. The refusal behaviour is narrower and more mechanical than it looks.
Published interpretability research (notably work showing that refusal in language models is mediated by a single direction in the model's internal activation space) found that the "I can't help with that" behaviour is largely governed by one identifiable direction in the residual stream. An "uncensored" or abliterated model is one where that refusal direction has been damped — conceptually, the weights are adjusted so they no longer project onto it — so the model stops emitting the canned refusal and just answers. It is not retrained, not made "smarter," and not given new knowledge; one specific reflex is turned down.
The model names in this space wear their recipe on their sleeve — you'll see tags describing the quantization and the modification. The technique is openly documented; there are academic papers and public write-ups explaining the mechanism. What I'm deliberately not putting on this page is a step-by-step how-to, because the value here is understanding what the thing is, not a copy-paste guide to stripping guardrails.
Turning down a model's refusal reflex does not turn down your responsibility. The same laws, ethics, and judgement apply to the output whether or not the model would have refused first. I use an uncensored local model to get unblocked on legitimate, lawful work that an over-tuned filter fumbles — and a human reviews what comes out. Removing the reflex is not permission; it just moves the entire burden of judgement onto the operator, where, honestly, it always belonged.
§4How it fits the workflow
The local tier is a tier, not a replacement. Routing looks like this:
| Task | Where it runs | Why |
|---|---|---|
| Hard judgement, architecture, risky changes | hosted frontier model | you want the sharpest reasoning available |
| Sensitive data you can't send out | local model | privacy is non-negotiable — nothing leaves the box |
| High-volume, low-stakes grinding | local model | zero marginal cost changes what's worth attempting |
| Legitimate work a hosted filter over-refuses | local (uncensored) | get unblocked, then review the output yourself |
| Offline / provider outage | local model | the tier that's always up |
Because the local server speaks the standard API, switching tiers is a config change (see tuning the knobs) — the same agent, the same tools, a different base_url. The local model gets its own dials too: quantization (smaller/faster vs. more faithful) and context length (how much it can hold before your hardware chokes).
Steal this
- Keep a local tier for privacy, cost, and offline resilience — it's a laptop-scale setup now.
- Serve it behind an OpenAI-compatible endpoint so switching to it is just a
base_urlchange. - Use a mixture-of-experts model + speculative decoding to get usable speed on your own hardware.
- Understand "uncensored" for what it is: one damped refusal direction, not a smarter or retrained model.
- An uncensored model removes a reflex, not your responsibility. Keep a human on the output.