Self-hosted inference

Local LLM

A personal project running large mixture-of-experts language models entirely offline, on a desktop with a single consumer-grade 8GB GPU. No cloud API, no subscription, no data leaving the machine — just a model file, an inference server, and two clients that talk to it.

llama.cpp CUDA Python GGUF Model Context Protocol Offline-first Try it in your browser ↓ View source on GitHub
Most of this page is still a write-up, not a demo. The three production models below, the GPU/CPU offload split, and the CLI/MCP tooling all run only on the author's own hardware and are not exposed to the internet — there is no live endpoint on this page connecting to any of that. The one part that is actually live: a small, entirely browser-side demo further down this page, where you can load a small GGUF file of your own and chat with it, no server involved on either end. Jump to it ↓

01 · Overview

What this is

The whole project rests on one idea: a model file on disk becomes a live HTTP server, and everything else — every tool, every interface — is just a client of that server.

Three GGUF model files sit on disk: Llama 4 Scout, Qwen3‑Coder‑Next, and Qwen3‑Next‑80B‑A3B‑Instruct. A GGUF file is inert: weights plus metadata, nothing more, until something loads it. That something is llama-server, the inference engine from the llama.cpp project, built from source with CUDA support rather than run through a wrapper like Ollama, for direct control over quantization, GPU/CPU offload, and context settings. Each running model gets its own llama-server process, and each one exposes an OpenAI-compatible HTTP API — the same POST /v1/chat/completions shape used by the OpenAI API and by most of the LLM tooling ecosystem. That compatibility is what lets a plain HTTP client speak to a local model exactly the way it would speak to a cloud one.

Two independent front ends sit on top of that API: a terminal chat client and a lightweight browser-based GUI. Neither one knows the other exists, and they don't share code — they share a contract. A model name always maps to one running server, and talking to it is always the same request shape. Any future front end (a different TUI, a small mobile client) would just be another client of that same contract; nothing about the inference layer would need to change.

02 · The offload trick

Fitting an 80B-parameter model into 8GB of VRAM

None of these models come close to fitting in 8GB of VRAM as a whole — their GGUF files run 43–65GB apiece. What makes them usable on a small GPU at all is that they're mixture-of-experts (MoE) architectures, combined with a specific offload recipe.

A dense model uses every one of its weights on every token it processes. An MoE model instead has a large bank of "expert" feed-forward sub-networks and a small routing mechanism that activates only a handful of them per token. Most of the model's bulk sits idle for any given token; only the attention layers and a small always-active shared/routing portion run on literally everything. Qwen3-Coder-Next and Qwen3-Next-80B-A3B-Instruct share the same underlying backbone — Alibaba's Qwen3-Next, roughly 80B total parameters with only about 3B active per token, one variant fine-tuned for code and the other instruct-tuned generally.

llama-server exposes two flags that exploit this directly: -ngl 999 tells it to offload every transformer layer it can to the GPU, and --cpu-moe overrides that for one specific category of tensor, pinning the MoE expert feed-forward weights to CPU RAM regardless. The combined effect is a deliberate split:

GPU · VRAM

Attention layers and shared/routing weights — the parts that run on every token — live in the scarce, fast memory where they benefit most from GPU throughput.

CPU · system RAM

The MoE expert weights — the bulk of the parameter count, but sparsely used by any single token — sit in cheap, plentiful RAM instead of competing for VRAM.

That's the whole trick that lets an 8GB consumer card meaningfully accelerate models many times larger than its own memory. Context length trades off against the same VRAM budget, so it's tuned per model: Scout's attention layers are larger (17B active parameters per token, across 16 experts), so its context window is trimmed further in GPU-offload mode to leave headroom, while both Qwen variants keep a larger window since their much smaller active-parameter footprint leaves more room to begin with.

There's also a plain CPU-only mode (-ngl 0, nothing offloaded) that never touches the GPU at all. It's slow — on the order of one token every few seconds for Scout and Qwen3-Next, and sometimes under one token per second for Qwen3-Coder-Next specifically, since its expert routing is expensive to do from CPU RAM — but it's the mode both front ends use by default, and a fully isolated way to sanity-check that a model and its chat template are working correctly before ever touching GPU offload.

03 · Model roster

The models

All three are quantized GGUF builds, chosen to cover a general assistant, a coding specialist, and a general-purpose sibling built on the coding model's own architecture.

Model Role MoE shape Size on disk
Llama 4 Scout General-purpose 16 experts · 17B active/token ~65GB (2 GGUF parts)
Qwen3-Coder-Next Coding specialist Qwen3-Next backbone · 80B total / ~3B active ~49GB
Qwen3-Next-80B-A3B-Instruct General-purpose Same backbone as above, instruct-tuned ~43GB

Qwen3-Coder-Next and Qwen3-Next-80B-A3B-Instruct are the same underlying model architecture, fine-tuned for different jobs — which is why they land in the same offload recipe and the same rough speed range on this hardware.

04 · Architecture

Top to bottom

  1. Model file (GGUF)
    Weights plus metadata on disk. Inert until loaded — does nothing by itself.
  2. llama-server
    Loads one GGUF file, applies the GPU/CPU offload split above, and exposes it as an OpenAI-compatible HTTP API. One process per running model.
  3. Shared API contract
    POST /v1/chat/completions, with the model's own embedded chat template applied automatically. Every client speaks this one shape.
  4. Front ends
    A terminal CLI and a browser-based GUI, both independent, interchangeable clients of the same contract.

05 · Terminal client

The CLI: a streaming REPL with real tool-calling

The base client is a straightforward terminal chat loop: a you> prompt, the reply streaming back token by token over server-sent events, and conversation history kept in memory across turns. /clear wipes history without restarting the server, /exit (or Ctrl+D) ends the session cleanly, and Ctrl+C interrupts mid-stream without leaving anything in a broken state.

An opt-in MCP tool-calling layer

A separate --tools mode layers a real Model Context Protocol client on top of that same chat loop, kept deliberately non-streaming and clearly marked as best-effort. On startup it connects, over stdio, to whichever MCP servers are enabled in a small config file, calls list_tools() on each, and merges the results into one tool schema the model can call against. Every tool is namespaced {source}__{name} — e.g. filesystem__read_file — which isn't cosmetic: a couple of the configured servers expose tools with names that would otherwise collide.

The safety gate is the same for every source, no exceptions: every tool call, before it executes, requires a manual y/n confirmation. There's no auto-execute path. A local filesystem server can read and write anything under its configured root, so that confirmation is the real safety boundary, not a formality. Servers that fail to connect don't take the CLI down with them — it prints one warning line and continues with whatever did connect.

An honest limitation, found rather than assumed: a merged tool schema can be bigger than these models' context windows. One configured browser-automation server alone measured out to roughly 4,400 tokens of schema; combined with a filesystem server's tool set, the merged schema came in at about 6,600 tokens against a live model running a 4,096-token context — over budget before a single word of conversation. The fix wasn't just a warning: that server now ships disabled by default until the context window is raised to accommodate it, and the CLI checks the connected server's actual context size at startup and warns proactively if the merged schema looks too large for it, instead of failing mid-conversation with a confusing error.

06 · Browser client

The GUI: a picker, not a rebuilt chat UI

Rather than build a custom chat interface, the GUI leans on the one llama.cpp already ships with each model server.

A tiny local picker page lists one entry per model. Selecting one launches that model's server, in CPU-only mode, if it isn't already running, then opens that model's own built-in llama.cpp chat page in an app-mode browser window — no tabs, no address bar, so it reads as a standalone app rather than a browser tab. If the picker's own backend isn't reachable for any reason, the page falls back to showing the exact command to run the same launch by hand, as a deliberate safety net rather than a missing feature. Once a model's chat window is open, it's just a plain web page talking to the same /v1/chat/completions contract as the CLI. This GUI path only ever launches CPU-only, by design; GPU-offload mode is something invoked directly, on purpose, not exposed through a click.

07 · Field notes

Three bugs worth remembering

A few things that came up building this, kept here because they're the kind of thing worth remembering the next time something looks similarly broken.

  • Wrong endpoint, garbled output

    llama-server exposes both a raw completion endpoint that takes a bare prompt string and an OpenAI-compatible chat endpoint that formats structured messages through the model's embedded template automatically. Hit the raw endpoint directly with a hand-written prompt and you skip that templating entirely — literal template control tokens and role markers leak into the visible output. Not a model defect, just a mismatch between the endpoint called and the input shape it expects. The fix is the rule now: always go through the chat-completions endpoint with a proper messages array.

  • Streamed emoji and accents turning to mojibake

    The model server's streaming response declares text/event-stream with no character-set parameter, so a naive HTTP client's default encoding guess can land on the wrong single-byte codec and mangle every multi-byte UTF-8 sequence — corrupting not just what's printed to the terminal but the in-memory conversation history itself. The fix was one line: explicitly force UTF-8 decoding on the response before iterating over it. Confirmed with a round-trip test asking a model to echo back a string full of accents and emoji and checking the bytes that came back.

  • A tool schema bigger than the context window

    Covered above in the CLI section, but worth restating as its own lesson: it's entirely possible for the description of the tools available to a model to exceed the model's own context window, with zero conversation in it yet. That's not an edge case to handle defensively "just in case" — it actually happened, on the first real attempt to combine two tool sources against a small-context model.

08 · Maturity check

Status

Not everything here is at the same level of maturity, so here's where each piece actually stands.

Verified

Core CLI

Streaming chat loop, history, and clean interrupt handling are working end to end.

Partially verified

MCP tool-calling

Config loading, connecting, schema merging, and the confirmation gate are all verified directly. A live model actually driving a full multi-round tool call against the merged schema is the next thing to confirm.

Verified

GUI picker

Launch requests confirmed to correctly start the right model server as a detached background process, including the already-running case.

09 · Live demo

Load your own GGUF and chat with it, right here

Unlike everything above, this part of the page is not a description of something running elsewhere — it actually runs, in this tab, right now. Pick a small GGUF file from your own machine below and it loads and runs entirely client-side, via wllama, a WebAssembly build of llama.cpp that runs inside the browser itself. Nothing you select is uploaded anywhere — there is no server on the other end of this page to send it to. This site is static file hosting, full stop.

This is not the 43–65GB production setup described above, and it can't be. A browser tab, running a single WebAssembly thread with no GPU offload, can realistically handle only a small quantized GGUF — roughly under 2GB, and even well under that will be noticeably slow compared to the CUDA-accelerated setup above. Point this at Scout or either Qwen model from the roster above and it will not work: it will hang the tab or run the browser out of memory long before it finishes loading. Use something genuinely small instead — a "tiny"/toy GGUF, or a small quantized instruct model in the hundreds-of-MB to low-single-digit-GB range.

Browser requirements, stated plainly. This demo deliberately disables wllama's own default fallback for browsers without the WebAssembly features it needs (JSPI and Memory64), because that fallback loads its files from a CDN, and this page makes zero requests to any host you didn't ask it to. The consequence: a browser without native support for those two features simply can't run this demo. Confirmed working in a recent Chromium-based desktop browser (Chrome); older browsers, and possibly Safari, may not work here at all — that's a real gap, not a hedge.

No model loaded.

Fixed for this demo: 2048-token context, 256-token replies, single WASM thread, no GPU offload.

10 · Try the real thing

Run the CLI yourself

The browser demo above is deliberately small and self-contained. The actual project it's standing in for — the terminal CLI described in section 05, talking to a real llama-server process on your own hardware — is real, runnable code, and the source is public. This section is the setup for running that, not the browser demo.

Get the source

github.com/coleman-sagil/local-llm is the real repository this whole page describes. 

git clone https://github.com/coleman-sagil/local-llm.git
cd local-llm
pip install requests

requests is the only third-party dependency the core chat client (cli/llmcli.py) imports; everything else it uses is Python standard library. The optional --tools layer additionally needs the MCP Python SDK: pip install mcp.

What the repo does and doesn't ship

Two directories that matter are deliberately not in the repo: models/ (the GGUF weight files themselves — tens of gigabytes, wrong tool for git) and llama.cpp/ (the inference engine, vendored and built from source, also gitignored). A fresh clone gets you the CLI, the GUI picker, and bin/start-model.sh/bin/stop-model.sh — but not a working llama-server binary or any models to point it at. To actually run something, you need your own compiled llama-server (build llama.cpp from source, CUDA support optional) and your own GGUF file, serving an OpenAI-compatible /v1/chat/completions endpoint.

Point the CLI at a server

cli/llmcli.py itself is a thin, generic client: it only knows how to POST to /v1/chat/completions on 127.0.0.1 at a fixed port per model name. It never starts a server itself. The three names it ships with are this project's own three models: 

scout      -> http://127.0.0.1:8090   (Llama-4-Scout)
qwen       -> http://127.0.0.1:8091   (Qwen3-Coder-Next)
qwen-next  -> http://127.0.0.1:8092   (Qwen3-Next-80B-A3B-Instruct)

Two ways to actually use it against your own setup: either edit the MODEL_PORTS dict at the top of cli/llmcli.py to add your own model name and port, or get your own llama-server listening on one of the three ports above and just use that model name as-is. If you have your own GGUF files at the exact paths bin/start-model.sh expects, that script will also start/stop the server for you: 

bin/start-model.sh scout --cpu-only   # or: qwen, qwen-next
python3 cli/llmcli.py --model scout
bin/stop-model.sh  scout

Once connected, it's the same REPL described in section 05: type at the you> prompt, the reply streams back token by token, /clear wipes conversation history without restarting the server, /exit (or Ctrl+D) ends the session, and Ctrl+C interrupts cleanly mid-stream. Add --tools to layer on the MCP tool-calling REPL described in the same section — non-streaming, every tool call gated behind a manual y/n confirmation, configured through your own cli/mcp_servers.json. 

python3 cli/llmcli.py --model qwen --tools

If the server isn't reachable, the CLI fails fast with the exact command to start it, rather than hanging or printing a stack trace — that behavior is real and unchanged from section 05.