Local AI, End to End: How to Choose, Run, and Scale Your Own Stack

Two years ago, running a model on your own machine meant slow answers, a weekend of setup, and a tool you abandoned by Tuesday. That is over. Open weight models got good, the software that runs them got fast, and memory got cheap enough that a box on your desk now handles most of what people pay monthly subscriptions for.

This guide goes layer by layer, from hardware to scaling, but it is not a parts list. Most founders approach local AI as a shopping problem: which GPU, which model, which engine is best. That framing is the trap. There is no best. Every layer is a tradeoff, and the only question that matters is which constraint you are willing to accept. Name the constraint and the choice makes itself.

Everything in this guide hangs on one idea:

LOCAL AI = CAPACITY × BANDWIDTH × SOFTWARE

• capacity — what fits
• bandwidth — how fast it runs
• software — how much of that you actually get to use

Miss one of the three and the other two cannot save you. And the same question repeats at every layer below, from the model down to how you scale it: which bottleneck am I buying?

1. Start with the bottleneck, not the hardware

The wrong question is "which hardware is best?" The right one is "which bottleneck am I buying?" Three smaller questions answer it:

• What must fit?
• What speed tier do I need?
• What software can actually deliver it?

Capacity, the memory in your GPU or machine, decides whether a model loads at all. Bandwidth, how fast that memory can be read, decides how quickly the model produces text, because generating words is mostly about moving data, not raw computing power.

That is why a 32GB RTX 5090 produces text faster than a 512GB Mac Studio when a model fits on both. Lots of memory is not the same as lots of speed.

Here is the landscape, fastest first:

The blunt summary. NVIDIA for raw speed. Apple Ultra when you want huge amounts of memory in one quiet box without splitting work across cards. DGX Spark and Strix Halo as developer appliances. Tenstorrent if you care about a fully open stack. Thin laptops are fine for small assistants and useless for heavy multi step or long context work.

And the line every quick guide forgets:

Fitting is not serving. "It runs" is a demo. "It serves" is real engineering.

Even when a model loads, you still pay for memory speed while it types, for the growing memory cost of long context, and for the overhead of handling many users at once.

2. Choosing the model

The model is your first hard constraint. Its size and architecture set a floor on memory use and a ceiling on speed before you have tuned a single thing, so pick it after you know what fits at the quantization level you will actually run. The main open weight families — Llama, Qwen, Mistral, Gemma and DeepSeek — cover most needs. The right one depends on the job, not on leaderboard scores.

Two things to weigh:

• Dense vs mixture of experts. A dense model uses all of itself for every word. A mixture of experts model only switches on a slice each time, so it can be quick to run but heavy on memory and harder to serve well.
• Size vs speed. The smallest model that does your task properly is usually the better choice. It runs faster and leaves room for longer context and more users.

Be honest about scope. A good local setup handles roughly 80 percent of daily work: writing, coding, summarizing, answering questions about your own documents, automation. The hardest 20 percent — the deep reasoning and the trickiest multi step problems — still benefits from the big cloud models. That last fifth does not justify paying cloud prices for everything else, but pretending local covers all of it is how people end up disappointed.

3. The engine that runs the model

The engine is the software that turns hardware into throughput. It loads the model, manages the KV cache, schedules incoming requests, and serves an API your apps talk to. This is where you meet, or avoid, the throughput bottleneck: a basic runner answers one request at a time, while a real serving engine packs many into each pass. You pick it to match your hardware and workload, not the other way around.

A working guide, depending on where you are:

• Getting started, on a Mac, or on unusual hardware: llama.cpp. It runs almost anywhere. Ollama is a friendly wrapper on top, great for trying things and onboarding a team, but treat it as a developer tool, not a production server.
• Building natively on a Mac: MLX, built around Apple's shared memory.
• A single consumer NVIDIA card: ExLlamaV2. For two to four cards, ExLlamaV3.
• Serving real users in production: vLLM. The default starting point.
• The hard cases (very long context, mixture of experts, heavy traffic): SGLang.
• Maximum speed in an NVIDIA datacenter: TensorRT-LLM. You trade easy portability for raw performance.
• Coordinating a whole fleet: NVIDIA Dynamo.

One honest note: the simple servers built into llama.cpp and MLX are convenient but were never meant for production. They ship with little or no security and no real traffic handling. Do not confuse a capable local runner with a proper serving setup.

4. Settings that actually matter

You do not need to tune everything. Four settings move the needle:

• Quantization. Running a model at 4 bit or 8 bit instead of full size, called quantization, is what makes large models fit and run fast. The catch: the formats are not interchangeable. GGUF, AWQ, GPTQ, FP8 and NVFP4 each only run fast on an engine with kernels built for them. Pick the format your engine is good at, not the one a random benchmark used.
• Context. Context is how much of the conversation the model keeps in mind, and it is expensive because the KV cache — the running memory of that context — grows with both context length and batch size. It can exhaust your memory even when the weights fit comfortably. Set the context to what you actually need, not the maximum the model allows.
• Prefix caching. When many requests share the same opening, like a common system prompt, prefix caching reuses that work instead of recomputing it. Turn it on when your traffic repeats. It is close to free latency.
• Continuous batching. Packing many requests into each forward pass keeps the GPU fed and is what makes serving many users viable. It is the single biggest reason a real serving engine beats a basic local runner past one user.

5. The interface or the agent

Decide whether you want a chat window or a system that takes actions, because they are different things and they fail in different ways. Here the bottleneck stops being raw speed and becomes reliability.

An interface is a chat surface. Open WebUI gives you a private, ChatGPT-style window that connects to your engine, supports multiple users, and keeps every conversation on your own machine. For coding, Continue.dev inside VS Code gives you autocomplete and chat against your local model.

An agent is a loop. The model makes a plan, uses tools, reads the results, and acts, with some memory of what it is doing. That is meaningfully harder to run reliably than a chat box, and it is where most real problems show up. If you are building agents, the plumbing that connects the model to tools matters more than the look of the chat window.

6. The connections

A model that cannot touch anything is a toy. What turns it into infrastructure is a small stack of four things: tools to act with, a standard to deliver them, skills that package recurring work, and automation that puts it all into motion.

• Tools. A tool is a single capability the model can call: search the web, run code, query a database, send an email, hit an internal API. The model decides when to use one, reads what comes back, and keeps going. Tools are what turn a text generator into something that can act, and they are the unit you actually grant or withhold when you reason about safety.
• MCP (Model Context Protocol) has become the standard way to deliver those tools and data to a model: files, databases, online services, your own internal systems. Building around MCP keeps your integrations portable, so they keep working if you swap models or engines later.
• Skills. A skill is packaged expertise for a recurring task: a set of instructions, and sometimes scripts or templates, that the model loads only when that task comes up. Instead of stuffing every procedure into one bloated system prompt, you give the model a library it pulls from on demand. The payoff is a lean context and consistent behavior, the same way a function beats copying the same code into every file.
• Automation. A platform like n8n, hosted by you, wires the model into messaging, email, calendar, your CRM, spreadsheets, payments, and a thousand other services, with the model sitting inside the workflow as the part that thinks rather than a dumb trigger. A message comes in, the model reads your rules, an action happens, all on your hardware.

The detail that makes this painless: most engines speak the same connection format as the big cloud providers. Your existing code keeps working. You point it at your own machine instead of a cloud address and stop paying per word.

7. Using it safely

Local does not mean safe by default. There are two separate risks.

Exposure. Local servers usually ship with little or no password protection and were not built for the open internet.

• Never put the raw server directly online.
• Place it behind a gateway (Caddy or Nginx) that handles passwords and encryption.
• Keep it on its own protected part of your network.

Trust. The moment you connect tools and data, a new attack appears. Untrusted content — a web page, an email, a document — can try to trick your agent into doing something it should not.

• Give the agent the smallest set of tools and the least access it needs.
• Require a human confirmation for anything destructive.
• Keep your models and software updated, because weak spots get found here like anywhere else.

The upside is the whole reason to do this: your data stays on hardware you control. That ownership, and the privacy that comes with it, is the real value, more than the cost saving. On that note, the "three dollars a month" framing you see online is optimistic. Factor in the hardware, real electricity when the machine is busy, upkeep, and your own time. The economics are good. They are not free.

8. Sharing it with others

Going from one user to a team is mostly a networking and access problem, not a model problem.

Open WebUI handles multiple users out of the box, with a separate login and history for each person, all stored locally. Put the same gateway from the safety section in front of it.

To reach the machine from outside without opening holes in your firewall, you have two clean options:

• A private network mesh like Tailscale or WireGuard.
• A secure tunnel through a small relay server.

Either one lets your team reach a box at home or in a datacenter as if it were next to them, without exposing the server to the public internet.

9. Benchmarking: numbers you can actually trust

Benchmarking is how you find which bottleneck you actually bought, not the one you assumed. And a single number tells you almost nothing. "I got 180 tokens per second" is meaningless on its own, because it hides the model, the settings, the hardware, and how many people were using it.

Before you compare anything, write down five things:

• The exact model and its size.
• The quantization level.
• The engine and its version.
• The hardware (GPU, memory, bandwidth).
• The workload (prompt and answer length, and how many users at once).

The numbers worth measuring, in plain terms:

• Time to first word. How long you wait before anything appears.
• Tokens per second. How fast it types after that.
• The slow requests. Not the average, but the one in twenty that lags. That is what users complain about.
• Cost per million words. What tells you whether it pays off at scale.

Here are realistic single user figures to set expectations. Treat them as ballpark, because your exact result depends on your exact setup.

That last row is the whole lesson in one line. The 70B model fit far more easily on the 512GB Mac than on the GPU, yet it ran slower. Capacity got it loaded. Bandwidth set the speed.

A newer path is pooling small coherent appliances instead of buying one big GPU. NVIDIA's DGX Spark is built for this, and stacking them scales nicely:

Same lesson again: more boxes buys you bigger models and longer context, while the speed per word stays modest.

Concurrency flips the picture entirely. With vLLM serving 50 people at once, each person's speed drops, but the total across everyone climbs into the many hundreds of tokens per second. For serving a team, that total is what matters, not the single user figure.

Two practical notes. Quality settings like creativity barely change speed, but quantization level, context length, and how many requests you batch together change it a lot. And test again after every driver, engine, or model update, because the numbers move more than you would expect.

10. Scaling it

Scaling is a series of steps, and each one buys you a new bottleneck:

• One GPU. Where you start.
• More GPUs. Helps, but watch the interconnect. With a fast link like NVLink, tensor parallelism splits a model across cards cleanly. Without it, the cards spend their time waiting on each other and pipeline parallelism often wins instead. Either way, more cards never means proportionally more speed.
• Switch engines. Move off the friendly local runners onto vLLM or SGLang once you need real batching, monitoring, and reliability.
• Multiple machines. Now the connections between machines become the limit, and you bring in coordination software like Dynamo to route traffic, share memory across the fleet, and grow or shrink capacity automatically.

Whatever stage you are at, let the benchmarking section drive the decision. Measure against your real prompts and your real number of users, watch the slow requests, and scale the part that is actually the bottleneck, not the part that is easiest to throw money at.

The bottom line

The same theme runs through every layer. Speed is mostly about moving data and scheduling work well, and fitting is never the same as serving. There is no best GPU, best model, or best engine — only the bottleneck you choose to accept at each layer and the discipline to name it before you spend. Get that right and the stack designs itself.

ChatGPT is a website. A model is a model. Local AI, done properly, is infrastructure. And infrastructure is what actually builds companies.