Fully local, private agent
A local OpenAI-compatible model server, an optional LiteLLM router, Pi as the harness, Basic Memory for on-disk Markdown memory, and Anthropic's sandbox runtime — all on your machine, with no hosted model API and no cloud sandbox.
Use case
Run a coding agent on your machine with a loopback model server, optional local router, local-first Markdown memory and an OS-level sandbox, without a hosted model API or a cloud sandbox.
Components
| Role | Component | Why |
|---|---|---|
| Local model server | Ollama | MIT local app and server that binds 127.0.0.1:11434 by default and exposes http://localhost:11434/v1. Swap in vLLM, SGLang, LocalAI or llama.cpp; start any of them with the companion guide. |
| Optional router | LiteLLM | MIT OpenAI-compatible proxy you can bind to 127.0.0.1:4000 and point at the local server. One client URL if you change backends. The proxy will not start in current releases without LITELLM_MASTER_KEY. |
| Harness | Pi | MIT terminal agent that documents models.json endpoints for Ollama, vLLM, SGLang and other OpenAI-compatible servers, plus a first-party llama.cpp router path. |
| Memory | Basic Memory | AGPL-3.0 local-first MCP memory stored as Markdown you keep on disk. Install with the documented local command; this stack does not use Basic Memory's hosted cloud path. |
| Sandbox | Anthropic Sandbox Runtime (srt) | Apache-2.0 OS-level sandbox (sandbox-exec, bubblewrap, or Windows srt-sandbox). Defaults deny network and writes until allowed; reads are allowed everywhere unless denied. Officially a beta research preview. |
When to use this stack
Use this stack when the agent should stay on your machine: the model is served on loopback, memory is files you hold, and tool processes are sandboxed locally. It is the opposite of Coding agent starter, which uses Claude Code as the harness, hosted MCP servers and E2B as a cloud microVM.
This page does not teach you how to start vLLM, SGLang, Ollama or LocalAI, or how to place LiteLLM in front of them. That is Run an agent on a local model with an OpenAI-compatible endpoint. Start there, then come back to wire the harness, memory and sandbox.
It is also not a substitute for Ollama vs llama.cpp for agents, LLM gateways compared, Memory layers for AI agents, Self-hosted sandboxes for agents or Give an agent long-term memory. Those pages compare alternatives; this stack picks one local set and shows how they fit.
The pieces
Model server. Start with Ollama if you want the default loopback bind (127.0.0.1:11434, per the FAQ) and an OpenAI-compatible URL at http://localhost:11434/v1. If you already run vLLM, SGLang, LocalAI or llama.cpp, keep that server; the companion guide shows the loopback flags. llama.cpp is a first-class option in Pi (see below) and is compared with Ollama on the serving-layer page, not repeated here.
Optional router. Put LiteLLM on 127.0.0.1:4000 when you want one client base URL in front of whichever local server you started. Current LiteLLM releases will not start the proxy without LITELLM_MASTER_KEY (or general_settings.master_key). Official Docker examples already use -p 127.0.0.1:4000:4000. Skip the router if the harness can talk to the model server directly.
Harness. Pi is a MIT terminal agent whose models page covers local GGUF files through a llama.cpp router (/llama) and OpenAI-, Anthropic- or Google-compatible endpoints in models.json. It names Ollama, LM Studio, vLLM, SGLang and other compatible servers as the models.json path. Pi is also one of the subjects in OpenClaw vs Hermes Agent vs Pi. You can swap in another local-capable harness from Terminal coding agents compared if you prefer; the rest of the stack does not depend on Pi-only APIs.
Memory. Basic Memory is an AGPL-3.0 local-first MCP server that stores memory as Markdown in a knowledge base you own. The documented local install is:
uv tool install basic-memory --prerelease=allow
The project README also offers a hosted cloud path ("Skip the install — try Basic Memory in the cloud"). That path is out of scope for this stack. Stay on the local install, and see Give an agent long-term memory and Memory servers for coding agents if you need to compare other memory layers.
Sandbox. sandbox-runtime (srt) is Apache-2.0 and officially a beta research preview: APIs and configuration may change. Install with the documented:
npm install -g @anthropic-ai/sandbox-runtime
It uses sandbox-exec on macOS, bubblewrap on Linux, and a Windows srt-sandbox user. The README's defaults: network is denied until you allow domains; writes are denied until you allow paths; reads are allowed everywhere unless you add a denyRead rule. For how this compares with other self-hosted sandboxes, and for the broader "treat output as untrusted" recipe, see Self-hosted sandboxes for agents and Run agent-generated code safely in a sandbox.
Wire it
- Start the model server on loopback, following the companion guide. Confirm you can complete a chat request with the OpenAI Python SDK against that
/v1URL. - Optionally start LiteLLM on
127.0.0.1:4000withLITELLM_MASTER_KEYset, pointingapi_baseat that same server. Clients then usehttp://127.0.0.1:4000/v1and the master key. - Point Pi at the endpoint. The official
models.jsonexample for Ollama (from the models page) is:
{
"$schema": "https://pi.dev/schemas/models.schema.json",
"providers": {
"ollama": {
"baseUrl": "http://localhost:11434/v1",
"api": "openai-completions",
"apiKey": "ollama",
"models": [
{ "id": "qwen2.5-coder:7b" }
]
}
}
}
The same page says the dummy key makes the model available to Pi and that Ollama ignores it. To go through LiteLLM instead, use the proxy's loopback /v1 URL as baseUrl and the master key as apiKey (via $LITELLM_MASTER_KEY or ${LITELLM_MASTER_KEY}, which Pi documents for environment interpolation). Swap the model id for one you have actually pulled; the qwen2.5-coder:7b value is Pi's example, not a model downloaded for this page. For GGUF files on disk, follow Pi's llama.cpp path rather than this JSON snippet.
- Add Basic Memory as a local MCP server in the harness, using the local install, not the cloud product.
- Wrap the agent's command and tool processes with
srtand a settings file that keeps network and writes on a default-deny list, then allow only the paths and hosts the agent should touch (for example the project directory, and127.0.0.1if the sandbox must reach the local model server).
Licences and limits
- Ollama, LiteLLM's open-source SDK and proxy, Pi and llama.cpp are MIT. vLLM and SGLang are Apache-2.0. LocalAI is MIT. LiteLLM Enterprise is a separate, quote-based product and is not part of this stack.
- Basic Memory is AGPL-3.0. If you distribute a modified server, that licence's conditions apply. Its hosted cloud offering is a different deployment, not this stack.
- sandbox-runtime is Apache-2.0 and a beta research preview. Do not assume a stable config schema.
- A "fully local" setup is only as private as its bind addresses, logs and tool egress. Leave every HTTP server on
127.0.0.1, and do not open sandbox network to the public internet "just in case".
Directory entries in this stack
- Ollama: MIT app and API for running open models locally, plus a paid Ollama Cloud with usage credits and per-token prices.
- LiteLLM: MIT Python SDK and self-hosted AI gateway that calls 100+ LLM providers in the OpenAI format. Paid Enterprise is separate and quote-based.
- Pi: Minimal, extensible coding-agent harness and TypeScript toolkit (unified LLM API, agent core, TUI) from Earendil; customize with extensions, skills and prompt templates.
- Basic Memory: Local-first MCP server that gives AI assistants persistent memory stored as Markdown files in a knowledge base you own.
- Anthropic Sandbox Runtime (srt): Lightweight OS-level sandbox enforcing filesystem and network restrictions on agents, MCP servers and arbitrary processes, without a container.
- vLLM: Apache-2.0 high-throughput LLM inference and serving engine with an OpenAI-compatible API, installable via uv or pip.
- SGLang: Apache-2.0 inference framework for LLM, vision-language and diffusion models, optimized for agentic workloads and large-scale serving.
- LocalAI: MIT self-hosted OpenAI-compatible engine that runs LLM, vision, voice and image backends locally, with Docker images as the documented install.
- llama.cpp: MIT C/C++ library and server for local LLM inference, widely used as a local OpenAI-compatible backend for coding agents.
- Claude Code: Anthropic's agentic coding tool that reads your codebase, edits files, runs commands and integrates with dev tools; available in terminal, IDE, desktop and browser.
- E2B: Open-source, secure cloud sandboxes for AI agents: an isolated machine per agent to run code, browse and use tools.
Related
- Run an agent on a local model with an OpenAI-compatible endpoint (Guide)
- Ollama vs llama.cpp for agents — tools, OpenAI APIs, grammars and local serving (Comparison)
- LLM gateways compared: OpenRouter vs Vercel AI Gateway vs Cloudflare AI Gateway vs Portkey vs LiteLLM (Comparison)
- Give an agent long-term memory (Guide)
- Run agent-generated code safely in a sandbox (Guide)
- Self-hosted sandboxes for agents compared (Comparison)
- Memory layers for AI agents (Comparison)
- Memory servers for coding agents compared: m3 Memory vs agentmemory vs Basic Memory vs MCP Memory Service (Comparison)
- Coding agent starter stack (Stack)
- OpenClaw vs Hermes Agent vs Pi: open-source agents you run yourself (Comparison)
- Terminal coding agents compared (Comparison)
Sources
- Ollama, OpenAI compatibility, accessed
- Ollama FAQ, accessed
- Pi docs, Models, accessed
- Pi docs, Local models with llama.cpp, accessed
- Pi docs, latest landing page, accessed
- LiteLLM docs, OpenAI-compatible endpoints, accessed
- LiteLLM docs, Config settings, accessed
- LiteLLM docs, Docker quick start, accessed
- Basic Memory README, accessed
- Anthropic sandbox-runtime README, accessed
- Anthropic sandbox-runtime LICENSE, accessed