{
  "type": "guide",
  "id": "run-an-agent-on-a-local-model",
  "title": "Run an agent on a local model with an OpenAI-compatible endpoint",
  "summary": "Start Ollama, vLLM, SGLang or LocalAI bound to 127.0.0.1, point the OpenAI Python SDK at the server's /v1 base URL, enable tool calling, and optionally route the same endpoint through a LiteLLM proxy config.",
  "author": "Agentica Author",
  "tags": [
    "inference",
    "local-models",
    "openai-compatible",
    "tool-calling",
    "litellm",
    "agents"
  ],
  "published": "2026-11-15",
  "last_verified": "2026-11-15",
  "difficulty": "intermediate",
  "time_estimate": "45 min",
  "entries": [
    "ollama",
    "vllm",
    "sglang",
    "localai",
    "litellm",
    "llama-cpp",
    "pi-coding-agent"
  ],
  "links": {
    "html": "https://indexagentica.com/guides/run-an-agent-on-a-local-model/",
    "markdown": "https://indexagentica.com/guides/run-an-agent-on-a-local-model.md",
    "json": "https://indexagentica.com/api/longform/guides/run-an-agent-on-a-local-model.json",
    "source": "https://github.com/Drudley/indexagentica/blob/main/content-long/guides/run-an-agent-on-a-local-model.md"
  },
  "status": "published",
  "prerequisites": [
    "Python 3.10 or newer (for the LiteLLM proxy extra; this box ran 1.109.1 on Python 3.13)",
    "A local OpenAI-compatible server already installed, or a documented install to follow (Ollama, vLLM, SGLang or LocalAI). This guide does not install vLLM or SGLang, which typically need a GPU.",
    "No cloud account or paid API key is required for the loopback path"
  ],
  "entries_detail": [
    {
      "id": "ollama",
      "name": "Ollama",
      "summary": "MIT app and API for running open models locally, plus a paid Ollama Cloud with usage credits and per-token prices.",
      "url": "https://indexagentica.com/entries/ollama/",
      "json": "https://indexagentica.com/api/entries/ollama.json"
    },
    {
      "id": "vllm",
      "name": "vLLM",
      "summary": "Apache-2.0 high-throughput LLM inference and serving engine with an OpenAI-compatible API, installable via uv or pip.",
      "url": "https://indexagentica.com/entries/vllm/",
      "json": "https://indexagentica.com/api/entries/vllm.json"
    },
    {
      "id": "sglang",
      "name": "SGLang",
      "summary": "Apache-2.0 inference framework for LLM, vision-language and diffusion models, optimized for agentic workloads and large-scale serving.",
      "url": "https://indexagentica.com/entries/sglang/",
      "json": "https://indexagentica.com/api/entries/sglang.json"
    },
    {
      "id": "localai",
      "name": "LocalAI",
      "summary": "MIT self-hosted OpenAI-compatible engine that runs LLM, vision, voice and image backends locally, with Docker images as the documented install.",
      "url": "https://indexagentica.com/entries/localai/",
      "json": "https://indexagentica.com/api/entries/localai.json"
    },
    {
      "id": "litellm",
      "name": "LiteLLM",
      "summary": "MIT Python SDK and self-hosted AI gateway that calls 100+ LLM providers in the OpenAI format. Paid Enterprise is separate and quote-based.",
      "url": "https://indexagentica.com/entries/litellm/",
      "json": "https://indexagentica.com/api/entries/litellm.json"
    },
    {
      "id": "llama-cpp",
      "name": "llama.cpp",
      "summary": "MIT C/C++ library and server for local LLM inference, widely used as a local OpenAI-compatible backend for coding agents.",
      "url": "https://indexagentica.com/entries/llama-cpp/",
      "json": "https://indexagentica.com/api/entries/llama-cpp.json"
    },
    {
      "id": "pi-coding-agent",
      "name": "Pi",
      "summary": "Minimal, extensible coding-agent harness and TypeScript toolkit (unified LLM API, agent core, TUI) from Earendil; customize with extensions, skills and prompt templates.",
      "url": "https://indexagentica.com/entries/pi-coding-agent/",
      "json": "https://indexagentica.com/api/entries/pi-coding-agent.json"
    }
  ],
  "related": [
    {
      "type": "stack",
      "id": "fully-local-private-agent",
      "title": "Fully local, private agent",
      "url": "https://indexagentica.com/stacks/fully-local-private-agent/",
      "json": "https://indexagentica.com/api/longform/stacks/fully-local-private-agent.json"
    },
    {
      "type": "comparison",
      "id": "local-llm-servers-for-agents",
      "title": "Ollama vs llama.cpp for agents — tools, OpenAI APIs, grammars and local serving",
      "url": "https://indexagentica.com/compare/local-llm-servers-for-agents/",
      "json": "https://indexagentica.com/api/longform/compare/local-llm-servers-for-agents.json"
    },
    {
      "type": "comparison",
      "id": "llm-gateways",
      "title": "LLM gateways compared: OpenRouter vs Vercel AI Gateway vs Cloudflare AI Gateway vs Portkey vs LiteLLM",
      "url": "https://indexagentica.com/compare/llm-gateways/",
      "json": "https://indexagentica.com/api/longform/compare/llm-gateways.json"
    },
    {
      "type": "comparison",
      "id": "personal-agent-runtimes",
      "title": "OpenClaw vs Hermes Agent vs Pi: open-source agents you run yourself",
      "url": "https://indexagentica.com/compare/personal-agent-runtimes/",
      "json": "https://indexagentica.com/api/longform/compare/personal-agent-runtimes.json"
    },
    {
      "type": "comparison",
      "id": "coding-agent-harnesses",
      "title": "Terminal coding agents compared",
      "url": "https://indexagentica.com/compare/coding-agent-harnesses/",
      "json": "https://indexagentica.com/api/longform/compare/coding-agent-harnesses.json"
    }
  ],
  "sources": [
    {
      "title": "Ollama, OpenAI compatibility",
      "url": "https://docs.ollama.com/api/openai-compatibility",
      "accessed": "2026-11-15"
    },
    {
      "title": "Ollama FAQ",
      "url": "https://docs.ollama.com/faq",
      "accessed": "2026-11-15"
    },
    {
      "title": "Ollama, Tool calling",
      "url": "https://docs.ollama.com/capabilities/tool-calling",
      "accessed": "2026-11-15"
    },
    {
      "title": "vLLM stable docs, Online serving",
      "url": "https://docs.vllm.ai/en/stable/serving/online_serving/",
      "accessed": "2026-11-15"
    },
    {
      "title": "vLLM stable docs, Tool calling",
      "url": "https://docs.vllm.ai/en/stable/features/tool_calling/",
      "accessed": "2026-11-15"
    },
    {
      "title": "vLLM stable docs, vllm serve",
      "url": "https://docs.vllm.ai/en/stable/cli/serve/",
      "accessed": "2026-11-15"
    },
    {
      "title": "vLLM stable docs, Security",
      "url": "https://docs.vllm.ai/en/stable/usage/security/",
      "accessed": "2026-11-15"
    },
    {
      "title": "vLLM PyPI project",
      "url": "https://pypi.org/project/vllm/",
      "accessed": "2026-11-15"
    },
    {
      "title": "vLLM latest GitHub release",
      "url": "https://api.github.com/repos/vllm-project/vllm/releases/latest",
      "accessed": "2026-11-15"
    },
    {
      "title": "SGLang docs, Quickstart",
      "url": "https://docs.sglang.io/docs/get-started/quickstart",
      "accessed": "2026-11-15"
    },
    {
      "title": "SGLang docs, Examples",
      "url": "https://docs.sglang.io/docs/get-started/examples",
      "accessed": "2026-11-15"
    },
    {
      "title": "SGLang docs, Server arguments",
      "url": "https://docs.sglang.io/docs/advanced_features/server_arguments",
      "accessed": "2026-11-15"
    },
    {
      "title": "SGLang docs, OpenAI-compatible APIs",
      "url": "https://docs.sglang.io/docs/basic_usage/openai_api",
      "accessed": "2026-11-15"
    },
    {
      "title": "SGLang PyPI project",
      "url": "https://pypi.org/project/sglang/",
      "accessed": "2026-11-15"
    },
    {
      "title": "SGLang latest GitHub release",
      "url": "https://api.github.com/repos/sgl-project/sglang/releases/latest",
      "accessed": "2026-11-15"
    },
    {
      "title": "LocalAI docs, Quickstart",
      "url": "https://localai.io/docs/start/quickstart/",
      "accessed": "2026-11-15"
    },
    {
      "title": "LocalAI docs, CLI reference",
      "url": "https://localai.io/docs/reference/cli-reference/",
      "accessed": "2026-11-15"
    },
    {
      "title": "LocalAI docs, OpenAI functions",
      "url": "https://localai.io/docs/api/openai-functions/",
      "accessed": "2026-11-15"
    },
    {
      "title": "LocalAI latest GitHub release",
      "url": "https://api.github.com/repos/mudler/LocalAI/releases/latest",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM docs, OpenAI-compatible endpoints",
      "url": "https://docs.litellm.ai/docs/providers/openai_compatible",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM docs, CLI quick start",
      "url": "https://docs.litellm.ai/docs/proxy/quick_start",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM docs, Docker quick start",
      "url": "https://docs.litellm.ai/docs/proxy/docker_quick_start",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM docs, Config settings",
      "url": "https://docs.litellm.ai/docs/proxy/config_settings",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM docs, Security best practices",
      "url": "https://docs.litellm.ai/docs/proxy/security_best_practices",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM docs, Master key rotations",
      "url": "https://docs.litellm.ai/docs/proxy/master_key_rotations",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM docs, Ollama provider",
      "url": "https://docs.litellm.ai/docs/providers/ollama",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM docs, vLLM provider",
      "url": "https://docs.litellm.ai/docs/providers/vllm",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM PyPI project",
      "url": "https://pypi.org/project/litellm/",
      "accessed": "2026-11-15"
    },
    {
      "title": "LiteLLM latest GitHub release",
      "url": "https://api.github.com/repos/BerriAI/litellm/releases/latest",
      "accessed": "2026-11-15"
    },
    {
      "title": "Pi docs, Models",
      "url": "https://docs.pi.dev/latest/models",
      "accessed": "2026-11-15"
    }
  ],
  "front_matter": {
    "id": "run-an-agent-on-a-local-model",
    "type": "guide",
    "title": "Run an agent on a local model with an OpenAI-compatible endpoint",
    "summary": "Start Ollama, vLLM, SGLang or LocalAI bound to 127.0.0.1, point the OpenAI Python SDK at the server's /v1 base URL, enable tool calling, and optionally route the same endpoint through a LiteLLM proxy config.",
    "description": "A practical how-to for running an agent against a local OpenAI-compatible server. It covers documented bind flags for Ollama, vLLM, SGLang and LocalAI, official OpenAI SDK base_url examples, tool-calling flags, and a LiteLLM config that talks to those servers. It is not a product comparison of Ollama and llama.cpp, and not a feature matrix of hosted LLM gateways. LiteLLM 1.109.1 was installed on this box and checked with --validate_config plus a loopback mock; vLLM, SGLang, Ollama and LocalAI were not installed or started.",
    "author": "Agentica Author",
    "difficulty": "intermediate",
    "time_estimate": "45 min",
    "prerequisites": [
      "Python 3.10 or newer (for the LiteLLM proxy extra; this box ran 1.109.1 on Python 3.13)",
      "A local OpenAI-compatible server already installed, or a documented install to follow (Ollama, vLLM, SGLang or LocalAI). This guide does not install vLLM or SGLang, which typically need a GPU.",
      "No cloud account or paid API key is required for the loopback path"
    ],
    "tags": [
      "inference",
      "local-models",
      "openai-compatible",
      "tool-calling",
      "litellm",
      "agents"
    ],
    "entries": [
      "ollama",
      "vllm",
      "sglang",
      "localai",
      "litellm",
      "llama-cpp",
      "pi-coding-agent"
    ],
    "related": [
      "fully-local-private-agent",
      "local-llm-servers-for-agents",
      "llm-gateways",
      "personal-agent-runtimes",
      "coding-agent-harnesses"
    ],
    "sources": [
      {
        "title": "Ollama, OpenAI compatibility",
        "url": "https://docs.ollama.com/api/openai-compatibility",
        "accessed": "2026-11-15"
      },
      {
        "title": "Ollama FAQ",
        "url": "https://docs.ollama.com/faq",
        "accessed": "2026-11-15"
      },
      {
        "title": "Ollama, Tool calling",
        "url": "https://docs.ollama.com/capabilities/tool-calling",
        "accessed": "2026-11-15"
      },
      {
        "title": "vLLM stable docs, Online serving",
        "url": "https://docs.vllm.ai/en/stable/serving/online_serving/",
        "accessed": "2026-11-15"
      },
      {
        "title": "vLLM stable docs, Tool calling",
        "url": "https://docs.vllm.ai/en/stable/features/tool_calling/",
        "accessed": "2026-11-15"
      },
      {
        "title": "vLLM stable docs, vllm serve",
        "url": "https://docs.vllm.ai/en/stable/cli/serve/",
        "accessed": "2026-11-15"
      },
      {
        "title": "vLLM stable docs, Security",
        "url": "https://docs.vllm.ai/en/stable/usage/security/",
        "accessed": "2026-11-15"
      },
      {
        "title": "vLLM PyPI project",
        "url": "https://pypi.org/project/vllm/",
        "accessed": "2026-11-15"
      },
      {
        "title": "vLLM latest GitHub release",
        "url": "https://api.github.com/repos/vllm-project/vllm/releases/latest",
        "accessed": "2026-11-15"
      },
      {
        "title": "SGLang docs, Quickstart",
        "url": "https://docs.sglang.io/docs/get-started/quickstart",
        "accessed": "2026-11-15"
      },
      {
        "title": "SGLang docs, Examples",
        "url": "https://docs.sglang.io/docs/get-started/examples",
        "accessed": "2026-11-15"
      },
      {
        "title": "SGLang docs, Server arguments",
        "url": "https://docs.sglang.io/docs/advanced_features/server_arguments",
        "accessed": "2026-11-15"
      },
      {
        "title": "SGLang docs, OpenAI-compatible APIs",
        "url": "https://docs.sglang.io/docs/basic_usage/openai_api",
        "accessed": "2026-11-15"
      },
      {
        "title": "SGLang PyPI project",
        "url": "https://pypi.org/project/sglang/",
        "accessed": "2026-11-15"
      },
      {
        "title": "SGLang latest GitHub release",
        "url": "https://api.github.com/repos/sgl-project/sglang/releases/latest",
        "accessed": "2026-11-15"
      },
      {
        "title": "LocalAI docs, Quickstart",
        "url": "https://localai.io/docs/start/quickstart/",
        "accessed": "2026-11-15"
      },
      {
        "title": "LocalAI docs, CLI reference",
        "url": "https://localai.io/docs/reference/cli-reference/",
        "accessed": "2026-11-15"
      },
      {
        "title": "LocalAI docs, OpenAI functions",
        "url": "https://localai.io/docs/api/openai-functions/",
        "accessed": "2026-11-15"
      },
      {
        "title": "LocalAI latest GitHub release",
        "url": "https://api.github.com/repos/mudler/LocalAI/releases/latest",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM docs, OpenAI-compatible endpoints",
        "url": "https://docs.litellm.ai/docs/providers/openai_compatible",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM docs, CLI quick start",
        "url": "https://docs.litellm.ai/docs/proxy/quick_start",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM docs, Docker quick start",
        "url": "https://docs.litellm.ai/docs/proxy/docker_quick_start",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM docs, Config settings",
        "url": "https://docs.litellm.ai/docs/proxy/config_settings",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM docs, Security best practices",
        "url": "https://docs.litellm.ai/docs/proxy/security_best_practices",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM docs, Master key rotations",
        "url": "https://docs.litellm.ai/docs/proxy/master_key_rotations",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM docs, Ollama provider",
        "url": "https://docs.litellm.ai/docs/providers/ollama",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM docs, vLLM provider",
        "url": "https://docs.litellm.ai/docs/providers/vllm",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM PyPI project",
        "url": "https://pypi.org/project/litellm/",
        "accessed": "2026-11-15"
      },
      {
        "title": "LiteLLM latest GitHub release",
        "url": "https://api.github.com/repos/BerriAI/litellm/releases/latest",
        "accessed": "2026-11-15"
      },
      {
        "title": "Pi docs, Models",
        "url": "https://docs.pi.dev/latest/models",
        "accessed": "2026-11-15"
      }
    ],
    "last_verified": "2026-11-15",
    "published": "2026-11-15"
  },
  "markdown": "\n## What this guide is (and is not)\n\nThis is a how-to: start a local model server so it listens only on loopback, point an [OpenAI Python SDK](https://github.com/openai/openai-python) client at that server's OpenAI-compatible `/v1` URL, send a tool-calling request, and optionally put [LiteLLM](https://indexagentica.com/entries/litellm/) in front as a local router.\n\nIt is not a feature-by-feature bake-off of local servers. For Ollama versus llama.cpp — tools, grammars, default ports and licences — see [Ollama vs llama.cpp for agents](https://indexagentica.com/compare/local-llm-servers-for-agents/). [llama.cpp](https://indexagentica.com/entries/llama-cpp/) is not a fifth subject here; use that comparison if that is the server you want. It is also not a comparison of hosted LLM gateways. LiteLLM appears here only as a local proxy you run yourself. For OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, Portkey and LiteLLM as a *gateway product*, see [LLM gateways compared](https://indexagentica.com/compare/llm-gateways/).\n\nTo assemble a whole on-machine agent (this serving path plus local memory and a local sandbox), see [Fully local, private agent](https://indexagentica.com/stacks/fully-local-private-agent/).\n\nVersions checked on 15 November 2026: [Ollama](https://indexagentica.com/entries/ollama/) GitHub v0.40.16 (from the catalog entry and the 14 November comparison; not re-pulled as a server here); [vLLM](https://indexagentica.com/entries/vllm/) PyPI **0.33.0**, while GitHub `/releases/latest` currently points at **v0.33.0rc3** with `prerelease` set to false (a `v0.33.0` tag also exists); [SGLang](https://indexagentica.com/entries/sglang/) PyPI and GitHub **v0.5.23**; [LocalAI](https://indexagentica.com/entries/localai/) GitHub **v4.12.1** (do not `pip install` the unrelated PyPI package named `localai`); [LiteLLM](https://indexagentica.com/entries/litellm/) PyPI and GitHub **v1.109.1**, in sync. LiteLLM's SDK and open-source proxy are MIT; the repository's `enterprise/` tree is a separate licence and is not used here.\n\n## Bind the server to loopback first\n\nThese servers speak HTTP. If the process listens on all interfaces, anything that can reach that port can usually call the model. This guide always sets an explicit loopback bind. Official examples do not always do that:\n\n- [Ollama's FAQ](https://docs.ollama.com/faq) states that the server binds `127.0.0.1:11434` by default. Change it with `OLLAMA_HOST` (their examples use `0.0.0.0:11434` when the goal is to expose the server).\n- [vLLM's `vllm serve` CLI](https://docs.vllm.ai/en/stable/cli/serve/) documents `--port` with default `8000`. `--host` is documented only as \"Host name\", with no default on that page, so pass `--host 127.0.0.1` explicitly. Official client examples use `http://localhost:8000/v1`.\n- [SGLang server arguments](https://docs.sglang.io/docs/advanced_features/server_arguments) document `--host` default `127.0.0.1` and `--port` default `30000`. The [quickstart](https://docs.sglang.io/docs/get-started/quickstart) overrides this with `--host 0.0.0.0 --port 30000`. The [examples](https://docs.sglang.io/docs/get-started/examples) page shows both a Docker publish of `127.0.0.1:30000:30000` (the process inside the container still uses `--host 0.0.0.0`) and a native command with `--host 127.0.0.1 --port 30000`.\n- [LocalAI's quickstart](https://localai.io/docs/start/quickstart/) states: \"By default, LocalAI listens on port 8080 on all interfaces, and authentication is off.\" The [CLI reference](https://localai.io/docs/reference/cli-reference/) documents `--address` / `$LOCALAI_ADDRESS` with default `:8080`. Pass `--address 127.0.0.1:8080`. Official Docker examples use `-p 8080:8080`, which publishes the port on all host interfaces; use `-p 127.0.0.1:8080:8080` instead.\n\nDo not run vendor `install.sh` scripts from this guide. Follow each project's documented install for your OS, then start the server with the flags below.\n\n## Start an OpenAI-compatible server\n\nPick one. You only need a single `/v1` endpoint for the rest of the guide.\n\n### Ollama\n\nOnce the Ollama app or daemon is installed, it already listens on loopback. The [OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility) page gives the local base URL as `http://localhost:11434/v1`.\n\nA generic OpenAI client looks like this (the dummy key is required by the SDK; Ollama does not treat it as a secret on the default local server):\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://127.0.0.1:11434/v1\", api_key=\"ollama\")\n```\n\nUse a model name you have already pulled. This guide does not download a model.\n\n### vLLM\n\nCite the [stable](https://docs.vllm.ai/en/stable/serving/online_serving/) docs, not `/en/latest/` (that tree is a developer preview). A loopback server, combining the official `vllm serve` form with the documented `--host` / `--port` flags:\n\n```bash\nvllm serve MODEL \\\n    --host 127.0.0.1 \\\n    --port 8000\n```\n\nReplace `MODEL` with a Hugging Face model id your install can load. The official tool-calling quickstart uses `meta-llama/Llama-3.1-8B-Instruct` and was not run here.\n\nThe [security](https://docs.vllm.ai/en/stable/usage/security/) page states that `--api-key` (or `VLLM_API_KEY`) authenticates only endpoints under the `/v1`, `/v2`, `/inference` and `/cohere` prefixes. Do not treat `--api-key` as protection for the whole process; bind to loopback first.\n\nOfficial client (from the [tool-calling](https://docs.vllm.ai/en/stable/features/tool_calling/) page):\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"dummy\")\n```\n\n### SGLang\n\nFrom the official [examples](https://docs.sglang.io/docs/get-started/examples) native path, which already uses loopback:\n\n```bash\nsglang serve MODEL_PATH --host 127.0.0.1 --port 30000\n```\n\nIf you copy the [quickstart](https://docs.sglang.io/docs/get-started/quickstart) command, it passes `--host 0.0.0.0 --port 30000`. Change `--host` to `127.0.0.1` unless you intend to publish the port. Official Docker examples map `-p 30000:30000` or `-p 127.0.0.1:30000:30000` and still pass `--host 0.0.0.0` *inside* the container; the host-side `-p` is what decides whether the port is reachable off-machine.\n\nOfficial client (same examples page):\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://localhost:30000/v1\",\n    api_key=\"EMPTY\",\n)\n```\n\nThe server is ready when it logs `The server is fired up and ready to roll!`.\n\n### LocalAI\n\nThe documented command is `local-ai run`. Bind loopback with the official `--address` flag:\n\n```bash\nlocal-ai run --address 127.0.0.1:8080\n```\n\nThe official quickstart also shows `local-ai run qwen3-4b`, which downloads that model on first run, and `docker run -p 8080:8080 --name local-ai -ti localai/localai:latest`. Prefer a host publish of `127.0.0.1:8080:8080` if you use Docker. The [OpenAI functions](https://localai.io/docs/api/openai-functions/) page uses `http://localhost:8080/v1/` as `base_url`.\n\nOfficial client (from that page; `api_key=\"test\"` is their example):\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=\"test\",\n    base_url=\"http://localhost:8080/v1/\"\n)\n```\n\n## Point the OpenAI SDK at `base_url`\n\nThe pattern is the same for every server above: construct `OpenAI(base_url=..., api_key=...)` and call `chat.completions.create`. Use the host and port you actually bound. `localhost` and `127.0.0.1` are equivalent on a typical machine; this guide prefers `127.0.0.1` in commands we write, and leaves official snippets unchanged when quoting them.\n\nA minimal completion, with the Ollama URL as the example:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://127.0.0.1:11434/v1\", api_key=\"ollama\")\nresponse = client.chat.completions.create(\n    model=\"MODEL\",\n    messages=[{\"role\": \"user\", \"content\": \"Reply with the word pong.\"}],\n)\nprint(response.choices[0].message.content)\n```\n\nReplace `MODEL` with a name the server already has. This snippet was not sent to a real local model on this box.\n\n## Enable tool calling\n\nTool calling is a server *and* a model feature. The request shape is the OpenAI `tools` array. The server often needs an extra flag so it can parse the model's tool calls back into that schema.\n\n**vLLM.** The [tool-calling](https://docs.vllm.ai/en/stable/features/tool_calling/) page marks `--enable-auto-tool-choice` as mandatory for auto tool choice, and `--tool-call-parser` as the parser to use (required with that flag). Their Llama 3.1 8B quickstart is:\n\n```bash\nvllm serve meta-llama/Llama-3.1-8B-Instruct \\\n    --enable-auto-tool-choice \\\n    --tool-call-parser llama3_json \\\n    --chat-template examples/tool_chat_template_llama3.1_json.jinja\n```\n\nAdd `--host 127.0.0.1 --port 8000` as above. That page documents `tool_choice` values `auto`, `required` (as of `vllm>=0.8.3`) and `none`, plus named function calling. Their official Python example builds a `get_weather` tool, passes `tools=tools, tool_choice=\"auto\"`, and reads `response.choices[0].message.tool_calls`.\n\n**SGLang.** Official [examples](https://docs.sglang.io/docs/get-started/examples) pass `--tool-call-parser qwen3_coder` on the Qwen3.8-27B-FP8 serve command. That same page notes that its small local Qwen3-0.6B example does *not* configure tool calling, and points at the tool-calling guide for other models. The OpenAI client is still `base_url=\"http://localhost:30000/v1\"` with `api_key=\"EMPTY\"`.\n\n**Ollama.** The [OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility) page documents `tools` as supported on `/v1/chat/completions` and `tool_choice` as not supported on that route. Use the native tools API or omit `tool_choice` on `/v1`. See the [tool-calling](https://docs.ollama.com/capabilities/tool-calling) docs and the [Ollama vs llama.cpp](https://indexagentica.com/compare/local-llm-servers-for-agents/) comparison for the native versus OpenAI surface; this guide does not repeat that matrix.\n\n**LocalAI.** The [OpenAI functions](https://localai.io/docs/api/openai-functions/) page shows the same `tools` / `tool_choice=\"auto\"` shape against `http://localhost:8080/v1/`. Use a model you have installed that actually supports tools (their quickstart names `qwen3-4b` as a small CPU model that does).\n\nAn official-style request, abbreviated from the vLLM quickstart (not executed here):\n\n```python\ntools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"get_weather\",\n            \"description\": \"Get the current weather in a given location\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"location\": {\"type\": \"string\"},\n                    \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n                },\n                \"required\": [\"location\", \"unit\"],\n            },\n        },\n    },\n]\n\nresponse = client.chat.completions.create(\n    model=\"MODEL\",\n    messages=[{\"role\": \"user\", \"content\": \"What is the weather in San Francisco?\"}],\n    tools=tools,\n    tool_choice=\"auto\",  # omit on Ollama /v1/chat/completions\n)\n```\n\nThen execute the returned `tool_calls` in your process and send the tool result back on a message with `role=\"tool\"`, following the provider's example.\n\n## Route through LiteLLM\n\nUse [LiteLLM](https://indexagentica.com/entries/litellm/) when you want one local OpenAI-compatible URL in front of whichever server you started, or when a client already speaks \"OpenAI\" and you would rather change a config file than the client. This is a router on `127.0.0.1`, not the hosted-gateway comparison.\n\nInstall the proxy extra (PyPI **1.109.1** on 15 November 2026, matching GitHub `v1.109.1`):\n\n```bash\npip install 'litellm[proxy]'\n```\n\nThe [OpenAI-compatible endpoints](https://docs.litellm.ai/docs/providers/openai_compatible) page says to prefix the model with `openai/`, set `api_base` to your server, and pass an `api_key` (the OpenAI provider path requires one; a dummy value is enough when the backend ignores it). If you see a `Not Found` error, that page says to give `api_base` a `/v1` postfix. A config that points at a loopback Ollama server:\n\n```yaml\nmodel_list:\n  - model_name: local-model\n    litellm_params:\n      model: openai/MODEL\n      api_base: http://127.0.0.1:11434/v1\n      api_key: dummy\n```\n\nChange `api_base` (and the model id after `openai/`) for vLLM (`http://127.0.0.1:8000/v1`), SGLang (`http://127.0.0.1:30000/v1`) or LocalAI (`http://127.0.0.1:8080/v1`). LiteLLM also documents provider prefixes such as `ollama/` with `api_base: http://localhost:11434` and `hosted_vllm/`; those are alternatives, not a second required router.\n\nCheck the file without starting a server (LiteLLM 1.109.1 CLI: \"Load and validate the config file ... then exit\"):\n\n```bash\nlitellm --config /path/to/config.yaml --validate_config\n```\n\nStart the proxy on loopback. The 1.109.1 CLI documents `--host` and `--port`. Official [CLI](https://docs.litellm.ai/docs/proxy/quick_start) logs say `Proxy running on http://0.0.0.0:4000` when you do not pass `--host`; pass it:\n\n```bash\nexport LITELLM_MASTER_KEY=\"sk-$(openssl rand -hex 32)\"\nlitellm --config /path/to/config.yaml --host 127.0.0.1 --port 4000\n```\n\nKeep that value out of the config file and out of source control. The [config settings](https://docs.litellm.ai/docs/proxy/config_settings) page states that the proxy will not start when the master key is unset, empty, or has a known unsafe value, and that `general_settings.master_key` falls back to `LITELLM_MASTER_KEY`. 1.109.1 on this box refused to boot without it, with the same message. The older wording on the [configs](https://docs.litellm.ai/docs/proxy/configs) page that still calls `master_key` optional lags that behaviour. There is a documented local-only escape hatch, `LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY`, which this guide does not recommend.\n\nThe official [Docker quick start](https://docs.litellm.ai/docs/proxy/docker_quick_start) already publishes loopback and expects a master key:\n\n```bash\ndocker run \\\n  -v \"$PWD/litellm_config.yaml:/app/config.yaml:ro\" \\\n  -e LITELLM_MASTER_KEY \\\n  -p 127.0.0.1:4000:4000 \\\n  docker.litellm.ai/berriai/litellm:latest \\\n  --config /app/config.yaml\n```\n\nCall the proxy with the OpenAI SDK. Official snippets often use `http://0.0.0.0:4000`; use loopback and the master key:\n\n```python\nimport os\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://127.0.0.1:4000/v1\",\n    api_key=os.environ[\"LITELLM_MASTER_KEY\"],\n)\nresponse = client.chat.completions.create(\n    model=\"local-model\",\n    messages=[{\"role\": \"user\", \"content\": \"Reply with the word pong.\"}],\n)\nprint(response.choices[0].message.content)\n```\n\nDo not use LiteLLM's `--test` flag as a \"local\" check. The official CLI help describes it as making a test request, and the docs route that path at a Hugging Face inference example (`gpt-3.5-turbo` / `bigcode/starcoder`). That leaves the machine. Use `--validate_config`, or a client request against a server you started on `127.0.0.1`.\n\nSee the official [security best practices](https://docs.litellm.ai/docs/proxy/security_best_practices) for storing the master key and provider credentials outside `config.yaml`.\n\n## What was verified on this box\n\nOn 15 November 2026 this box installed `litellm[proxy]==1.109.1` (Python 3.13). `litellm --config ... --validate_config` exited 0 on a config whose `api_base` was `http://127.0.0.1:18080/v1`. Starting the proxy with `--host 127.0.0.1 --port 14000` and no `LITELLM_MASTER_KEY` failed with `UnsafeMasterKeyError`. With a locally generated `LITELLM_MASTER_KEY`, the same proxy listened only on `127.0.0.1:14000`. A loopback mock OpenAI server on `127.0.0.1:18080` answered `POST /v1/chat/completions`; both `curl` to `/chat/completions` and the OpenAI SDK (`base_url=\"http://127.0.0.1:14000/v1\"`) returned the mock body `mock-ok`. Both processes were stopped afterwards. No vLLM, SGLang, Ollama or LocalAI server was installed or started, and no model weights were downloaded.\n\n## Next step\n\nWire this endpoint into a local harness, memory store and sandbox with [Fully local, private agent](https://indexagentica.com/stacks/fully-local-private-agent/). [Pi](https://indexagentica.com/entries/pi-coding-agent/) documents a `models.json` provider with `\"baseUrl\": \"http://localhost:11434/v1\"` and `\"api\": \"openai-completions\"` for Ollama, and the same compatible-endpoint path for vLLM, SGLang and other OpenAI-style servers.\n",
  "raw": "---\nid: run-an-agent-on-a-local-model\ntype: guide\ntitle: \"Run an agent on a local model with an OpenAI-compatible endpoint\"\nsummary: \"Start Ollama, vLLM, SGLang or LocalAI bound to 127.0.0.1, point the OpenAI Python SDK at the server's /v1 base URL, enable tool calling, and optionally route the same endpoint through a LiteLLM proxy config.\"\ndescription: \"A practical how-to for running an agent against a local OpenAI-compatible server. It covers documented bind flags for Ollama, vLLM, SGLang and LocalAI, official OpenAI SDK base_url examples, tool-calling flags, and a LiteLLM config that talks to those servers. It is not a product comparison of Ollama and llama.cpp, and not a feature matrix of hosted LLM gateways. LiteLLM 1.109.1 was installed on this box and checked with --validate_config plus a loopback mock; vLLM, SGLang, Ollama and LocalAI were not installed or started.\"\nauthor: Agentica Author\ndifficulty: intermediate\ntime_estimate: 45 min\nprerequisites:\n  - \"Python 3.10 or newer (for the LiteLLM proxy extra; this box ran 1.109.1 on Python 3.13)\"\n  - \"A local OpenAI-compatible server already installed, or a documented install to follow (Ollama, vLLM, SGLang or LocalAI). This guide does not install vLLM or SGLang, which typically need a GPU.\"\n  - \"No cloud account or paid API key is required for the loopback path\"\ntags: [inference, local-models, openai-compatible, tool-calling, litellm, agents]\nentries: [ollama, vllm, sglang, localai, litellm, llama-cpp, pi-coding-agent]\nrelated: [fully-local-private-agent, local-llm-servers-for-agents, llm-gateways, personal-agent-runtimes, coding-agent-harnesses]\nsources:\n  - title: \"Ollama, OpenAI compatibility\"\n    url: https://docs.ollama.com/api/openai-compatibility\n    accessed: 2026-11-15\n  - title: \"Ollama FAQ\"\n    url: https://docs.ollama.com/faq\n    accessed: 2026-11-15\n  - title: \"Ollama, Tool calling\"\n    url: https://docs.ollama.com/capabilities/tool-calling\n    accessed: 2026-11-15\n  - title: \"vLLM stable docs, Online serving\"\n    url: https://docs.vllm.ai/en/stable/serving/online_serving/\n    accessed: 2026-11-15\n  - title: \"vLLM stable docs, Tool calling\"\n    url: https://docs.vllm.ai/en/stable/features/tool_calling/\n    accessed: 2026-11-15\n  - title: \"vLLM stable docs, vllm serve\"\n    url: https://docs.vllm.ai/en/stable/cli/serve/\n    accessed: 2026-11-15\n  - title: \"vLLM stable docs, Security\"\n    url: https://docs.vllm.ai/en/stable/usage/security/\n    accessed: 2026-11-15\n  - title: \"vLLM PyPI project\"\n    url: https://pypi.org/project/vllm/\n    accessed: 2026-11-15\n  - title: \"vLLM latest GitHub release\"\n    url: https://api.github.com/repos/vllm-project/vllm/releases/latest\n    accessed: 2026-11-15\n  - title: \"SGLang docs, Quickstart\"\n    url: https://docs.sglang.io/docs/get-started/quickstart\n    accessed: 2026-11-15\n  - title: \"SGLang docs, Examples\"\n    url: https://docs.sglang.io/docs/get-started/examples\n    accessed: 2026-11-15\n  - title: \"SGLang docs, Server arguments\"\n    url: https://docs.sglang.io/docs/advanced_features/server_arguments\n    accessed: 2026-11-15\n  - title: \"SGLang docs, OpenAI-compatible APIs\"\n    url: https://docs.sglang.io/docs/basic_usage/openai_api\n    accessed: 2026-11-15\n  - title: \"SGLang PyPI project\"\n    url: https://pypi.org/project/sglang/\n    accessed: 2026-11-15\n  - title: \"SGLang latest GitHub release\"\n    url: https://api.github.com/repos/sgl-project/sglang/releases/latest\n    accessed: 2026-11-15\n  - title: \"LocalAI docs, Quickstart\"\n    url: https://localai.io/docs/start/quickstart/\n    accessed: 2026-11-15\n  - title: \"LocalAI docs, CLI reference\"\n    url: https://localai.io/docs/reference/cli-reference/\n    accessed: 2026-11-15\n  - title: \"LocalAI docs, OpenAI functions\"\n    url: https://localai.io/docs/api/openai-functions/\n    accessed: 2026-11-15\n  - title: \"LocalAI latest GitHub release\"\n    url: https://api.github.com/repos/mudler/LocalAI/releases/latest\n    accessed: 2026-11-15\n  - title: \"LiteLLM docs, OpenAI-compatible endpoints\"\n    url: https://docs.litellm.ai/docs/providers/openai_compatible\n    accessed: 2026-11-15\n  - title: \"LiteLLM docs, CLI quick start\"\n    url: https://docs.litellm.ai/docs/proxy/quick_start\n    accessed: 2026-11-15\n  - title: \"LiteLLM docs, Docker quick start\"\n    url: https://docs.litellm.ai/docs/proxy/docker_quick_start\n    accessed: 2026-11-15\n  - title: \"LiteLLM docs, Config settings\"\n    url: https://docs.litellm.ai/docs/proxy/config_settings\n    accessed: 2026-11-15\n  - title: \"LiteLLM docs, Security best practices\"\n    url: https://docs.litellm.ai/docs/proxy/security_best_practices\n    accessed: 2026-11-15\n  - title: \"LiteLLM docs, Master key rotations\"\n    url: https://docs.litellm.ai/docs/proxy/master_key_rotations\n    accessed: 2026-11-15\n  - title: \"LiteLLM docs, Ollama provider\"\n    url: https://docs.litellm.ai/docs/providers/ollama\n    accessed: 2026-11-15\n  - title: \"LiteLLM docs, vLLM provider\"\n    url: https://docs.litellm.ai/docs/providers/vllm\n    accessed: 2026-11-15\n  - title: \"LiteLLM PyPI project\"\n    url: https://pypi.org/project/litellm/\n    accessed: 2026-11-15\n  - title: \"LiteLLM latest GitHub release\"\n    url: https://api.github.com/repos/BerriAI/litellm/releases/latest\n    accessed: 2026-11-15\n  - title: \"Pi docs, Models\"\n    url: https://docs.pi.dev/latest/models\n    accessed: 2026-11-15\nlast_verified: 2026-11-15\npublished: 2026-11-15\n---\n\n## What this guide is (and is not)\n\nThis is a how-to: start a local model server so it listens only on loopback, point an [OpenAI Python SDK](https://github.com/openai/openai-python) client at that server's OpenAI-compatible `/v1` URL, send a tool-calling request, and optionally put [LiteLLM](https://indexagentica.com/entries/litellm/) in front as a local router.\n\nIt is not a feature-by-feature bake-off of local servers. For Ollama versus llama.cpp — tools, grammars, default ports and licences — see [Ollama vs llama.cpp for agents](https://indexagentica.com/compare/local-llm-servers-for-agents/). [llama.cpp](https://indexagentica.com/entries/llama-cpp/) is not a fifth subject here; use that comparison if that is the server you want. It is also not a comparison of hosted LLM gateways. LiteLLM appears here only as a local proxy you run yourself. For OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, Portkey and LiteLLM as a *gateway product*, see [LLM gateways compared](https://indexagentica.com/compare/llm-gateways/).\n\nTo assemble a whole on-machine agent (this serving path plus local memory and a local sandbox), see [Fully local, private agent](https://indexagentica.com/stacks/fully-local-private-agent/).\n\nVersions checked on 15 November 2026: [Ollama](https://indexagentica.com/entries/ollama/) GitHub v0.40.16 (from the catalog entry and the 14 November comparison; not re-pulled as a server here); [vLLM](https://indexagentica.com/entries/vllm/) PyPI **0.33.0**, while GitHub `/releases/latest` currently points at **v0.33.0rc3** with `prerelease` set to false (a `v0.33.0` tag also exists); [SGLang](https://indexagentica.com/entries/sglang/) PyPI and GitHub **v0.5.23**; [LocalAI](https://indexagentica.com/entries/localai/) GitHub **v4.12.1** (do not `pip install` the unrelated PyPI package named `localai`); [LiteLLM](https://indexagentica.com/entries/litellm/) PyPI and GitHub **v1.109.1**, in sync. LiteLLM's SDK and open-source proxy are MIT; the repository's `enterprise/` tree is a separate licence and is not used here.\n\n## Bind the server to loopback first\n\nThese servers speak HTTP. If the process listens on all interfaces, anything that can reach that port can usually call the model. This guide always sets an explicit loopback bind. Official examples do not always do that:\n\n- [Ollama's FAQ](https://docs.ollama.com/faq) states that the server binds `127.0.0.1:11434` by default. Change it with `OLLAMA_HOST` (their examples use `0.0.0.0:11434` when the goal is to expose the server).\n- [vLLM's `vllm serve` CLI](https://docs.vllm.ai/en/stable/cli/serve/) documents `--port` with default `8000`. `--host` is documented only as \"Host name\", with no default on that page, so pass `--host 127.0.0.1` explicitly. Official client examples use `http://localhost:8000/v1`.\n- [SGLang server arguments](https://docs.sglang.io/docs/advanced_features/server_arguments) document `--host` default `127.0.0.1` and `--port` default `30000`. The [quickstart](https://docs.sglang.io/docs/get-started/quickstart) overrides this with `--host 0.0.0.0 --port 30000`. The [examples](https://docs.sglang.io/docs/get-started/examples) page shows both a Docker publish of `127.0.0.1:30000:30000` (the process inside the container still uses `--host 0.0.0.0`) and a native command with `--host 127.0.0.1 --port 30000`.\n- [LocalAI's quickstart](https://localai.io/docs/start/quickstart/) states: \"By default, LocalAI listens on port 8080 on all interfaces, and authentication is off.\" The [CLI reference](https://localai.io/docs/reference/cli-reference/) documents `--address` / `$LOCALAI_ADDRESS` with default `:8080`. Pass `--address 127.0.0.1:8080`. Official Docker examples use `-p 8080:8080`, which publishes the port on all host interfaces; use `-p 127.0.0.1:8080:8080` instead.\n\nDo not run vendor `install.sh` scripts from this guide. Follow each project's documented install for your OS, then start the server with the flags below.\n\n## Start an OpenAI-compatible server\n\nPick one. You only need a single `/v1` endpoint for the rest of the guide.\n\n### Ollama\n\nOnce the Ollama app or daemon is installed, it already listens on loopback. The [OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility) page gives the local base URL as `http://localhost:11434/v1`.\n\nA generic OpenAI client looks like this (the dummy key is required by the SDK; Ollama does not treat it as a secret on the default local server):\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://127.0.0.1:11434/v1\", api_key=\"ollama\")\n```\n\nUse a model name you have already pulled. This guide does not download a model.\n\n### vLLM\n\nCite the [stable](https://docs.vllm.ai/en/stable/serving/online_serving/) docs, not `/en/latest/` (that tree is a developer preview). A loopback server, combining the official `vllm serve` form with the documented `--host` / `--port` flags:\n\n```bash\nvllm serve MODEL \\\n    --host 127.0.0.1 \\\n    --port 8000\n```\n\nReplace `MODEL` with a Hugging Face model id your install can load. The official tool-calling quickstart uses `meta-llama/Llama-3.1-8B-Instruct` and was not run here.\n\nThe [security](https://docs.vllm.ai/en/stable/usage/security/) page states that `--api-key` (or `VLLM_API_KEY`) authenticates only endpoints under the `/v1`, `/v2`, `/inference` and `/cohere` prefixes. Do not treat `--api-key` as protection for the whole process; bind to loopback first.\n\nOfficial client (from the [tool-calling](https://docs.vllm.ai/en/stable/features/tool_calling/) page):\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"dummy\")\n```\n\n### SGLang\n\nFrom the official [examples](https://docs.sglang.io/docs/get-started/examples) native path, which already uses loopback:\n\n```bash\nsglang serve MODEL_PATH --host 127.0.0.1 --port 30000\n```\n\nIf you copy the [quickstart](https://docs.sglang.io/docs/get-started/quickstart) command, it passes `--host 0.0.0.0 --port 30000`. Change `--host` to `127.0.0.1` unless you intend to publish the port. Official Docker examples map `-p 30000:30000` or `-p 127.0.0.1:30000:30000` and still pass `--host 0.0.0.0` *inside* the container; the host-side `-p` is what decides whether the port is reachable off-machine.\n\nOfficial client (same examples page):\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://localhost:30000/v1\",\n    api_key=\"EMPTY\",\n)\n```\n\nThe server is ready when it logs `The server is fired up and ready to roll!`.\n\n### LocalAI\n\nThe documented command is `local-ai run`. Bind loopback with the official `--address` flag:\n\n```bash\nlocal-ai run --address 127.0.0.1:8080\n```\n\nThe official quickstart also shows `local-ai run qwen3-4b`, which downloads that model on first run, and `docker run -p 8080:8080 --name local-ai -ti localai/localai:latest`. Prefer a host publish of `127.0.0.1:8080:8080` if you use Docker. The [OpenAI functions](https://localai.io/docs/api/openai-functions/) page uses `http://localhost:8080/v1/` as `base_url`.\n\nOfficial client (from that page; `api_key=\"test\"` is their example):\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=\"test\",\n    base_url=\"http://localhost:8080/v1/\"\n)\n```\n\n## Point the OpenAI SDK at `base_url`\n\nThe pattern is the same for every server above: construct `OpenAI(base_url=..., api_key=...)` and call `chat.completions.create`. Use the host and port you actually bound. `localhost` and `127.0.0.1` are equivalent on a typical machine; this guide prefers `127.0.0.1` in commands we write, and leaves official snippets unchanged when quoting them.\n\nA minimal completion, with the Ollama URL as the example:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://127.0.0.1:11434/v1\", api_key=\"ollama\")\nresponse = client.chat.completions.create(\n    model=\"MODEL\",\n    messages=[{\"role\": \"user\", \"content\": \"Reply with the word pong.\"}],\n)\nprint(response.choices[0].message.content)\n```\n\nReplace `MODEL` with a name the server already has. This snippet was not sent to a real local model on this box.\n\n## Enable tool calling\n\nTool calling is a server *and* a model feature. The request shape is the OpenAI `tools` array. The server often needs an extra flag so it can parse the model's tool calls back into that schema.\n\n**vLLM.** The [tool-calling](https://docs.vllm.ai/en/stable/features/tool_calling/) page marks `--enable-auto-tool-choice` as mandatory for auto tool choice, and `--tool-call-parser` as the parser to use (required with that flag). Their Llama 3.1 8B quickstart is:\n\n```bash\nvllm serve meta-llama/Llama-3.1-8B-Instruct \\\n    --enable-auto-tool-choice \\\n    --tool-call-parser llama3_json \\\n    --chat-template examples/tool_chat_template_llama3.1_json.jinja\n```\n\nAdd `--host 127.0.0.1 --port 8000` as above. That page documents `tool_choice` values `auto`, `required` (as of `vllm>=0.8.3`) and `none`, plus named function calling. Their official Python example builds a `get_weather` tool, passes `tools=tools, tool_choice=\"auto\"`, and reads `response.choices[0].message.tool_calls`.\n\n**SGLang.** Official [examples](https://docs.sglang.io/docs/get-started/examples) pass `--tool-call-parser qwen3_coder` on the Qwen3.8-27B-FP8 serve command. That same page notes that its small local Qwen3-0.6B example does *not* configure tool calling, and points at the tool-calling guide for other models. The OpenAI client is still `base_url=\"http://localhost:30000/v1\"` with `api_key=\"EMPTY\"`.\n\n**Ollama.** The [OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility) page documents `tools` as supported on `/v1/chat/completions` and `tool_choice` as not supported on that route. Use the native tools API or omit `tool_choice` on `/v1`. See the [tool-calling](https://docs.ollama.com/capabilities/tool-calling) docs and the [Ollama vs llama.cpp](https://indexagentica.com/compare/local-llm-servers-for-agents/) comparison for the native versus OpenAI surface; this guide does not repeat that matrix.\n\n**LocalAI.** The [OpenAI functions](https://localai.io/docs/api/openai-functions/) page shows the same `tools` / `tool_choice=\"auto\"` shape against `http://localhost:8080/v1/`. Use a model you have installed that actually supports tools (their quickstart names `qwen3-4b` as a small CPU model that does).\n\nAn official-style request, abbreviated from the vLLM quickstart (not executed here):\n\n```python\ntools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"get_weather\",\n            \"description\": \"Get the current weather in a given location\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"location\": {\"type\": \"string\"},\n                    \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n                },\n                \"required\": [\"location\", \"unit\"],\n            },\n        },\n    },\n]\n\nresponse = client.chat.completions.create(\n    model=\"MODEL\",\n    messages=[{\"role\": \"user\", \"content\": \"What is the weather in San Francisco?\"}],\n    tools=tools,\n    tool_choice=\"auto\",  # omit on Ollama /v1/chat/completions\n)\n```\n\nThen execute the returned `tool_calls` in your process and send the tool result back on a message with `role=\"tool\"`, following the provider's example.\n\n## Route through LiteLLM\n\nUse [LiteLLM](https://indexagentica.com/entries/litellm/) when you want one local OpenAI-compatible URL in front of whichever server you started, or when a client already speaks \"OpenAI\" and you would rather change a config file than the client. This is a router on `127.0.0.1`, not the hosted-gateway comparison.\n\nInstall the proxy extra (PyPI **1.109.1** on 15 November 2026, matching GitHub `v1.109.1`):\n\n```bash\npip install 'litellm[proxy]'\n```\n\nThe [OpenAI-compatible endpoints](https://docs.litellm.ai/docs/providers/openai_compatible) page says to prefix the model with `openai/`, set `api_base` to your server, and pass an `api_key` (the OpenAI provider path requires one; a dummy value is enough when the backend ignores it). If you see a `Not Found` error, that page says to give `api_base` a `/v1` postfix. A config that points at a loopback Ollama server:\n\n```yaml\nmodel_list:\n  - model_name: local-model\n    litellm_params:\n      model: openai/MODEL\n      api_base: http://127.0.0.1:11434/v1\n      api_key: dummy\n```\n\nChange `api_base` (and the model id after `openai/`) for vLLM (`http://127.0.0.1:8000/v1`), SGLang (`http://127.0.0.1:30000/v1`) or LocalAI (`http://127.0.0.1:8080/v1`). LiteLLM also documents provider prefixes such as `ollama/` with `api_base: http://localhost:11434` and `hosted_vllm/`; those are alternatives, not a second required router.\n\nCheck the file without starting a server (LiteLLM 1.109.1 CLI: \"Load and validate the config file ... then exit\"):\n\n```bash\nlitellm --config /path/to/config.yaml --validate_config\n```\n\nStart the proxy on loopback. The 1.109.1 CLI documents `--host` and `--port`. Official [CLI](https://docs.litellm.ai/docs/proxy/quick_start) logs say `Proxy running on http://0.0.0.0:4000` when you do not pass `--host`; pass it:\n\n```bash\nexport LITELLM_MASTER_KEY=\"sk-$(openssl rand -hex 32)\"\nlitellm --config /path/to/config.yaml --host 127.0.0.1 --port 4000\n```\n\nKeep that value out of the config file and out of source control. The [config settings](https://docs.litellm.ai/docs/proxy/config_settings) page states that the proxy will not start when the master key is unset, empty, or has a known unsafe value, and that `general_settings.master_key` falls back to `LITELLM_MASTER_KEY`. 1.109.1 on this box refused to boot without it, with the same message. The older wording on the [configs](https://docs.litellm.ai/docs/proxy/configs) page that still calls `master_key` optional lags that behaviour. There is a documented local-only escape hatch, `LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY`, which this guide does not recommend.\n\nThe official [Docker quick start](https://docs.litellm.ai/docs/proxy/docker_quick_start) already publishes loopback and expects a master key:\n\n```bash\ndocker run \\\n  -v \"$PWD/litellm_config.yaml:/app/config.yaml:ro\" \\\n  -e LITELLM_MASTER_KEY \\\n  -p 127.0.0.1:4000:4000 \\\n  docker.litellm.ai/berriai/litellm:latest \\\n  --config /app/config.yaml\n```\n\nCall the proxy with the OpenAI SDK. Official snippets often use `http://0.0.0.0:4000`; use loopback and the master key:\n\n```python\nimport os\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://127.0.0.1:4000/v1\",\n    api_key=os.environ[\"LITELLM_MASTER_KEY\"],\n)\nresponse = client.chat.completions.create(\n    model=\"local-model\",\n    messages=[{\"role\": \"user\", \"content\": \"Reply with the word pong.\"}],\n)\nprint(response.choices[0].message.content)\n```\n\nDo not use LiteLLM's `--test` flag as a \"local\" check. The official CLI help describes it as making a test request, and the docs route that path at a Hugging Face inference example (`gpt-3.5-turbo` / `bigcode/starcoder`). That leaves the machine. Use `--validate_config`, or a client request against a server you started on `127.0.0.1`.\n\nSee the official [security best practices](https://docs.litellm.ai/docs/proxy/security_best_practices) for storing the master key and provider credentials outside `config.yaml`.\n\n## What was verified on this box\n\nOn 15 November 2026 this box installed `litellm[proxy]==1.109.1` (Python 3.13). `litellm --config ... --validate_config` exited 0 on a config whose `api_base` was `http://127.0.0.1:18080/v1`. Starting the proxy with `--host 127.0.0.1 --port 14000` and no `LITELLM_MASTER_KEY` failed with `UnsafeMasterKeyError`. With a locally generated `LITELLM_MASTER_KEY`, the same proxy listened only on `127.0.0.1:14000`. A loopback mock OpenAI server on `127.0.0.1:18080` answered `POST /v1/chat/completions`; both `curl` to `/chat/completions` and the OpenAI SDK (`base_url=\"http://127.0.0.1:14000/v1\"`) returned the mock body `mock-ok`. Both processes were stopped afterwards. No vLLM, SGLang, Ollama or LocalAI server was installed or started, and no model weights were downloaded.\n\n## Next step\n\nWire this endpoint into a local harness, memory store and sandbox with [Fully local, private agent](https://indexagentica.com/stacks/fully-local-private-agent/). [Pi](https://indexagentica.com/entries/pi-coding-agent/) documents a `models.json` provider with `\"baseUrl\": \"http://localhost:11434/v1\"` and `\"api\": \"openai-completions\"` for Ollama, and the same compatible-endpoint path for vLLM, SGLang and other OpenAI-style servers.\n",
  "html": "<h2>What this guide is (and is not)</h2>\n<p>This is a how-to: start a local model server so it listens only on loopback, point an <a href=\"https://github.com/openai/openai-python\">OpenAI Python SDK</a> client at that server&#39;s OpenAI-compatible <code>/v1</code> URL, send a tool-calling request, and optionally put <a href=\"/entries/litellm/\">LiteLLM</a> in front as a local router.</p>\n<p>It is not a feature-by-feature bake-off of local servers. For Ollama versus llama.cpp — tools, grammars, default ports and licences — see <a href=\"/compare/local-llm-servers-for-agents/\">Ollama vs llama.cpp for agents</a>. <a href=\"/entries/llama-cpp/\">llama.cpp</a> is not a fifth subject here; use that comparison if that is the server you want. It is also not a comparison of hosted LLM gateways. LiteLLM appears here only as a local proxy you run yourself. For OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, Portkey and LiteLLM as a <em>gateway product</em>, see <a href=\"/compare/llm-gateways/\">LLM gateways compared</a>.</p>\n<p>To assemble a whole on-machine agent (this serving path plus local memory and a local sandbox), see <a href=\"/stacks/fully-local-private-agent/\">Fully local, private agent</a>.</p>\n<p>Versions checked on 15 November 2026: <a href=\"/entries/ollama/\">Ollama</a> GitHub v0.40.16 (from the catalog entry and the 14 November comparison; not re-pulled as a server here); <a href=\"/entries/vllm/\">vLLM</a> PyPI <strong>0.33.0</strong>, while GitHub <code>/releases/latest</code> currently points at <strong>v0.33.0rc3</strong> with <code>prerelease</code> set to false (a <code>v0.33.0</code> tag also exists); <a href=\"/entries/sglang/\">SGLang</a> PyPI and GitHub <strong>v0.5.23</strong>; <a href=\"/entries/localai/\">LocalAI</a> GitHub <strong>v4.12.1</strong> (do not <code>pip install</code> the unrelated PyPI package named <code>localai</code>); <a href=\"/entries/litellm/\">LiteLLM</a> PyPI and GitHub <strong>v1.109.1</strong>, in sync. LiteLLM&#39;s SDK and open-source proxy are MIT; the repository&#39;s <code>enterprise/</code> tree is a separate licence and is not used here.</p>\n<h2>Bind the server to loopback first</h2>\n<p>These servers speak HTTP. If the process listens on all interfaces, anything that can reach that port can usually call the model. This guide always sets an explicit loopback bind. Official examples do not always do that:</p>\n<ul><li><a href=\"https://docs.ollama.com/faq\">Ollama&#39;s FAQ</a> states that the server binds <code>127.0.0.1:11434</code> by default. Change it with <code>OLLAMA_HOST</code> (their examples use <code>0.0.0.0:11434</code> when the goal is to expose the server).</li><li><a href=\"https://docs.vllm.ai/en/stable/cli/serve/\">vLLM&#39;s <code>vllm serve</code> CLI</a> documents <code>--port</code> with default <code>8000</code>. <code>--host</code> is documented only as &quot;Host name&quot;, with no default on that page, so pass <code>--host 127.0.0.1</code> explicitly. Official client examples use <code>http://localhost:8000/v1</code>.</li><li><a href=\"https://docs.sglang.io/docs/advanced_features/server_arguments\">SGLang server arguments</a> document <code>--host</code> default <code>127.0.0.1</code> and <code>--port</code> default <code>30000</code>. The <a href=\"https://docs.sglang.io/docs/get-started/quickstart\">quickstart</a> overrides this with <code>--host 0.0.0.0 --port 30000</code>. The <a href=\"https://docs.sglang.io/docs/get-started/examples\">examples</a> page shows both a Docker publish of <code>127.0.0.1:30000:30000</code> (the process inside the container still uses <code>--host 0.0.0.0</code>) and a native command with <code>--host 127.0.0.1 --port 30000</code>.</li><li><a href=\"https://localai.io/docs/start/quickstart/\">LocalAI&#39;s quickstart</a> states: &quot;By default, LocalAI listens on port 8080 on all interfaces, and authentication is off.&quot; The <a href=\"https://localai.io/docs/reference/cli-reference/\">CLI reference</a> documents <code>--address</code> / <code>$LOCALAI_ADDRESS</code> with default <code>:8080</code>. Pass <code>--address 127.0.0.1:8080</code>. Official Docker examples use <code>-p 8080:8080</code>, which publishes the port on all host interfaces; use <code>-p 127.0.0.1:8080:8080</code> instead.</li></ul>\n<p>Do not run vendor <code>install.sh</code> scripts from this guide. Follow each project&#39;s documented install for your OS, then start the server with the flags below.</p>\n<h2>Start an OpenAI-compatible server</h2>\n<p>Pick one. You only need a single <code>/v1</code> endpoint for the rest of the guide.</p>\n<h3>Ollama</h3>\n<p>Once the Ollama app or daemon is installed, it already listens on loopback. The <a href=\"https://docs.ollama.com/api/openai-compatibility\">OpenAI compatibility</a> page gives the local base URL as <code>http://localhost:11434/v1</code>.</p>\n<p>A generic OpenAI client looks like this (the dummy key is required by the SDK; Ollama does not treat it as a secret on the default local server):</p>\n<pre><code class=\"language-python\">from openai import OpenAI\n\nclient = OpenAI(base_url=&quot;http://127.0.0.1:11434/v1&quot;, api_key=&quot;ollama&quot;)</code></pre>\n<p>Use a model name you have already pulled. This guide does not download a model.</p>\n<h3>vLLM</h3>\n<p>Cite the <a href=\"https://docs.vllm.ai/en/stable/serving/online_serving/\">stable</a> docs, not <code>/en/latest/</code> (that tree is a developer preview). A loopback server, combining the official <code>vllm serve</code> form with the documented <code>--host</code> / <code>--port</code> flags:</p>\n<pre><code class=\"language-bash\">vllm serve MODEL \\\n    --host 127.0.0.1 \\\n    --port 8000</code></pre>\n<p>Replace <code>MODEL</code> with a Hugging Face model id your install can load. The official tool-calling quickstart uses <code>meta-llama/Llama-3.1-8B-Instruct</code> and was not run here.</p>\n<p>The <a href=\"https://docs.vllm.ai/en/stable/usage/security/\">security</a> page states that <code>--api-key</code> (or <code>VLLM_API_KEY</code>) authenticates only endpoints under the <code>/v1</code>, <code>/v2</code>, <code>/inference</code> and <code>/cohere</code> prefixes. Do not treat <code>--api-key</code> as protection for the whole process; bind to loopback first.</p>\n<p>Official client (from the <a href=\"https://docs.vllm.ai/en/stable/features/tool_calling/\">tool-calling</a> page):</p>\n<pre><code class=\"language-python\">from openai import OpenAI\n\nclient = OpenAI(base_url=&quot;http://localhost:8000/v1&quot;, api_key=&quot;dummy&quot;)</code></pre>\n<h3>SGLang</h3>\n<p>From the official <a href=\"https://docs.sglang.io/docs/get-started/examples\">examples</a> native path, which already uses loopback:</p>\n<pre><code class=\"language-bash\">sglang serve MODEL_PATH --host 127.0.0.1 --port 30000</code></pre>\n<p>If you copy the <a href=\"https://docs.sglang.io/docs/get-started/quickstart\">quickstart</a> command, it passes <code>--host 0.0.0.0 --port 30000</code>. Change <code>--host</code> to <code>127.0.0.1</code> unless you intend to publish the port. Official Docker examples map <code>-p 30000:30000</code> or <code>-p 127.0.0.1:30000:30000</code> and still pass <code>--host 0.0.0.0</code> <em>inside</em> the container; the host-side <code>-p</code> is what decides whether the port is reachable off-machine.</p>\n<p>Official client (same examples page):</p>\n<pre><code class=\"language-python\">from openai import OpenAI\n\nclient = OpenAI(\n    base_url=&quot;http://localhost:30000/v1&quot;,\n    api_key=&quot;EMPTY&quot;,\n)</code></pre>\n<p>The server is ready when it logs <code>The server is fired up and ready to roll!</code>.</p>\n<h3>LocalAI</h3>\n<p>The documented command is <code>local-ai run</code>. Bind loopback with the official <code>--address</code> flag:</p>\n<pre><code class=\"language-bash\">local-ai run --address 127.0.0.1:8080</code></pre>\n<p>The official quickstart also shows <code>local-ai run qwen3-4b</code>, which downloads that model on first run, and <code>docker run -p 8080:8080 --name local-ai -ti localai/localai:latest</code>. Prefer a host publish of <code>127.0.0.1:8080:8080</code> if you use Docker. The <a href=\"https://localai.io/docs/api/openai-functions/\">OpenAI functions</a> page uses <code>http://localhost:8080/v1/</code> as <code>base_url</code>.</p>\n<p>Official client (from that page; <code>api_key=&quot;test&quot;</code> is their example):</p>\n<pre><code class=\"language-python\">from openai import OpenAI\n\nclient = OpenAI(\n    api_key=&quot;test&quot;,\n    base_url=&quot;http://localhost:8080/v1/&quot;\n)</code></pre>\n<h2>Point the OpenAI SDK at <code>base_url</code></h2>\n<p>The pattern is the same for every server above: construct <code>OpenAI(base_url=..., api_key=...)</code> and call <code>chat.completions.create</code>. Use the host and port you actually bound. <code>localhost</code> and <code>127.0.0.1</code> are equivalent on a typical machine; this guide prefers <code>127.0.0.1</code> in commands we write, and leaves official snippets unchanged when quoting them.</p>\n<p>A minimal completion, with the Ollama URL as the example:</p>\n<pre><code class=\"language-python\">from openai import OpenAI\n\nclient = OpenAI(base_url=&quot;http://127.0.0.1:11434/v1&quot;, api_key=&quot;ollama&quot;)\nresponse = client.chat.completions.create(\n    model=&quot;MODEL&quot;,\n    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Reply with the word pong.&quot;}],\n)\nprint(response.choices[0].message.content)</code></pre>\n<p>Replace <code>MODEL</code> with a name the server already has. This snippet was not sent to a real local model on this box.</p>\n<h2>Enable tool calling</h2>\n<p>Tool calling is a server <em>and</em> a model feature. The request shape is the OpenAI <code>tools</code> array. The server often needs an extra flag so it can parse the model&#39;s tool calls back into that schema.</p>\n<p><strong>vLLM.</strong> The <a href=\"https://docs.vllm.ai/en/stable/features/tool_calling/\">tool-calling</a> page marks <code>--enable-auto-tool-choice</code> as mandatory for auto tool choice, and <code>--tool-call-parser</code> as the parser to use (required with that flag). Their Llama 3.1 8B quickstart is:</p>\n<pre><code class=\"language-bash\">vllm serve meta-llama/Llama-3.1-8B-Instruct \\\n    --enable-auto-tool-choice \\\n    --tool-call-parser llama3_json \\\n    --chat-template examples/tool_chat_template_llama3.1_json.jinja</code></pre>\n<p>Add <code>--host 127.0.0.1 --port 8000</code> as above. That page documents <code>tool_choice</code> values <code>auto</code>, <code>required</code> (as of <code>vllm&gt;=0.8.3</code>) and <code>none</code>, plus named function calling. Their official Python example builds a <code>get_weather</code> tool, passes <code>tools=tools, tool_choice=&quot;auto&quot;</code>, and reads <code>response.choices[0].message.tool_calls</code>.</p>\n<p><strong>SGLang.</strong> Official <a href=\"https://docs.sglang.io/docs/get-started/examples\">examples</a> pass <code>--tool-call-parser qwen3_coder</code> on the Qwen3.8-27B-FP8 serve command. That same page notes that its small local Qwen3-0.6B example does <em>not</em> configure tool calling, and points at the tool-calling guide for other models. The OpenAI client is still <code>base_url=&quot;http://localhost:30000/v1&quot;</code> with <code>api_key=&quot;EMPTY&quot;</code>.</p>\n<p><strong>Ollama.</strong> The <a href=\"https://docs.ollama.com/api/openai-compatibility\">OpenAI compatibility</a> page documents <code>tools</code> as supported on <code>/v1/chat/completions</code> and <code>tool_choice</code> as not supported on that route. Use the native tools API or omit <code>tool_choice</code> on <code>/v1</code>. See the <a href=\"https://docs.ollama.com/capabilities/tool-calling\">tool-calling</a> docs and the <a href=\"/compare/local-llm-servers-for-agents/\">Ollama vs llama.cpp</a> comparison for the native versus OpenAI surface; this guide does not repeat that matrix.</p>\n<p><strong>LocalAI.</strong> The <a href=\"https://localai.io/docs/api/openai-functions/\">OpenAI functions</a> page shows the same <code>tools</code> / <code>tool_choice=&quot;auto&quot;</code> shape against <code>http://localhost:8080/v1/</code>. Use a model you have installed that actually supports tools (their quickstart names <code>qwen3-4b</code> as a small CPU model that does).</p>\n<p>An official-style request, abbreviated from the vLLM quickstart (not executed here):</p>\n<pre><code class=\"language-python\">tools = [\n    {\n        &quot;type&quot;: &quot;function&quot;,\n        &quot;function&quot;: {\n            &quot;name&quot;: &quot;get_weather&quot;,\n            &quot;description&quot;: &quot;Get the current weather in a given location&quot;,\n            &quot;parameters&quot;: {\n                &quot;type&quot;: &quot;object&quot;,\n                &quot;properties&quot;: {\n                    &quot;location&quot;: {&quot;type&quot;: &quot;string&quot;},\n                    &quot;unit&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;enum&quot;: [&quot;celsius&quot;, &quot;fahrenheit&quot;]},\n                },\n                &quot;required&quot;: [&quot;location&quot;, &quot;unit&quot;],\n            },\n        },\n    },\n]\n\nresponse = client.chat.completions.create(\n    model=&quot;MODEL&quot;,\n    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is the weather in San Francisco?&quot;}],\n    tools=tools,\n    tool_choice=&quot;auto&quot;,  # omit on Ollama /v1/chat/completions\n)</code></pre>\n<p>Then execute the returned <code>tool_calls</code> in your process and send the tool result back on a message with <code>role=&quot;tool&quot;</code>, following the provider&#39;s example.</p>\n<h2>Route through LiteLLM</h2>\n<p>Use <a href=\"/entries/litellm/\">LiteLLM</a> when you want one local OpenAI-compatible URL in front of whichever server you started, or when a client already speaks &quot;OpenAI&quot; and you would rather change a config file than the client. This is a router on <code>127.0.0.1</code>, not the hosted-gateway comparison.</p>\n<p>Install the proxy extra (PyPI <strong>1.109.1</strong> on 15 November 2026, matching GitHub <code>v1.109.1</code>):</p>\n<pre><code class=\"language-bash\">pip install &#39;litellm[proxy]&#39;</code></pre>\n<p>The <a href=\"https://docs.litellm.ai/docs/providers/openai_compatible\">OpenAI-compatible endpoints</a> page says to prefix the model with <code>openai/</code>, set <code>api_base</code> to your server, and pass an <code>api_key</code> (the OpenAI provider path requires one; a dummy value is enough when the backend ignores it). If you see a <code>Not Found</code> error, that page says to give <code>api_base</code> a <code>/v1</code> postfix. A config that points at a loopback Ollama server:</p>\n<pre><code class=\"language-yaml\">model_list:\n  - model_name: local-model\n    litellm_params:\n      model: openai/MODEL\n      api_base: http://127.0.0.1:11434/v1\n      api_key: dummy</code></pre>\n<p>Change <code>api_base</code> (and the model id after <code>openai/</code>) for vLLM (<code>http://127.0.0.1:8000/v1</code>), SGLang (<code>http://127.0.0.1:30000/v1</code>) or LocalAI (<code>http://127.0.0.1:8080/v1</code>). LiteLLM also documents provider prefixes such as <code>ollama/</code> with <code>api_base: http://localhost:11434</code> and <code>hosted_vllm/</code>; those are alternatives, not a second required router.</p>\n<p>Check the file without starting a server (LiteLLM 1.109.1 CLI: &quot;Load and validate the config file ... then exit&quot;):</p>\n<pre><code class=\"language-bash\">litellm --config /path/to/config.yaml --validate_config</code></pre>\n<p>Start the proxy on loopback. The 1.109.1 CLI documents <code>--host</code> and <code>--port</code>. Official <a href=\"https://docs.litellm.ai/docs/proxy/quick_start\">CLI</a> logs say <code>Proxy running on http://0.0.0.0:4000</code> when you do not pass <code>--host</code>; pass it:</p>\n<pre><code class=\"language-bash\">export LITELLM_MASTER_KEY=&quot;sk-$(openssl rand -hex 32)&quot;\nlitellm --config /path/to/config.yaml --host 127.0.0.1 --port 4000</code></pre>\n<p>Keep that value out of the config file and out of source control. The <a href=\"https://docs.litellm.ai/docs/proxy/config_settings\">config settings</a> page states that the proxy will not start when the master key is unset, empty, or has a known unsafe value, and that <code>general_settings.master_key</code> falls back to <code>LITELLM_MASTER_KEY</code>. 1.109.1 on this box refused to boot without it, with the same message. The older wording on the <a href=\"https://docs.litellm.ai/docs/proxy/configs\">configs</a> page that still calls <code>master_key</code> optional lags that behaviour. There is a documented local-only escape hatch, <code>LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY</code>, which this guide does not recommend.</p>\n<p>The official <a href=\"https://docs.litellm.ai/docs/proxy/docker_quick_start\">Docker quick start</a> already publishes loopback and expects a master key:</p>\n<pre><code class=\"language-bash\">docker run \\\n  -v &quot;$PWD/litellm_config.yaml:/app/config.yaml:ro&quot; \\\n  -e LITELLM_MASTER_KEY \\\n  -p 127.0.0.1:4000:4000 \\\n  docker.litellm.ai/berriai/litellm:latest \\\n  --config /app/config.yaml</code></pre>\n<p>Call the proxy with the OpenAI SDK. Official snippets often use <code>http://0.0.0.0:4000</code>; use loopback and the master key:</p>\n<pre><code class=\"language-python\">import os\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=&quot;http://127.0.0.1:4000/v1&quot;,\n    api_key=os.environ[&quot;LITELLM_MASTER_KEY&quot;],\n)\nresponse = client.chat.completions.create(\n    model=&quot;local-model&quot;,\n    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Reply with the word pong.&quot;}],\n)\nprint(response.choices[0].message.content)</code></pre>\n<p>Do not use LiteLLM&#39;s <code>--test</code> flag as a &quot;local&quot; check. The official CLI help describes it as making a test request, and the docs route that path at a Hugging Face inference example (<code>gpt-3.5-turbo</code> / <code>bigcode/starcoder</code>). That leaves the machine. Use <code>--validate_config</code>, or a client request against a server you started on <code>127.0.0.1</code>.</p>\n<p>See the official <a href=\"https://docs.litellm.ai/docs/proxy/security_best_practices\">security best practices</a> for storing the master key and provider credentials outside <code>config.yaml</code>.</p>\n<h2>What was verified on this box</h2>\n<p>On 15 November 2026 this box installed <code>litellm[proxy]==1.109.1</code> (Python 3.13). <code>litellm --config ... --validate_config</code> exited 0 on a config whose <code>api_base</code> was <code>http://127.0.0.1:18080/v1</code>. Starting the proxy with <code>--host 127.0.0.1 --port 14000</code> and no <code>LITELLM_MASTER_KEY</code> failed with <code>UnsafeMasterKeyError</code>. With a locally generated <code>LITELLM_MASTER_KEY</code>, the same proxy listened only on <code>127.0.0.1:14000</code>. A loopback mock OpenAI server on <code>127.0.0.1:18080</code> answered <code>POST /v1/chat/completions</code>; both <code>curl</code> to <code>/chat/completions</code> and the OpenAI SDK (<code>base_url=&quot;http://127.0.0.1:14000/v1&quot;</code>) returned the mock body <code>mock-ok</code>. Both processes were stopped afterwards. No vLLM, SGLang, Ollama or LocalAI server was installed or started, and no model weights were downloaded.</p>\n<h2>Next step</h2>\n<p>Wire this endpoint into a local harness, memory store and sandbox with <a href=\"/stacks/fully-local-private-agent/\">Fully local, private agent</a>. <a href=\"/entries/pi-coding-agent/\">Pi</a> documents a <code>models.json</code> provider with <code>&quot;baseUrl&quot;: &quot;http://localhost:11434/v1&quot;</code> and <code>&quot;api&quot;: &quot;openai-completions&quot;</code> for Ollama, and the same compatible-endpoint path for vLLM, SGLang and other OpenAI-style servers.</p>"
}
