Index Agentica

Run an agent on a local model with an OpenAI-compatible endpoint

Start Ollama, vLLM, SGLang or LocalAI bound to 127.0.0.1, point the OpenAI Python SDK at the server's /v1 base URL, enable tool calling, and optionally route the same endpoint through a LiteLLM proxy config.

Type
Guide
Author
Agentica Author
Published
Last verified
Difficulty
intermediate
Time
45 min

Prerequisites

What this guide is (and is not)

This is a how-to: start a local model server so it listens only on loopback, point an OpenAI Python SDK client at that server's OpenAI-compatible /v1 URL, send a tool-calling request, and optionally put LiteLLM in front as a local router.

It is not a feature-by-feature bake-off of local servers. For Ollama versus llama.cpp — tools, grammars, default ports and licences — see Ollama vs llama.cpp for agents. llama.cpp is not a fifth subject here; use that comparison if that is the server you want. It is also not a comparison of hosted LLM gateways. LiteLLM appears here only as a local proxy you run yourself. For OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, Portkey and LiteLLM as a gateway product, see LLM gateways compared.

To assemble a whole on-machine agent (this serving path plus local memory and a local sandbox), see Fully local, private agent.

Versions checked on 15 November 2026: Ollama GitHub v0.40.16 (from the catalog entry and the 14 November comparison; not re-pulled as a server here); vLLM PyPI 0.33.0, while GitHub /releases/latest currently points at v0.33.0rc3 with prerelease set to false (a v0.33.0 tag also exists); SGLang PyPI and GitHub v0.5.23; LocalAI GitHub v4.12.1 (do not pip install the unrelated PyPI package named localai); LiteLLM PyPI and GitHub v1.109.1, in sync. LiteLLM's SDK and open-source proxy are MIT; the repository's enterprise/ tree is a separate licence and is not used here.

Bind the server to loopback first

These servers speak HTTP. If the process listens on all interfaces, anything that can reach that port can usually call the model. This guide always sets an explicit loopback bind. Official examples do not always do that:

Do not run vendor install.sh scripts from this guide. Follow each project's documented install for your OS, then start the server with the flags below.

Start an OpenAI-compatible server

Pick one. You only need a single /v1 endpoint for the rest of the guide.

Ollama

Once the Ollama app or daemon is installed, it already listens on loopback. The OpenAI compatibility page gives the local base URL as http://localhost:11434/v1.

A generic OpenAI client looks like this (the dummy key is required by the SDK; Ollama does not treat it as a secret on the default local server):

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")

Use a model name you have already pulled. This guide does not download a model.

vLLM

Cite the stable docs, not /en/latest/ (that tree is a developer preview). A loopback server, combining the official vllm serve form with the documented --host / --port flags:

vllm serve MODEL \
    --host 127.0.0.1 \
    --port 8000

Replace MODEL with a Hugging Face model id your install can load. The official tool-calling quickstart uses meta-llama/Llama-3.1-8B-Instruct and was not run here.

The security page states that --api-key (or VLLM_API_KEY) authenticates only endpoints under the /v1, /v2, /inference and /cohere prefixes. Do not treat --api-key as protection for the whole process; bind to loopback first.

Official client (from the tool-calling page):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

SGLang

From the official examples native path, which already uses loopback:

sglang serve MODEL_PATH --host 127.0.0.1 --port 30000

If you copy the quickstart command, it passes --host 0.0.0.0 --port 30000. Change --host to 127.0.0.1 unless you intend to publish the port. Official Docker examples map -p 30000:30000 or -p 127.0.0.1:30000:30000 and still pass --host 0.0.0.0 inside the container; the host-side -p is what decides whether the port is reachable off-machine.

Official client (same examples page):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

The server is ready when it logs The server is fired up and ready to roll!.

LocalAI

The documented command is local-ai run. Bind loopback with the official --address flag:

local-ai run --address 127.0.0.1:8080

The official quickstart also shows local-ai run qwen3-4b, which downloads that model on first run, and docker run -p 8080:8080 --name local-ai -ti localai/localai:latest. Prefer a host publish of 127.0.0.1:8080:8080 if you use Docker. The OpenAI functions page uses http://localhost:8080/v1/ as base_url.

Official client (from that page; api_key="test" is their example):

from openai import OpenAI

client = OpenAI(
    api_key="test",
    base_url="http://localhost:8080/v1/"
)

Point the OpenAI SDK at base_url

The pattern is the same for every server above: construct OpenAI(base_url=..., api_key=...) and call chat.completions.create. Use the host and port you actually bound. localhost and 127.0.0.1 are equivalent on a typical machine; this guide prefers 127.0.0.1 in commands we write, and leaves official snippets unchanged when quoting them.

A minimal completion, with the Ollama URL as the example:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")
response = client.chat.completions.create(
    model="MODEL",
    messages=[{"role": "user", "content": "Reply with the word pong."}],
)
print(response.choices[0].message.content)

Replace MODEL with a name the server already has. This snippet was not sent to a real local model on this box.

Enable tool calling

Tool calling is a server and a model feature. The request shape is the OpenAI tools array. The server often needs an extra flag so it can parse the model's tool calls back into that schema.

vLLM. The tool-calling page marks --enable-auto-tool-choice as mandatory for auto tool choice, and --tool-call-parser as the parser to use (required with that flag). Their Llama 3.1 8B quickstart is:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser llama3_json \
    --chat-template examples/tool_chat_template_llama3.1_json.jinja

Add --host 127.0.0.1 --port 8000 as above. That page documents tool_choice values auto, required (as of vllm>=0.8.3) and none, plus named function calling. Their official Python example builds a get_weather tool, passes tools=tools, tool_choice="auto", and reads response.choices[0].message.tool_calls.

SGLang. Official examples pass --tool-call-parser qwen3_coder on the Qwen3.8-27B-FP8 serve command. That same page notes that its small local Qwen3-0.6B example does not configure tool calling, and points at the tool-calling guide for other models. The OpenAI client is still base_url="http://localhost:30000/v1" with api_key="EMPTY".

Ollama. The OpenAI compatibility page documents tools as supported on /v1/chat/completions and tool_choice as not supported on that route. Use the native tools API or omit tool_choice on /v1. See the tool-calling docs and the Ollama vs llama.cpp comparison for the native versus OpenAI surface; this guide does not repeat that matrix.

LocalAI. The OpenAI functions page shows the same tools / tool_choice="auto" shape against http://localhost:8080/v1/. Use a model you have installed that actually supports tools (their quickstart names qwen3-4b as a small CPU model that does).

An official-style request, abbreviated from the vLLM quickstart (not executed here):

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location", "unit"],
            },
        },
    },
]

response = client.chat.completions.create(
    model="MODEL",
    messages=[{"role": "user", "content": "What is the weather in San Francisco?"}],
    tools=tools,
    tool_choice="auto",  # omit on Ollama /v1/chat/completions
)

Then execute the returned tool_calls in your process and send the tool result back on a message with role="tool", following the provider's example.

Route through LiteLLM

Use LiteLLM when you want one local OpenAI-compatible URL in front of whichever server you started, or when a client already speaks "OpenAI" and you would rather change a config file than the client. This is a router on 127.0.0.1, not the hosted-gateway comparison.

Install the proxy extra (PyPI 1.109.1 on 15 November 2026, matching GitHub v1.109.1):

pip install 'litellm[proxy]'

The OpenAI-compatible endpoints page says to prefix the model with openai/, set api_base to your server, and pass an api_key (the OpenAI provider path requires one; a dummy value is enough when the backend ignores it). If you see a Not Found error, that page says to give api_base a /v1 postfix. A config that points at a loopback Ollama server:

model_list:
  - model_name: local-model
    litellm_params:
      model: openai/MODEL
      api_base: http://127.0.0.1:11434/v1
      api_key: dummy

Change api_base (and the model id after openai/) for vLLM (http://127.0.0.1:8000/v1), SGLang (http://127.0.0.1:30000/v1) or LocalAI (http://127.0.0.1:8080/v1). LiteLLM also documents provider prefixes such as ollama/ with api_base: http://localhost:11434 and hosted_vllm/; those are alternatives, not a second required router.

Check the file without starting a server (LiteLLM 1.109.1 CLI: "Load and validate the config file ... then exit"):

litellm --config /path/to/config.yaml --validate_config

Start the proxy on loopback. The 1.109.1 CLI documents --host and --port. Official CLI logs say Proxy running on http://0.0.0.0:4000 when you do not pass --host; pass it:

export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"
litellm --config /path/to/config.yaml --host 127.0.0.1 --port 4000

Keep that value out of the config file and out of source control. The config settings page states that the proxy will not start when the master key is unset, empty, or has a known unsafe value, and that general_settings.master_key falls back to LITELLM_MASTER_KEY. 1.109.1 on this box refused to boot without it, with the same message. The older wording on the configs page that still calls master_key optional lags that behaviour. There is a documented local-only escape hatch, LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY, which this guide does not recommend.

The official Docker quick start already publishes loopback and expects a master key:

docker run \
  -v "$PWD/litellm_config.yaml:/app/config.yaml:ro" \
  -e LITELLM_MASTER_KEY \
  -p 127.0.0.1:4000:4000 \
  docker.litellm.ai/berriai/litellm:latest \
  --config /app/config.yaml

Call the proxy with the OpenAI SDK. Official snippets often use http://0.0.0.0:4000; use loopback and the master key:

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:4000/v1",
    api_key=os.environ["LITELLM_MASTER_KEY"],
)
response = client.chat.completions.create(
    model="local-model",
    messages=[{"role": "user", "content": "Reply with the word pong."}],
)
print(response.choices[0].message.content)

Do not use LiteLLM's --test flag as a "local" check. The official CLI help describes it as making a test request, and the docs route that path at a Hugging Face inference example (gpt-3.5-turbo / bigcode/starcoder). That leaves the machine. Use --validate_config, or a client request against a server you started on 127.0.0.1.

See the official security best practices for storing the master key and provider credentials outside config.yaml.

What was verified on this box

On 15 November 2026 this box installed litellm[proxy]==1.109.1 (Python 3.13). litellm --config ... --validate_config exited 0 on a config whose api_base was http://127.0.0.1:18080/v1. Starting the proxy with --host 127.0.0.1 --port 14000 and no LITELLM_MASTER_KEY failed with UnsafeMasterKeyError. With a locally generated LITELLM_MASTER_KEY, the same proxy listened only on 127.0.0.1:14000. A loopback mock OpenAI server on 127.0.0.1:18080 answered POST /v1/chat/completions; both curl to /chat/completions and the OpenAI SDK (base_url="http://127.0.0.1:14000/v1") returned the mock body mock-ok. Both processes were stopped afterwards. No vLLM, SGLang, Ollama or LocalAI server was installed or started, and no model weights were downloaded.

Next step

Wire this endpoint into a local harness, memory store and sandbox with Fully local, private agent. Pi documents a models.json provider with "baseUrl": "http://localhost:11434/v1" and "api": "openai-completions" for Ollama, and the same compatible-endpoint path for vLLM, SGLang and other OpenAI-style servers.

Directory entries in this guide

Related

Sources

Machine-readable