Ollama vs llama.cpp for agents — tools, OpenAI APIs, grammars and local serving
Compare Ollama and llama.cpp as local model servers for agents: tool and function calling, OpenAI-compatible endpoints, structured output and grammars, how each is served, licences and current versions. Distinct from the structured-output libraries comparison and from LLM gateways.
Comparison
| Entry | What you run | Tool and function calling | OpenAI-compatible API | Structured output and grammars | Local serving and cloud | License and versions |
|---|---|---|---|---|---|---|
| Ollama | An MIT app and local server, plus official Python and JavaScript clients. Installers are documented for macOS, Windows and Linux (including a documented Linux install script, not executed) and the Docker image ollama/ollama. The README lists llama.cpp as its supported inference backend. | Official docs cover tool calling on /api/chat with OpenAI-style tools and tool result messages (role tool). The Python client can pass functions or a JSON schema in tools. ollama launch is documented for integrations including Claude Code, Codex, Copilot CLI, OpenCode and OpenClaw. | Local base URL http://localhost:11434/v1 (cloud OpenAI-compat path is /v1 on ollama.com). /v1/chat/completions documents tools as supported and tool_choice as not supported, plus JSON mode and response_format. Also documents /v1/responses, embeddings, audio, and Anthropic-compatible /v1/messages. | The structured-outputs guide uses a format field of json or a JSON schema (Python can pass Pydantic model_json_schema()). Official note: Ollama's cloud currently does not support structured outputs. GBNF grammars are not documented as an Ollama API. | FAQ: binds 127.0.0.1:11434 by default; change with OLLAMA_HOST (examples use 0.0.0.0:11434 to expose the server). Optional paid Ollama Cloud needs an account and API key; cloud features can be disabled in ~/.ollama/server.json. See ollama.com/pricing; rates are not copied here. | App/server MIT (repository LICENSE). GitHub latest v0.40.16 (9 November 2026). Official Python client PyPI ollama 0.6.3 (29 September 2026, MIT, Python >=3.8, repo ollama/ollama-python). |
| llama.cpp | MIT C/C++ library, CLI and HTTP server (`llama serve` in the README; server docs also name `llama-server`). Install via documented binaries, Docker, source, or conda-forge/Homebrew/Winget/MacPorts/Nix. Not the third-party PyPI package llama-cpp-python. | OpenAI-style function calling, used with --jinja (enabled by default in the current flag list). Native handlers for several model families, plus a generic Jinja fallback. Experimental built-in tools and stdio MCP sit behind --tools / --agent; docs warn not to enable them in untrusted environments. | Default http://127.0.0.1:9931/v1. Documents /v1/chat/completions, /v1/responses, /v1/completions, /v1/embeddings, /v1/audio/speech, and Anthropic-compatible /v1/messages. The server README says it makes no strong compatibility claims, but that the API has been enough for many apps. | Built-in GBNF (--grammar / grammar) and a JSON Schema-to-GBNF path (--json-schema, json_schema, or response_format). Optional llguidance via cmake -DLLAMA_LLGUIDANCE=ON (grammars prefixed %llguidance). See the structured-output libraries comparison for engine-level coverage, not repeated here. | Server default is 127.0.0.1:9931 (--host / --port). Docker examples pass --host 0.0.0.0 when publishing a port. llama.app is a desktop app by the llama.cpp team and Hugging Face; this page is the llama.cpp project, not that app. No hosted cloud product is documented in llama.cpp itself. | MIT (repository LICENSE, copyright 2023-2026 The ggml authors). GitHub latest v1.0.0 (13 November 2026). There is no official first-party PyPI package for this repo. |
Verdict: Want an app, a model library, official Python/JS clients, documented coding-agent launchers, and an optional paid cloud path? Start with Ollama, and remember it documents llama.cpp as its inference backend. Want the C/C++ server itself, GBNF or JSON-schema constraints (and optional llguidance), a default loopback server on port 9931, or experimental built-in tools/MCP? Use llama.cpp directly. This is a serving layer choice, not a substitute for the structured-output libraries comparison or for hosted LLM gateways.
Local servers for agents, not gateways or grammar libraries
LLM gateways compared covers hosted and self-hosted proxies in front of many model providers. Structured-output libraries compared covers constraint engines such as Outlines, XGrammar, Guidance and llguidance, some of which can sit on an Ollama or llama.cpp client. Personal agent runtimes mentions llama.cpp as one model backend for Pi. This page is the layer in between: how you serve an open model locally so an agent can call tools against it.
The two directory entries are Ollama and llama.cpp. They are not unrelated stacks. Ollama's README lists llama.cpp as its supported inference backend. Treat Ollama as the app, API and model-library layer; treat llama.cpp as the C/C++ engine and HTTP server you can also run yourself.
The comparison table is generated from the front matter. Do not read vendor speed claims, token prices or catalog sizes as independent measurements. Ollama Cloud list prices live on ollama.com/pricing and are not copied here.
What you actually install
Ollama is an MIT-licensed app and daemon with official macOS, Windows and Linux installers, including a documented Linux install script (not executed here), plus the Docker image ollama/ollama. Official clients are ollama-python and ollama-js. The docs describe ollama launch for wiring Claude Code, Codex, Copilot CLI, OpenCode and OpenClaw, among others.
llama.cpp is an MIT-licensed C/C++ library with a CLI and HTTP server. The project README's quick start uses llama cli and llama serve; the server and function-calling docs still name llama-server. Documented install paths include https://llama.app/install.sh (not executed), Docker, GitHub release binaries, building from source, and packages listed in the install guide (conda-forge, Homebrew, Winget, MacPorts, Nix). Do not treat the third-party PyPI package llama-cpp-python (abetlen/llama-cpp-python) as the official install path for this repository. llama.app is a Mac/Windows desktop app described as being by the llama.cpp team and Hugging Face; this comparison is the llama.cpp project, not a separate listing of that app.
Tool calling and compatible APIs
Ollama documents tool calling on its native /api/chat endpoint: a tools array of function definitions, assistant tool_calls, and follow-up role: tool messages. The Python client can pass Python functions or a JSON schema in tools. The OpenAI compatibility page puts the same local server at http://localhost:11434/v1. For /v1/chat/completions it marks tools as supported and tool_choice as not supported, and it also documents /v1/responses (including tools), embeddings, audio, and an Anthropic-compatible messages API. Direct cloud access uses the ollama.com host with the /v1 (OpenAI-compatible) or /api (native) paths and an API key, as documented in the API introduction.
llama.cpp documents an OpenAI-compatible server, defaulting to http://127.0.0.1:9931. The server README lists /v1/chat/completions, /v1/responses, /v1/completions, /v1/embeddings and /v1/audio/speech, plus Anthropic-compatible /v1/messages. It states that it makes no strong compatibility claims, but that the API has been enough for many apps. Function calling is described as OpenAI-style tool use on llama-server with --jinja. The current server flag table lists --jinja as enabled by default. Native format handlers are documented for several model families (including Llama 3.x, Hermes, Qwen 2.5, Mistral Nemo and GPT-OSS); otherwise a generic Jinja handler is used. Parallel tool calls are documented as off by default and enabled with "parallel_tool_calls": true on supported templates.
The same server also documents experimental built-in agent tools (--tools, or --agent to turn them all on) and stdio MCP servers. The docs warn not to enable those in untrusted environments, because they can expose file and shell access.
Structured output and grammars
Ollama's structured outputs guide uses a format field: the string "json", or a JSON schema object (the Python examples pass model_json_schema() from Pydantic). The same page states that Ollama's cloud currently does not support structured outputs. GBNF is not documented as an Ollama request parameter.
llama.cpp has a first-class GBNF grammar language (--grammar / --grammar-file, or a grammar field on completions) and a documented path from a subset of JSON Schema to GBNF (--json-schema, a json_schema field, or OpenAI-style response_format). The grammars guide notes that a JSON schema used this way constrains sampling and is not injected into the prompt, except for tool calling, where schemas are injected. Optional llguidance support is built with cmake -DLLAMA_LLGUIDANCE=ON; grammars that start with %llguidance, and JSON Schema requests, are then handled by llguidance. For engine-level constraint coverage, start from Structured-output libraries compared rather than repeating those matrices here.
Local bind addresses, and Ollama Cloud
Both projects document a loopback default, which matters if you start a server on a shared machine.
- Ollama's FAQ states that it binds 127.0.0.1 port 11434 by default, and that you change the bind address with
OLLAMA_HOST(its examples use0.0.0.0:11434when you intentionally expose the server). - llama.cpp's server flag table defaults
--hostto 127.0.0.1 and--portto 9931. Docker examples in that README pass--host 0.0.0.0when publishing a container port.
Ollama Cloud is an optional paid path for running larger models on Ollama's service. It requires an account. Docs say you can keep inference on your own hardware and disable cloud features with disable_ollama_cloud in ~/.ollama/server.json. Current plan and per-token prices are on ollama.com/pricing; they were not copied into this article. llama.cpp itself is not documented as a hosted cloud product.
Licences and versions (14 November 2026)
| Project | Licence file | What to pin | Notes |
|---|---|---|---|
| Ollama app/server | MIT (LICENSE, copyright Ollama) | GitHub latest v0.40.16 (9 November 2026) | Official Python client is a separate package: PyPI ollama 0.6.3 (29 September 2026, MIT, Python >=3.8) |
| llama.cpp | MIT (LICENSE, copyright 2023-2026 The ggml authors) | GitHub latest v1.0.0 (13 November 2026) | No official first-party PyPI package. Do not confuse this with third-party llama-cpp-python |
When linking the llama.cpp repository, use ggml-org/llama.cpp. The older ggerganov/llama.cpp path redirects to it. For Ollama's HTTP API docs, use docs.ollama.com/api/introduction and docs.ollama.com/api/openai-compatibility; the short /api and /openai paths redirect.
What we ran, and what we did not
We installed the official Python client ollama==0.6.3 in a virtualenv and imported it (package version 0.6.3; chat and Client are present). No Ollama or llama.cpp server was running (nothing listened on ports 11434 or 9931).
We did not execute ollama.com/install.sh or llama.app/install.sh, did not install the Ollama app or a llama.cpp binary, did not download a model (including the README's Qwen3.5-0.8B-GGUF example), and did not call Ollama Cloud or any paid API. No tool-calling or grammar example was executed against a live model.
A safe selection sequence
- Name the layer. If you need a multi-provider gateway, start from LLM gateways compared. If you need constraint libraries (Outlines, XGrammar, Guidance, llguidance), use Structured-output libraries compared. If you need a local OpenAI-compatible server for an agent, stay on this page.
- Decide whether you want the app or the engine. Ollama is the documented app/API layer (and it uses llama.cpp under the hood). llama.cpp is the engine and server you run directly when you want GBNF, JSON-schema constraints, optional llguidance, or the experimental tools/MCP flags.
- Check the API your agent already speaks. Both document OpenAI-compatible
/v1/chat/completionsand an Anthropic-compatible messages API. If you rely ontool_choice, note that Ollama's OpenAI-compat table marks it unsupported. - Keep the server on loopback unless you have a reason to change
OLLAMA_HOSTor--host. Do not enable llama.cpp--toolsor--agenton an untrusted host. - Pin versions. Record v0.40.16 / v1.0.0 (or newer pins), the bind address and port, and whether structured output is required (not available on Ollama Cloud, per the structured-outputs guide).
Directory entries in this comparison
- Ollama: MIT app and API for running open models locally, plus a paid Ollama Cloud with usage credits and per-token prices.
- llama.cpp: MIT C/C++ library and server for local LLM inference, widely used as a local OpenAI-compatible backend for coding agents.
- llguidance: High-performance structured-output / constrained-decoding engine from the Guidance project (Rust core, Python bindings).
- OpenClaw: Open-source personal AI assistant that runs on your own machine behind one Gateway and talks to you over WhatsApp, Telegram, Slack, Discord, iMessage and 20+ other channels.
- Claude Code: Anthropic's agentic coding tool that reads your codebase, edits files, runs commands and integrates with dev tools; available in terminal, IDE, desktop and browser.
- Outlines: Structured LLM output library from .txt (dottxt): JSON/schema/grammar constraints with broad model integrations.
- XGrammar: Fast portable structured-generation engine for LLM grammars and constrained decoding (MLC AI).
- Guidance: Microsoft Research library for constrained LLM generation with a Python guidance language, token healing and multi-backend support.
Related
- Structured-output libraries compared — Outlines, XGrammar, Guidance and llguidance (Comparison)
- OpenClaw vs Hermes Agent vs Pi: open-source agents you run yourself (Comparison)
- LLM gateways compared: OpenRouter vs Vercel AI Gateway vs Cloudflare AI Gateway vs Portkey vs LiteLLM (Comparison)
- Terminal coding agents compared (Comparison)
Sources
- ollama/ollama README, accessed
- ollama/ollama LICENSE, accessed
- ollama/ollama latest GitHub release, accessed
- Ollama documentation, accessed
- Ollama, Tool calling, accessed
- Ollama, Structured Outputs, accessed
- Ollama, API introduction, accessed
- Ollama, OpenAI compatibility, accessed
- Ollama, Anthropic compatibility, accessed
- Ollama, Cloud, accessed
- Ollama FAQ, accessed
- Ollama quickstart, accessed
- Ollama CLI reference, accessed
- Ollama pricing, accessed
- ollama PyPI metadata, accessed
- ollama/ollama-python, accessed
- ggml-org/llama.cpp README, accessed
- ggml-org/llama.cpp LICENSE, accessed
- ggml-org/llama.cpp latest GitHub release, accessed
- llama.cpp v1.0.0 release, accessed
- llama.cpp HTTP server README, accessed
- llama.cpp function calling, accessed
- llama.cpp GBNF guide, accessed
- llama.cpp LLGuidance support, accessed
- llama.cpp install pre-built binaries, accessed
- llama.cpp build guide, accessed
- llama.app, accessed