vLLM
Apache-2.0 high-throughput LLM inference and serving engine with an OpenAI-compatible API, installable via uv or pip.
Tags
Description
Established open project (repo vllm-project/vllm; not archived; created 2023-02-09; originally developed in the Sky Computing Lab; not presented as a 2026 launch). Root LICENSE is Apache-2.0. Site https://vllm.ai and docs https://docs.vllm.ai/en/latest/ were HTTP 200 on 2026-11-15; the docs root https://docs.vllm.ai 302s to /en/latest/ and is not used here. PyPI vllm 0.33.0 (not yanked; first wheel uploaded 2026-11-02T10:12:56Z = 2026-11-02 11:12:56 Europe/Stockholm). GitHub Releases “latest” is tag v0.33.0rc3, published 2026-11-02T10:10:05Z = 2026-11-02 11:10:05 Europe/Stockholm, with prerelease=false despite “rc3” in the tag name. A git tag v0.33.0 also exists (annotated tag; https://github.com/vllm-project/vllm/releases/tag/v0.33.0 is HTTP 200) but the GitHub Releases API has no Release object for v0.33.0 (404). Prefer stating PyPI 0.33.0 and that the GitHub latest release tag is v0.33.0rc3. README install: uv pip install vllm, or pip, or build from source. Documented install commands were not executed. Model weights and any GPU or host resources are licensed and billed separately. Local or cluster servers can execute the model and tool traffic you point at them.
Agent access
Related
- SGLang: Apache-2.0 inference framework for LLM, vision-language and diffusion models, optimized for agentic workloads and large-scale serving.
- llama.cpp: MIT C/C++ library and server for local LLM inference, widely used as a local OpenAI-compatible backend for coding agents.
- Ollama: MIT app and API for running open models locally, plus a paid Ollama Cloud with usage credits and per-token prices.
- LocalAI: MIT self-hosted OpenAI-compatible engine that runs LLM, vision, voice and image backends locally, with Docker images as the documented install.
- MLX LM: MIT Python package for text generation and fine-tuning of large language models on Apple silicon with MLX.
Guides & comparisons
- Run an agent on a local model with an OpenAI-compatible endpoint (Guide): Start Ollama, vLLM, SGLang or LocalAI bound to 127.0.0.1, point the OpenAI Python SDK at the server's /v1 base URL, enable tool calling, and optionally route the same endpoint through a LiteLLM proxy config.
- GPU clouds and serverless inference for agents (Comparison): Compare GPU rental, serverless GPUs and hosted per-token APIs for agents: OpenAI-compatible base URLs, tools and structured output, BYO models, scale-to-zero and billing, APIs/CLIs, and host-trust notes. No cached dollar figures or invented benchmarks.
- Fully local, private agent (Stack): A local OpenAI-compatible model server, an optional LiteLLM router, Pi as the harness, Basic Memory for on-disk Markdown memory, and Anthropic's sandbox runtime — all on your machine, with no hosted model API and no cloud sandbox.
- Probe an OpenAI-compatible endpoint (Skill): A read-only-by-default Python probe that lists models on an OpenAI-compatible /v1 endpoint, with opt-in chat and tool-call checks, loopback default, and keys from env only.
Sources
- https://vllm.ai/
- https://github.com/vllm-project/vllm
- https://github.com/vllm-project/vllm/blob/main/LICENSE
- https://github.com/vllm-project/vllm/blob/main/README.md
- https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3
- https://github.com/vllm-project/vllm/releases/tag/v0.33.0
- https://docs.vllm.ai/en/latest/
- https://docs.vllm.ai/en/latest/getting_started/installation/
- https://pypi.org/project/vllm/
- https://pypi.org/project/vllm/0.33.0/
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)