OpenRLHF
Apache-2.0 high-performance RLHF / post-training framework built on Ray, with support for multiple reward and policy-optimization recipes.
Tags
Description
Established open-source RLHF toolkit (OpenRLHF/OpenRLHF; not archived; not a 2026 launch). GitHub license Apache-2.0. PyPI openrlhf 0.11.2 verified 2026-11-11; docs https://openrlhf.readthedocs.io/en/latest/ verified 2026-11-12 (https://openrlhf.readthedocs.io/ redirects there). Training requires substantial GPU resources; model licenses and cluster costs are separate from the framework.
Agent access
Related
- TRL: Hugging Face Transformer Reinforcement Learning library for post-training methods such as SFT, DPO, PPO and GRPO on top of transformers.
- verl: Distributed reinforcement-learning training library for LLMs, with GRPO/PPO dataflows and pluggable training and rollout engines for agentic post-training.
- SkyRL: Modular reinforcement-learning stack with a training engine, long-horizon agent layer and Gymnasium-style tool-use environments for LLMs.
- rLLM: Reinforcement-learning framework that reuses agent harnesses for evaluation and training, collecting rollouts and rewards across sandbox backends.
- AReaL: Apache-2.0 fully asynchronous reinforcement-learning framework for large reasoning models, with a public Python package and docs.
Guides & comparisons
- Post-training a tool-use model — TRL, PEFT, and scale-out RL (Stack): A recommended post-training stack for chat and tool-use models: TRL and PEFT for SFT, LoRA/QLoRA, DPO and GRPO, with OpenRLHF and AReaL as the distributed and asynchronous RL options. Cross-links the fine-tuning frameworks and agent RL stacks comparisons.
Sources
- https://github.com/OpenRLHF/OpenRLHF
- https://github.com/OpenRLHF/OpenRLHF/blob/main/LICENSE
- https://pypi.org/project/openrlhf/
- https://openrlhf.readthedocs.io/en/latest/
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)