{
  "type": "stack",
  "id": "post-training-tool-use-model",
  "title": "Post-training a tool-use model — TRL, PEFT, and scale-out RL",
  "summary": "A recommended post-training stack for chat and tool-use models: TRL and PEFT for SFT, LoRA/QLoRA, DPO and GRPO, with OpenRLHF and AReaL as the distributed and asynchronous RL options. Cross-links the fine-tuning frameworks and agent RL stacks comparisons.",
  "author": "Agentica Author",
  "tags": [
    "fine-tuning",
    "training",
    "lora",
    "reinforcement-learning",
    "tool-calling",
    "python"
  ],
  "published": "2026-11-12",
  "updated": "2026-11-12",
  "last_verified": "2026-11-12",
  "entries": [
    "trl",
    "peft",
    "openrlhf",
    "areal",
    "axolotl",
    "llama-factory",
    "unsloth",
    "verl",
    "skyrl",
    "torchtune"
  ],
  "links": {
    "html": "https://indexagentica.com/stacks/post-training-tool-use-model/",
    "markdown": "https://indexagentica.com/stacks/post-training-tool-use-model.md",
    "json": "https://indexagentica.com/api/longform/stacks/post-training-tool-use-model.json",
    "source": "https://github.com/Drudley/indexagentica/blob/main/content-long/stacks/post-training-tool-use-model.md"
  },
  "status": "published",
  "stack": {
    "use_case": "Start with TRL and PEFT for supervised fine-tuning, LoRA or QLoRA, and library-level DPO or GRPO on a chat or tool-use model. Add OpenRLHF or AReaL when you need distributed or fully asynchronous agentic RL.",
    "components": [
      {
        "role": "Library trainers for SFT, DPO and GRPO",
        "entry": "trl",
        "name": "TRL",
        "url": "https://indexagentica.com/entries/trl/",
        "why": "Transformers Trainer wrappers. The README documents SFTTrainer, GRPOTrainer, DPOTrainer, KTOTrainer and RewardTrainer, plus a CLI (trl sft, trl dpo, trl kto). Trainers natively support Accelerate DDP, DeepSpeed ZeRO and FSDP, and accept a PEFT peft_config for LoRA/QLoRA. The SFT docs also describe a tool-calling dataset format."
      },
      {
        "role": "LoRA and QLoRA adapters",
        "entry": "peft",
        "name": "PEFT",
        "url": "https://indexagentica.com/entries/peft/",
        "why": "Adapter library used with Transformers and TRL. The official LoRA path is LoraConfig plus get_peft_model, or SFTTrainer(peft_config=LoraConfig()). TRL's PEFT integration docs show combining peft_config with a BitsAndBytes quantization_config for QLoRA."
      },
      {
        "role": "Ray-based distributed RLHF",
        "entry": "openrlhf",
        "name": "OpenRLHF",
        "url": "https://indexagentica.com/entries/openrlhf/",
        "why": "Scale-out RLHF option built on Ray, vLLM generation and DeepSpeed ZeRO-3, with a Hybrid Engine and Hugging Face Transformers models. Documents SingleTurnExecutor and MultiTurnExecutor, async training via --train.async_enable, and algorithms including PPO, REINFORCE++, GRPO (group_norm), RLOO, Dr. GRPO and FlashREINFORCE."
      },
      {
        "role": "Asynchronous agentic RL",
        "entry": "areal",
        "name": "AReaL",
        "url": "https://indexagentica.com/entries/areal/",
        "why": "Scale-out option for fully asynchronous RL, including documented agentic RL examples. The support matrix lists GRPO, GSPO, PPO, DAPO, LitePPO, Dr.GRPO, REINFORCE++, RLOO, SAPO, IcePop, KPop, M2PO, DPO, reward modelling, SFT and distillation; training backends Megatron, PyTorch FSDP and PyTorch Archon; inference backends vLLM and SGLang. Pin carefully: PyPI areal is 1.0.4; GitHub latest is v2.1.1."
      }
    ]
  },
  "entries_detail": [
    {
      "id": "trl",
      "name": "TRL",
      "summary": "Hugging Face Transformer Reinforcement Learning library for post-training methods such as SFT, DPO, PPO and GRPO on top of transformers.",
      "url": "https://indexagentica.com/entries/trl/",
      "json": "https://indexagentica.com/api/entries/trl.json"
    },
    {
      "id": "peft",
      "name": "PEFT",
      "summary": "Hugging Face Parameter-Efficient Fine-Tuning library (LoRA and related methods) for adapting large models without full fine-tuning.",
      "url": "https://indexagentica.com/entries/peft/",
      "json": "https://indexagentica.com/api/entries/peft.json"
    },
    {
      "id": "openrlhf",
      "name": "OpenRLHF",
      "summary": "Apache-2.0 high-performance RLHF / post-training framework built on Ray, with support for multiple reward and policy-optimization recipes.",
      "url": "https://indexagentica.com/entries/openrlhf/",
      "json": "https://indexagentica.com/api/entries/openrlhf.json"
    },
    {
      "id": "areal",
      "name": "AReaL",
      "summary": "Apache-2.0 fully asynchronous reinforcement-learning framework for large reasoning models, with a public Python package and docs.",
      "url": "https://indexagentica.com/entries/areal/",
      "json": "https://indexagentica.com/api/entries/areal.json"
    },
    {
      "id": "axolotl",
      "name": "Axolotl",
      "summary": "Apache-2.0 post-training / fine-tuning framework for a wide range of LLMs, with CLI, YAML configs, Docker and distributed-training options.",
      "url": "https://indexagentica.com/entries/axolotl/",
      "json": "https://indexagentica.com/api/entries/axolotl.json"
    },
    {
      "id": "llama-factory",
      "name": "LLaMA-Factory",
      "summary": "Apache-2.0 toolkit to fine-tune 100+ LLMs and VLMs via zero-code CLI and the LLaMA Board Gradio web UI.",
      "url": "https://indexagentica.com/entries/llama-factory/",
      "json": "https://indexagentica.com/api/entries/llama-factory.json"
    },
    {
      "id": "unsloth",
      "name": "Unsloth",
      "summary": "Open-source LLM fine-tuning library and desktop app for running and training models locally, with a dual Apache-2.0 / AGPL-3.0 license split.",
      "url": "https://indexagentica.com/entries/unsloth/",
      "json": "https://indexagentica.com/api/entries/unsloth.json"
    },
    {
      "id": "verl",
      "name": "verl",
      "summary": "Distributed reinforcement-learning training library for LLMs, with GRPO/PPO dataflows and pluggable training and rollout engines for agentic post-training.",
      "url": "https://indexagentica.com/entries/verl/",
      "json": "https://indexagentica.com/api/entries/verl.json"
    },
    {
      "id": "skyrl",
      "name": "SkyRL",
      "summary": "Modular reinforcement-learning stack with a training engine, long-horizon agent layer and Gymnasium-style tool-use environments for LLMs.",
      "url": "https://indexagentica.com/entries/skyrl/",
      "json": "https://indexagentica.com/api/entries/skyrl.json"
    },
    {
      "id": "torchtune",
      "name": "torchtune",
      "summary": "PyTorch-native BSD-3-Clause library for authoring, fine-tuning and evaluating LLMs with native PyTorch training primitives.",
      "url": "https://indexagentica.com/entries/torchtune/",
      "json": "https://indexagentica.com/api/entries/torchtune.json"
    }
  ],
  "related": [
    {
      "type": "comparison",
      "id": "fine-tuning-frameworks",
      "title": "Fine-tuning frameworks compared — Axolotl, LLaMA-Factory and Unsloth",
      "url": "https://indexagentica.com/compare/fine-tuning-frameworks/",
      "json": "https://indexagentica.com/api/longform/compare/fine-tuning-frameworks.json"
    },
    {
      "type": "comparison",
      "id": "agent-reinforcement-learning-stacks",
      "title": "Agent RL stacks compared — verl, SkyRL, Agent Lightning and ART",
      "url": "https://indexagentica.com/compare/agent-reinforcement-learning-stacks/",
      "json": "https://indexagentica.com/api/longform/compare/agent-reinforcement-learning-stacks.json"
    }
  ],
  "sources": [
    {
      "title": "TRL PEFT integration",
      "url": "https://huggingface.co/docs/trl/peft_integration",
      "accessed": "2026-11-12"
    },
    {
      "title": "huggingface/trl README",
      "url": "https://raw.githubusercontent.com/huggingface/trl/main/README.md",
      "accessed": "2026-11-12"
    },
    {
      "title": "huggingface/trl LICENSE",
      "url": "https://raw.githubusercontent.com/huggingface/trl/main/LICENSE",
      "accessed": "2026-11-12"
    },
    {
      "title": "TRL PyPI metadata",
      "url": "https://pypi.org/pypi/trl/json",
      "accessed": "2026-11-12"
    },
    {
      "title": "huggingface/trl latest GitHub release",
      "url": "https://api.github.com/repos/huggingface/trl/releases/latest",
      "accessed": "2026-11-12"
    },
    {
      "title": "TRL documentation index",
      "url": "https://huggingface.co/docs/trl/index",
      "accessed": "2026-11-12"
    },
    {
      "title": "TRL SFT Trainer documentation",
      "url": "https://huggingface.co/docs/trl/sft_trainer",
      "accessed": "2026-11-12"
    },
    {
      "title": "TRL dataset formats",
      "url": "https://huggingface.co/docs/trl/dataset_formats",
      "accessed": "2026-11-12"
    },
    {
      "title": "huggingface/peft README",
      "url": "https://raw.githubusercontent.com/huggingface/peft/main/README.md",
      "accessed": "2026-11-12"
    },
    {
      "title": "huggingface/peft LICENSE",
      "url": "https://raw.githubusercontent.com/huggingface/peft/main/LICENSE",
      "accessed": "2026-11-12"
    },
    {
      "title": "PEFT PyPI metadata",
      "url": "https://pypi.org/pypi/peft/json",
      "accessed": "2026-11-12"
    },
    {
      "title": "huggingface/peft latest GitHub release",
      "url": "https://api.github.com/repos/huggingface/peft/releases/latest",
      "accessed": "2026-11-12"
    },
    {
      "title": "PEFT documentation index",
      "url": "https://huggingface.co/docs/peft/index",
      "accessed": "2026-11-12"
    },
    {
      "title": "OpenRLHF/OpenRLHF README",
      "url": "https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/README.md",
      "accessed": "2026-11-12"
    },
    {
      "title": "OpenRLHF/OpenRLHF LICENSE",
      "url": "https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/LICENSE",
      "accessed": "2026-11-12"
    },
    {
      "title": "OpenRLHF PyPI metadata",
      "url": "https://pypi.org/pypi/openrlhf/json",
      "accessed": "2026-11-12"
    },
    {
      "title": "OpenRLHF/OpenRLHF latest GitHub release",
      "url": "https://api.github.com/repos/OpenRLHF/OpenRLHF/releases/latest",
      "accessed": "2026-11-12"
    },
    {
      "title": "OpenRLHF documentation",
      "url": "https://openrlhf.readthedocs.io/en/latest/",
      "accessed": "2026-11-12"
    },
    {
      "title": "areal-project/AReaL README",
      "url": "https://raw.githubusercontent.com/areal-project/AReaL/main/README.md",
      "accessed": "2026-11-12"
    },
    {
      "title": "areal-project/AReaL LICENSE",
      "url": "https://raw.githubusercontent.com/areal-project/AReaL/main/LICENSE",
      "accessed": "2026-11-12"
    },
    {
      "title": "AReaL PyPI metadata",
      "url": "https://pypi.org/pypi/areal/json",
      "accessed": "2026-11-12"
    },
    {
      "title": "areal-project/AReaL latest GitHub release",
      "url": "https://api.github.com/repos/areal-project/AReaL/releases/latest",
      "accessed": "2026-11-12"
    },
    {
      "title": "AReaL documentation, Introduction",
      "url": "https://areal-ai.io/docs/en/intro.html",
      "accessed": "2026-11-12"
    },
    {
      "title": "AReaL documentation, Agentic RL",
      "url": "https://areal-ai.io/docs/en/tutorial/agentic_rl.html",
      "accessed": "2026-11-12"
    },
    {
      "title": "AReaL documentation, Asynchronous RL",
      "url": "https://areal-ai.io/docs/en/algorithms/async.html",
      "accessed": "2026-11-12"
    },
    {
      "title": "meta-pytorch/torchtune README",
      "url": "https://raw.githubusercontent.com/meta-pytorch/torchtune/main/README.md",
      "accessed": "2026-11-12"
    },
    {
      "title": "The future of torchtune (GitHub issue #2883)",
      "url": "https://github.com/meta-pytorch/torchtune/issues/2883",
      "accessed": "2026-11-12"
    },
    {
      "title": "torchtune PyPI metadata",
      "url": "https://pypi.org/pypi/torchtune/json",
      "accessed": "2026-11-12"
    }
  ],
  "front_matter": {
    "id": "post-training-tool-use-model",
    "type": "stack",
    "title": "Post-training a tool-use model — TRL, PEFT, and scale-out RL",
    "summary": "A recommended post-training stack for chat and tool-use models: TRL and PEFT for SFT, LoRA/QLoRA, DPO and GRPO, with OpenRLHF and AReaL as the distributed and asynchronous RL options. Cross-links the fine-tuning frameworks and agent RL stacks comparisons.",
    "description": "Use Hugging Face TRL (PyPI 1.17.1, Apache-2.0) and PEFT (0.21.2, Apache-2.0) as the library layer for supervised fine-tuning, LoRA/QLoRA, DPO, KTO, reward modelling and GRPO, including TRL's documented tool-calling SFT dataset format. Scale out with OpenRLHF (0.11.2, Apache-2.0; Ray, vLLM, DeepSpeed ZeRO-3, agent executors, async RL) or AReaL (GitHub latest v2.1.1; PyPI package still 1.0.4 as of 12 November 2026; fully asynchronous and agentic RL, Megatron/FSDP/Archon, vLLM/SGLang). Mentions torchtune only as deprecated. Official TRL+PEFT SFT sample included; no training run and no unbacked performance numbers.",
    "author": "Agentica Author",
    "use_case": "Start with TRL and PEFT for supervised fine-tuning, LoRA or QLoRA, and library-level DPO or GRPO on a chat or tool-use model. Add OpenRLHF or AReaL when you need distributed or fully asynchronous agentic RL.",
    "components": [
      {
        "role": "Library trainers for SFT, DPO and GRPO",
        "entry": "trl",
        "why": "Transformers Trainer wrappers. The README documents SFTTrainer, GRPOTrainer, DPOTrainer, KTOTrainer and RewardTrainer, plus a CLI (trl sft, trl dpo, trl kto). Trainers natively support Accelerate DDP, DeepSpeed ZeRO and FSDP, and accept a PEFT peft_config for LoRA/QLoRA. The SFT docs also describe a tool-calling dataset format."
      },
      {
        "role": "LoRA and QLoRA adapters",
        "entry": "peft",
        "why": "Adapter library used with Transformers and TRL. The official LoRA path is LoraConfig plus get_peft_model, or SFTTrainer(peft_config=LoraConfig()). TRL's PEFT integration docs show combining peft_config with a BitsAndBytes quantization_config for QLoRA."
      },
      {
        "role": "Ray-based distributed RLHF",
        "entry": "openrlhf",
        "why": "Scale-out RLHF option built on Ray, vLLM generation and DeepSpeed ZeRO-3, with a Hybrid Engine and Hugging Face Transformers models. Documents SingleTurnExecutor and MultiTurnExecutor, async training via --train.async_enable, and algorithms including PPO, REINFORCE++, GRPO (group_norm), RLOO, Dr. GRPO and FlashREINFORCE."
      },
      {
        "role": "Asynchronous agentic RL",
        "entry": "areal",
        "why": "Scale-out option for fully asynchronous RL, including documented agentic RL examples. The support matrix lists GRPO, GSPO, PPO, DAPO, LitePPO, Dr.GRPO, REINFORCE++, RLOO, SAPO, IcePop, KPop, M2PO, DPO, reward modelling, SFT and distillation; training backends Megatron, PyTorch FSDP and PyTorch Archon; inference backends vLLM and SGLang. Pin carefully: PyPI areal is 1.0.4; GitHub latest is v2.1.1."
      }
    ],
    "tags": [
      "fine-tuning",
      "training",
      "lora",
      "reinforcement-learning",
      "tool-calling",
      "python"
    ],
    "entries": [
      "trl",
      "peft",
      "openrlhf",
      "areal"
    ],
    "related": [
      "fine-tuning-frameworks",
      "agent-reinforcement-learning-stacks"
    ],
    "sources": [
      {
        "title": "TRL PEFT integration",
        "url": "https://huggingface.co/docs/trl/peft_integration",
        "accessed": "2026-11-12"
      },
      {
        "title": "huggingface/trl README",
        "url": "https://raw.githubusercontent.com/huggingface/trl/main/README.md",
        "accessed": "2026-11-12"
      },
      {
        "title": "huggingface/trl LICENSE",
        "url": "https://raw.githubusercontent.com/huggingface/trl/main/LICENSE",
        "accessed": "2026-11-12"
      },
      {
        "title": "TRL PyPI metadata",
        "url": "https://pypi.org/pypi/trl/json",
        "accessed": "2026-11-12"
      },
      {
        "title": "huggingface/trl latest GitHub release",
        "url": "https://api.github.com/repos/huggingface/trl/releases/latest",
        "accessed": "2026-11-12"
      },
      {
        "title": "TRL documentation index",
        "url": "https://huggingface.co/docs/trl/index",
        "accessed": "2026-11-12"
      },
      {
        "title": "TRL SFT Trainer documentation",
        "url": "https://huggingface.co/docs/trl/sft_trainer",
        "accessed": "2026-11-12"
      },
      {
        "title": "TRL dataset formats",
        "url": "https://huggingface.co/docs/trl/dataset_formats",
        "accessed": "2026-11-12"
      },
      {
        "title": "huggingface/peft README",
        "url": "https://raw.githubusercontent.com/huggingface/peft/main/README.md",
        "accessed": "2026-11-12"
      },
      {
        "title": "huggingface/peft LICENSE",
        "url": "https://raw.githubusercontent.com/huggingface/peft/main/LICENSE",
        "accessed": "2026-11-12"
      },
      {
        "title": "PEFT PyPI metadata",
        "url": "https://pypi.org/pypi/peft/json",
        "accessed": "2026-11-12"
      },
      {
        "title": "huggingface/peft latest GitHub release",
        "url": "https://api.github.com/repos/huggingface/peft/releases/latest",
        "accessed": "2026-11-12"
      },
      {
        "title": "PEFT documentation index",
        "url": "https://huggingface.co/docs/peft/index",
        "accessed": "2026-11-12"
      },
      {
        "title": "OpenRLHF/OpenRLHF README",
        "url": "https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/README.md",
        "accessed": "2026-11-12"
      },
      {
        "title": "OpenRLHF/OpenRLHF LICENSE",
        "url": "https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/LICENSE",
        "accessed": "2026-11-12"
      },
      {
        "title": "OpenRLHF PyPI metadata",
        "url": "https://pypi.org/pypi/openrlhf/json",
        "accessed": "2026-11-12"
      },
      {
        "title": "OpenRLHF/OpenRLHF latest GitHub release",
        "url": "https://api.github.com/repos/OpenRLHF/OpenRLHF/releases/latest",
        "accessed": "2026-11-12"
      },
      {
        "title": "OpenRLHF documentation",
        "url": "https://openrlhf.readthedocs.io/en/latest/",
        "accessed": "2026-11-12"
      },
      {
        "title": "areal-project/AReaL README",
        "url": "https://raw.githubusercontent.com/areal-project/AReaL/main/README.md",
        "accessed": "2026-11-12"
      },
      {
        "title": "areal-project/AReaL LICENSE",
        "url": "https://raw.githubusercontent.com/areal-project/AReaL/main/LICENSE",
        "accessed": "2026-11-12"
      },
      {
        "title": "AReaL PyPI metadata",
        "url": "https://pypi.org/pypi/areal/json",
        "accessed": "2026-11-12"
      },
      {
        "title": "areal-project/AReaL latest GitHub release",
        "url": "https://api.github.com/repos/areal-project/AReaL/releases/latest",
        "accessed": "2026-11-12"
      },
      {
        "title": "AReaL documentation, Introduction",
        "url": "https://areal-ai.io/docs/en/intro.html",
        "accessed": "2026-11-12"
      },
      {
        "title": "AReaL documentation, Agentic RL",
        "url": "https://areal-ai.io/docs/en/tutorial/agentic_rl.html",
        "accessed": "2026-11-12"
      },
      {
        "title": "AReaL documentation, Asynchronous RL",
        "url": "https://areal-ai.io/docs/en/algorithms/async.html",
        "accessed": "2026-11-12"
      },
      {
        "title": "meta-pytorch/torchtune README",
        "url": "https://raw.githubusercontent.com/meta-pytorch/torchtune/main/README.md",
        "accessed": "2026-11-12"
      },
      {
        "title": "The future of torchtune (GitHub issue #2883)",
        "url": "https://github.com/meta-pytorch/torchtune/issues/2883",
        "accessed": "2026-11-12"
      },
      {
        "title": "torchtune PyPI metadata",
        "url": "https://pypi.org/pypi/torchtune/json",
        "accessed": "2026-11-12"
      }
    ],
    "last_verified": "2026-11-12",
    "published": "2026-11-12",
    "updated": "2026-11-12"
  },
  "markdown": "## Start at the library layer, then scale out\n\nWhen you need to post-train a chat or tool-use model, start with [TRL](https://indexagentica.com/entries/trl/) and [PEFT](https://indexagentica.com/entries/peft/). TRL's trainers are documented as light wrappers around the Hugging Face Transformers Trainer. The project README lists [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer), [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer), [`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer), [`KTOTrainer`](https://huggingface.co/docs/trl/kto_trainer) and [`RewardTrainer`](https://huggingface.co/docs/trl/reward_trainer), plus a CLI (`trl sft`, `trl dpo`, `trl kto`). The trainers natively support distributed training with Accelerate DDP, DeepSpeed ZeRO and FSDP, and the README states full integration with PEFT for quantization and LoRA/QLoRA. That TRL+PEFT combination is the recommended core of this stack.\n\nYAML- and UI-oriented SFT/LoRA frameworks ([Axolotl](https://indexagentica.com/entries/axolotl/), [LLaMA-Factory](https://indexagentica.com/entries/llama-factory/), [Unsloth](https://indexagentica.com/entries/unsloth/)) are compared separately in [Fine-tuning frameworks compared](https://indexagentica.com/compare/fine-tuning-frameworks/). Distributed agent RL engines such as [verl](https://indexagentica.com/entries/verl/) and [SkyRL](https://indexagentica.com/entries/skyrl/) are compared in [Agent RL stacks compared](https://indexagentica.com/compare/agent-reinforcement-learning-stacks/). Use those pages to choose a framework family. Use this stack when you want the Hugging Face trainer layer first, then a documented path to scale-out or asynchronous agentic RL.\n\n[OpenRLHF](https://indexagentica.com/entries/openrlhf/) and [AReaL](https://indexagentica.com/entries/areal/) are the scale-out options in this stack. They are alternatives, not a requirement to run both at once.\n\n## What each project documents\n\n**TRL** (PyPI and GitHub latest **1.17.1**, Apache-2.0, Python >=3.11; docs at [huggingface.co/docs/trl/index](https://huggingface.co/docs/trl/index)) is the trainer library. Besides the trainers above, the [SFT trainer documentation](https://huggingface.co/docs/trl/sft_trainer) has a section on training adapters with PEFT (`peft_config=LoraConfig()`) and notes that you can pass an already-loaded `PeftModel` without `peft_config`. The [PEFT integration page](https://huggingface.co/docs/trl/peft_integration) documents QLoRA by combining `peft_config` with `quantization_config` (`BitsAndBytesConfig`). A separate \"Tool Calling with SFT\" section, together with the [dataset formats](https://huggingface.co/docs/trl/dataset_formats) page, specifies that each example should include conversation messages (with `tool_calls` and `tool` role messages) and a `tools` column of JSON schemas. The README also mentions optional Unsloth kernels; if you want Unsloth as a user-facing fine-tuning product, use the frameworks comparison rather than this stack.\n\n**PEFT** (PyPI and GitHub latest **0.21.2**, Apache-2.0, Python >=3.10; docs at [huggingface.co/docs/peft/index](https://huggingface.co/docs/peft/index)) is the adapter library. The README quickstart wraps a causal LM with `LoraConfig` (`r`, `lora_alpha`, `task_type=TaskType.CAUSAL_LM`) and `get_peft_model`. In this stack you typically let TRL apply that config through `SFTTrainer` rather than wrapping the model yourself, unless you are continuing an existing `PeftModel`.\n\n**OpenRLHF** (PyPI and GitHub latest **0.11.2**, Apache-2.0 in the repository `LICENSE`, Python >=3.10; docs at [openrlhf.readthedocs.io/en/latest/](https://openrlhf.readthedocs.io/en/latest/)) is the Ray-based scale-out RLHF option. The README describes a Ray scheduler that can place Actor, Reward, Reference and Critic models on separate GPUs, a Hybrid Engine, generation with vLLM, and training with DeepSpeed ZeRO-3, deepcompile, AutoTP and RingAttention, on Hugging Face Transformers models. Agent execution uses `AgentExecutorBase` with `SingleTurnExecutor` and `MultiTurnExecutor`; the README says algorithms are decoupled from execution mode. Async RLHF is documented via `--train.async_enable`, and custom or multi-turn agents via `--train.agent_func_path`. The algorithm table lists **PPO** (default), **REINFORCE++** (`reinforce`), **RLOO** (`rloo`), **GRPO** (`group_norm`), **Dr. GRPO** (`dr_grpo`) and **FlashREINFORCE** (`flash_reinforce`); the surrounding text also names **REINFORCE++-baseline**. The project additionally documents SFT, DPO/IPO/cDPO and reward-model scripts. Project wording such as \"the first RLHF framework\" and claims about generation-time share or model scale are vendor statements and are not used as facts here.\n\n**AReaL** is the asynchronous / agentic RL option. The project README presents it as a large-scale asynchronous reinforcement learning system, \"built on a fully asynchronous RL training paradigm,\" with a documented [agentic RL](https://areal-ai.io/docs/en/tutorial/agentic_rl.html) tutorial and an [asynchronous RL](https://areal-ai.io/docs/en/algorithms/async.html) guide. The README says all listed RL algorithms support asynchronous and synchronous versions by setting `max_head_offpolicyness=0`. The **support matrix** (checked on the `main` README on 12 November 2026) lists algorithms **GRPO, GSPO, PPO, DAPO, LitePPO, Dr.GRPO, REINFORCE++, RLOO, SAPO, IcePop, KPop, M2PO, DPO**, plus reward modelling, SFT and distillation; training backends **Megatron**, **PyTorch FSDP** and **PyTorch Archon**; and inference backends **vLLM** and **SGLang**. Documented agentic examples include a general agent workflow, tool-integrated reasoning, a search agent, coding-agent RL, and integrations with agent frameworks. Official docs live at [areal-ai.io/docs/en/intro.html](https://areal-ai.io/docs/en/intro.html). Do not use the older `areal-project.github.io` documentation URLs, which redirect, or treat the stale documentation URL still listed on PyPI as the current docs location.\n\nPin AReaL carefully: the latest **GitHub release** on 12 November 2026 is **v2.1.1** (published 4 November 2026). The latest **PyPI** package `areal` is still **1.0.4** (uploaded 10 June 2026, Apache-2.0, Python >=3.11,<3.13). That wheel predates the README's AReaL 2.0 notes (1 July 2026) and there is no 2.x series on PyPI. Install from the project's current docs, not from an assumption that `pip install areal` matches `main`.\n\n## A minimal SFT + LoRA example\n\nThis sample is copied from the official [SFT trainer](https://huggingface.co/docs/trl/sft_trainer) section \"Train adapters with PEFT.\" It is the documented way to combine TRL and PEFT for LoRA SFT:\n\n```python\nfrom datasets import load_dataset\nfrom trl import SFTTrainer\nfrom peft import LoraConfig\n\ndataset = load_dataset(\"trl-lib/Capybara\", split=\"train\")\n\ntrainer = SFTTrainer(\n    \"Qwen/Qwen3-0.6B\",\n    train_dataset=dataset,\n    peft_config=LoraConfig(),\n)\n\ntrainer.train()\n```\n\nThe same page documents passing a trainable `PeftModel` instead of `peft_config` when you continue an existing adapter. For QLoRA, the [PEFT integration page](https://huggingface.co/docs/trl/peft_integration) shows combining `peft_config` with `quantization_config`. For tool-use SFT, keep this trainer wiring and follow the tool-calling dataset format on the [dataset formats](https://huggingface.co/docs/trl/dataset_formats) page rather than inventing a schema. The CLI equivalent for plain SFT (without the PEFT argument shown above) is documented in the TRL README as `trl sft`.\n\n## When to add OpenRLHF or AReaL\n\nStay on TRL+PEFT while you are doing single-process or Accelerate-managed SFT, LoRA, DPO or GRPO and the official trainers cover the method. Move to **OpenRLHF** when you want that library's Ray + vLLM + DeepSpeed layout, its single-turn or multi-turn agent executors, or its async flag. Move to **AReaL** when you want the project's fully asynchronous training loop and the agentic RL examples and backends in its support matrix (including Megatron or SGLang, which OpenRLHF's README does not present as first-class in the same way). For other agent RL engines, start from [Agent RL stacks compared](https://indexagentica.com/compare/agent-reinforcement-learning-stacks/) instead of forcing them into this stack.\n\n## Deprecated: do not start with torchtune\n\n[torchtune](https://indexagentica.com/entries/torchtune/) is listed here only so it is not mistaken for a current recommendation. Its README states that it is no longer actively maintained and that development wound down in 2025, pointing at GitHub issue [#2883](https://github.com/meta-pytorch/torchtune/issues/2883) (\"The future of torchtune\"), which was still open on 12 November 2026. PyPI remains at **0.6.1** (BSD-3-Clause, uploaded 7 April 2025). The repository is not archived. Do not start a new post-training stack with it.\n\n## Licences and versions (12 November 2026)\n\nChecked on PyPI, GitHub latest releases, and each repository's licence file:\n\n| Package | Licence file | PyPI / GitHub latest | Requires Python |\n|---|---|---|---|\n| trl | Apache License 2.0 (`LICENSE`) | 1.17.1 / v1.17.1 | >=3.11 |\n| peft | Apache License 2.0 (`LICENSE`) | 0.21.2 / v0.21.2 | >=3.10 |\n| openrlhf | Apache License 2.0 (`LICENSE`) | 0.11.2 / v0.11.2 | >=3.10 |\n| areal | Apache License 2.0 (`LICENSE`) | PyPI 1.0.4; GitHub latest v2.1.1 | PyPI 1.0.4: >=3.11,<3.13 |\n\nOpenRLHF's PyPI metadata does not set a licence field; the Apache-2.0 text is from the repo `LICENSE`. Model weights, GPUs, and any hosted inference remain separate from these software licences.\n\n## What we ran, and what we did not\n\nWe did **not** install TRL, PEFT, PyTorch, OpenRLHF or AReaL, and we did not run the official SFT+LoRA sample. That sample loads `Qwen/Qwen3-0.6B` and the `trl-lib/Capybara` training split; a useful CPU-only smoke test was not feasible here because PyTorch was not already present, installing it is a large download, and the documented example is not a tiny CPU model. No fine-tune was run, and no speed, memory or accuracy figure from project READMEs or blogs was reproduced.\n",
  "raw": "---\nid: post-training-tool-use-model\ntype: stack\ntitle: \"Post-training a tool-use model — TRL, PEFT, and scale-out RL\"\nsummary: \"A recommended post-training stack for chat and tool-use models: TRL and PEFT for SFT, LoRA/QLoRA, DPO and GRPO, with OpenRLHF and AReaL as the distributed and asynchronous RL options. Cross-links the fine-tuning frameworks and agent RL stacks comparisons.\"\ndescription: \"Use Hugging Face TRL (PyPI 1.17.1, Apache-2.0) and PEFT (0.21.2, Apache-2.0) as the library layer for supervised fine-tuning, LoRA/QLoRA, DPO, KTO, reward modelling and GRPO, including TRL's documented tool-calling SFT dataset format. Scale out with OpenRLHF (0.11.2, Apache-2.0; Ray, vLLM, DeepSpeed ZeRO-3, agent executors, async RL) or AReaL (GitHub latest v2.1.1; PyPI package still 1.0.4 as of 12 November 2026; fully asynchronous and agentic RL, Megatron/FSDP/Archon, vLLM/SGLang). Mentions torchtune only as deprecated. Official TRL+PEFT SFT sample included; no training run and no unbacked performance numbers.\"\nauthor: Agentica Author\nuse_case: \"Start with TRL and PEFT for supervised fine-tuning, LoRA or QLoRA, and library-level DPO or GRPO on a chat or tool-use model. Add OpenRLHF or AReaL when you need distributed or fully asynchronous agentic RL.\"\ncomponents:\n  - role: Library trainers for SFT, DPO and GRPO\n    entry: trl\n    why: \"Transformers Trainer wrappers. The README documents SFTTrainer, GRPOTrainer, DPOTrainer, KTOTrainer and RewardTrainer, plus a CLI (trl sft, trl dpo, trl kto). Trainers natively support Accelerate DDP, DeepSpeed ZeRO and FSDP, and accept a PEFT peft_config for LoRA/QLoRA. The SFT docs also describe a tool-calling dataset format.\"\n  - role: LoRA and QLoRA adapters\n    entry: peft\n    why: \"Adapter library used with Transformers and TRL. The official LoRA path is LoraConfig plus get_peft_model, or SFTTrainer(peft_config=LoraConfig()). TRL's PEFT integration docs show combining peft_config with a BitsAndBytes quantization_config for QLoRA.\"\n  - role: Ray-based distributed RLHF\n    entry: openrlhf\n    why: \"Scale-out RLHF option built on Ray, vLLM generation and DeepSpeed ZeRO-3, with a Hybrid Engine and Hugging Face Transformers models. Documents SingleTurnExecutor and MultiTurnExecutor, async training via --train.async_enable, and algorithms including PPO, REINFORCE++, GRPO (group_norm), RLOO, Dr. GRPO and FlashREINFORCE.\"\n  - role: Asynchronous agentic RL\n    entry: areal\n    why: \"Scale-out option for fully asynchronous RL, including documented agentic RL examples. The support matrix lists GRPO, GSPO, PPO, DAPO, LitePPO, Dr.GRPO, REINFORCE++, RLOO, SAPO, IcePop, KPop, M2PO, DPO, reward modelling, SFT and distillation; training backends Megatron, PyTorch FSDP and PyTorch Archon; inference backends vLLM and SGLang. Pin carefully: PyPI areal is 1.0.4; GitHub latest is v2.1.1.\"\ntags: [fine-tuning, training, lora, reinforcement-learning, tool-calling, python]\nentries: [trl, peft, openrlhf, areal]\nrelated: [fine-tuning-frameworks, agent-reinforcement-learning-stacks]\nsources:\n  - title: \"TRL PEFT integration\"\n    url: https://huggingface.co/docs/trl/peft_integration\n    accessed: 2026-11-12\n  - title: \"huggingface/trl README\"\n    url: https://raw.githubusercontent.com/huggingface/trl/main/README.md\n    accessed: 2026-11-12\n  - title: \"huggingface/trl LICENSE\"\n    url: https://raw.githubusercontent.com/huggingface/trl/main/LICENSE\n    accessed: 2026-11-12\n  - title: \"TRL PyPI metadata\"\n    url: https://pypi.org/pypi/trl/json\n    accessed: 2026-11-12\n  - title: \"huggingface/trl latest GitHub release\"\n    url: https://api.github.com/repos/huggingface/trl/releases/latest\n    accessed: 2026-11-12\n  - title: \"TRL documentation index\"\n    url: https://huggingface.co/docs/trl/index\n    accessed: 2026-11-12\n  - title: \"TRL SFT Trainer documentation\"\n    url: https://huggingface.co/docs/trl/sft_trainer\n    accessed: 2026-11-12\n  - title: \"TRL dataset formats\"\n    url: https://huggingface.co/docs/trl/dataset_formats\n    accessed: 2026-11-12\n  - title: \"huggingface/peft README\"\n    url: https://raw.githubusercontent.com/huggingface/peft/main/README.md\n    accessed: 2026-11-12\n  - title: \"huggingface/peft LICENSE\"\n    url: https://raw.githubusercontent.com/huggingface/peft/main/LICENSE\n    accessed: 2026-11-12\n  - title: \"PEFT PyPI metadata\"\n    url: https://pypi.org/pypi/peft/json\n    accessed: 2026-11-12\n  - title: \"huggingface/peft latest GitHub release\"\n    url: https://api.github.com/repos/huggingface/peft/releases/latest\n    accessed: 2026-11-12\n  - title: \"PEFT documentation index\"\n    url: https://huggingface.co/docs/peft/index\n    accessed: 2026-11-12\n  - title: \"OpenRLHF/OpenRLHF README\"\n    url: https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/README.md\n    accessed: 2026-11-12\n  - title: \"OpenRLHF/OpenRLHF LICENSE\"\n    url: https://raw.githubusercontent.com/OpenRLHF/OpenRLHF/main/LICENSE\n    accessed: 2026-11-12\n  - title: \"OpenRLHF PyPI metadata\"\n    url: https://pypi.org/pypi/openrlhf/json\n    accessed: 2026-11-12\n  - title: \"OpenRLHF/OpenRLHF latest GitHub release\"\n    url: https://api.github.com/repos/OpenRLHF/OpenRLHF/releases/latest\n    accessed: 2026-11-12\n  - title: \"OpenRLHF documentation\"\n    url: https://openrlhf.readthedocs.io/en/latest/\n    accessed: 2026-11-12\n  - title: \"areal-project/AReaL README\"\n    url: https://raw.githubusercontent.com/areal-project/AReaL/main/README.md\n    accessed: 2026-11-12\n  - title: \"areal-project/AReaL LICENSE\"\n    url: https://raw.githubusercontent.com/areal-project/AReaL/main/LICENSE\n    accessed: 2026-11-12\n  - title: \"AReaL PyPI metadata\"\n    url: https://pypi.org/pypi/areal/json\n    accessed: 2026-11-12\n  - title: \"areal-project/AReaL latest GitHub release\"\n    url: https://api.github.com/repos/areal-project/AReaL/releases/latest\n    accessed: 2026-11-12\n  - title: \"AReaL documentation, Introduction\"\n    url: https://areal-ai.io/docs/en/intro.html\n    accessed: 2026-11-12\n  - title: \"AReaL documentation, Agentic RL\"\n    url: https://areal-ai.io/docs/en/tutorial/agentic_rl.html\n    accessed: 2026-11-12\n  - title: \"AReaL documentation, Asynchronous RL\"\n    url: https://areal-ai.io/docs/en/algorithms/async.html\n    accessed: 2026-11-12\n  - title: \"meta-pytorch/torchtune README\"\n    url: https://raw.githubusercontent.com/meta-pytorch/torchtune/main/README.md\n    accessed: 2026-11-12\n  - title: \"The future of torchtune (GitHub issue #2883)\"\n    url: https://github.com/meta-pytorch/torchtune/issues/2883\n    accessed: 2026-11-12\n  - title: \"torchtune PyPI metadata\"\n    url: https://pypi.org/pypi/torchtune/json\n    accessed: 2026-11-12\nlast_verified: 2026-11-12\npublished: 2026-11-12\nupdated: 2026-11-12\n---\n## Start at the library layer, then scale out\n\nWhen you need to post-train a chat or tool-use model, start with [TRL](https://indexagentica.com/entries/trl/) and [PEFT](https://indexagentica.com/entries/peft/). TRL's trainers are documented as light wrappers around the Hugging Face Transformers Trainer. The project README lists [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer), [`GRPOTrainer`](https://huggingface.co/docs/trl/grpo_trainer), [`DPOTrainer`](https://huggingface.co/docs/trl/dpo_trainer), [`KTOTrainer`](https://huggingface.co/docs/trl/kto_trainer) and [`RewardTrainer`](https://huggingface.co/docs/trl/reward_trainer), plus a CLI (`trl sft`, `trl dpo`, `trl kto`). The trainers natively support distributed training with Accelerate DDP, DeepSpeed ZeRO and FSDP, and the README states full integration with PEFT for quantization and LoRA/QLoRA. That TRL+PEFT combination is the recommended core of this stack.\n\nYAML- and UI-oriented SFT/LoRA frameworks ([Axolotl](https://indexagentica.com/entries/axolotl/), [LLaMA-Factory](https://indexagentica.com/entries/llama-factory/), [Unsloth](https://indexagentica.com/entries/unsloth/)) are compared separately in [Fine-tuning frameworks compared](https://indexagentica.com/compare/fine-tuning-frameworks/). Distributed agent RL engines such as [verl](https://indexagentica.com/entries/verl/) and [SkyRL](https://indexagentica.com/entries/skyrl/) are compared in [Agent RL stacks compared](https://indexagentica.com/compare/agent-reinforcement-learning-stacks/). Use those pages to choose a framework family. Use this stack when you want the Hugging Face trainer layer first, then a documented path to scale-out or asynchronous agentic RL.\n\n[OpenRLHF](https://indexagentica.com/entries/openrlhf/) and [AReaL](https://indexagentica.com/entries/areal/) are the scale-out options in this stack. They are alternatives, not a requirement to run both at once.\n\n## What each project documents\n\n**TRL** (PyPI and GitHub latest **1.17.1**, Apache-2.0, Python >=3.11; docs at [huggingface.co/docs/trl/index](https://huggingface.co/docs/trl/index)) is the trainer library. Besides the trainers above, the [SFT trainer documentation](https://huggingface.co/docs/trl/sft_trainer) has a section on training adapters with PEFT (`peft_config=LoraConfig()`) and notes that you can pass an already-loaded `PeftModel` without `peft_config`. The [PEFT integration page](https://huggingface.co/docs/trl/peft_integration) documents QLoRA by combining `peft_config` with `quantization_config` (`BitsAndBytesConfig`). A separate \"Tool Calling with SFT\" section, together with the [dataset formats](https://huggingface.co/docs/trl/dataset_formats) page, specifies that each example should include conversation messages (with `tool_calls` and `tool` role messages) and a `tools` column of JSON schemas. The README also mentions optional Unsloth kernels; if you want Unsloth as a user-facing fine-tuning product, use the frameworks comparison rather than this stack.\n\n**PEFT** (PyPI and GitHub latest **0.21.2**, Apache-2.0, Python >=3.10; docs at [huggingface.co/docs/peft/index](https://huggingface.co/docs/peft/index)) is the adapter library. The README quickstart wraps a causal LM with `LoraConfig` (`r`, `lora_alpha`, `task_type=TaskType.CAUSAL_LM`) and `get_peft_model`. In this stack you typically let TRL apply that config through `SFTTrainer` rather than wrapping the model yourself, unless you are continuing an existing `PeftModel`.\n\n**OpenRLHF** (PyPI and GitHub latest **0.11.2**, Apache-2.0 in the repository `LICENSE`, Python >=3.10; docs at [openrlhf.readthedocs.io/en/latest/](https://openrlhf.readthedocs.io/en/latest/)) is the Ray-based scale-out RLHF option. The README describes a Ray scheduler that can place Actor, Reward, Reference and Critic models on separate GPUs, a Hybrid Engine, generation with vLLM, and training with DeepSpeed ZeRO-3, deepcompile, AutoTP and RingAttention, on Hugging Face Transformers models. Agent execution uses `AgentExecutorBase` with `SingleTurnExecutor` and `MultiTurnExecutor`; the README says algorithms are decoupled from execution mode. Async RLHF is documented via `--train.async_enable`, and custom or multi-turn agents via `--train.agent_func_path`. The algorithm table lists **PPO** (default), **REINFORCE++** (`reinforce`), **RLOO** (`rloo`), **GRPO** (`group_norm`), **Dr. GRPO** (`dr_grpo`) and **FlashREINFORCE** (`flash_reinforce`); the surrounding text also names **REINFORCE++-baseline**. The project additionally documents SFT, DPO/IPO/cDPO and reward-model scripts. Project wording such as \"the first RLHF framework\" and claims about generation-time share or model scale are vendor statements and are not used as facts here.\n\n**AReaL** is the asynchronous / agentic RL option. The project README presents it as a large-scale asynchronous reinforcement learning system, \"built on a fully asynchronous RL training paradigm,\" with a documented [agentic RL](https://areal-ai.io/docs/en/tutorial/agentic_rl.html) tutorial and an [asynchronous RL](https://areal-ai.io/docs/en/algorithms/async.html) guide. The README says all listed RL algorithms support asynchronous and synchronous versions by setting `max_head_offpolicyness=0`. The **support matrix** (checked on the `main` README on 12 November 2026) lists algorithms **GRPO, GSPO, PPO, DAPO, LitePPO, Dr.GRPO, REINFORCE++, RLOO, SAPO, IcePop, KPop, M2PO, DPO**, plus reward modelling, SFT and distillation; training backends **Megatron**, **PyTorch FSDP** and **PyTorch Archon**; and inference backends **vLLM** and **SGLang**. Documented agentic examples include a general agent workflow, tool-integrated reasoning, a search agent, coding-agent RL, and integrations with agent frameworks. Official docs live at [areal-ai.io/docs/en/intro.html](https://areal-ai.io/docs/en/intro.html). Do not use the older `areal-project.github.io` documentation URLs, which redirect, or treat the stale documentation URL still listed on PyPI as the current docs location.\n\nPin AReaL carefully: the latest **GitHub release** on 12 November 2026 is **v2.1.1** (published 4 November 2026). The latest **PyPI** package `areal` is still **1.0.4** (uploaded 10 June 2026, Apache-2.0, Python >=3.11,<3.13). That wheel predates the README's AReaL 2.0 notes (1 July 2026) and there is no 2.x series on PyPI. Install from the project's current docs, not from an assumption that `pip install areal` matches `main`.\n\n## A minimal SFT + LoRA example\n\nThis sample is copied from the official [SFT trainer](https://huggingface.co/docs/trl/sft_trainer) section \"Train adapters with PEFT.\" It is the documented way to combine TRL and PEFT for LoRA SFT:\n\n```python\nfrom datasets import load_dataset\nfrom trl import SFTTrainer\nfrom peft import LoraConfig\n\ndataset = load_dataset(\"trl-lib/Capybara\", split=\"train\")\n\ntrainer = SFTTrainer(\n    \"Qwen/Qwen3-0.6B\",\n    train_dataset=dataset,\n    peft_config=LoraConfig(),\n)\n\ntrainer.train()\n```\n\nThe same page documents passing a trainable `PeftModel` instead of `peft_config` when you continue an existing adapter. For QLoRA, the [PEFT integration page](https://huggingface.co/docs/trl/peft_integration) shows combining `peft_config` with `quantization_config`. For tool-use SFT, keep this trainer wiring and follow the tool-calling dataset format on the [dataset formats](https://huggingface.co/docs/trl/dataset_formats) page rather than inventing a schema. The CLI equivalent for plain SFT (without the PEFT argument shown above) is documented in the TRL README as `trl sft`.\n\n## When to add OpenRLHF or AReaL\n\nStay on TRL+PEFT while you are doing single-process or Accelerate-managed SFT, LoRA, DPO or GRPO and the official trainers cover the method. Move to **OpenRLHF** when you want that library's Ray + vLLM + DeepSpeed layout, its single-turn or multi-turn agent executors, or its async flag. Move to **AReaL** when you want the project's fully asynchronous training loop and the agentic RL examples and backends in its support matrix (including Megatron or SGLang, which OpenRLHF's README does not present as first-class in the same way). For other agent RL engines, start from [Agent RL stacks compared](https://indexagentica.com/compare/agent-reinforcement-learning-stacks/) instead of forcing them into this stack.\n\n## Deprecated: do not start with torchtune\n\n[torchtune](https://indexagentica.com/entries/torchtune/) is listed here only so it is not mistaken for a current recommendation. Its README states that it is no longer actively maintained and that development wound down in 2025, pointing at GitHub issue [#2883](https://github.com/meta-pytorch/torchtune/issues/2883) (\"The future of torchtune\"), which was still open on 12 November 2026. PyPI remains at **0.6.1** (BSD-3-Clause, uploaded 7 April 2025). The repository is not archived. Do not start a new post-training stack with it.\n\n## Licences and versions (12 November 2026)\n\nChecked on PyPI, GitHub latest releases, and each repository's licence file:\n\n| Package | Licence file | PyPI / GitHub latest | Requires Python |\n|---|---|---|---|\n| trl | Apache License 2.0 (`LICENSE`) | 1.17.1 / v1.17.1 | >=3.11 |\n| peft | Apache License 2.0 (`LICENSE`) | 0.21.2 / v0.21.2 | >=3.10 |\n| openrlhf | Apache License 2.0 (`LICENSE`) | 0.11.2 / v0.11.2 | >=3.10 |\n| areal | Apache License 2.0 (`LICENSE`) | PyPI 1.0.4; GitHub latest v2.1.1 | PyPI 1.0.4: >=3.11,<3.13 |\n\nOpenRLHF's PyPI metadata does not set a licence field; the Apache-2.0 text is from the repo `LICENSE`. Model weights, GPUs, and any hosted inference remain separate from these software licences.\n\n## What we ran, and what we did not\n\nWe did **not** install TRL, PEFT, PyTorch, OpenRLHF or AReaL, and we did not run the official SFT+LoRA sample. That sample loads `Qwen/Qwen3-0.6B` and the `trl-lib/Capybara` training split; a useful CPU-only smoke test was not feasible here because PyTorch was not already present, installing it is a large download, and the documented example is not a tiny CPU model. No fine-tune was run, and no speed, memory or accuracy figure from project READMEs or blogs was reproduced.\n",
  "html": "<h2>Start at the library layer, then scale out</h2>\n<p>When you need to post-train a chat or tool-use model, start with <a href=\"/entries/trl/\">TRL</a> and <a href=\"/entries/peft/\">PEFT</a>. TRL&#39;s trainers are documented as light wrappers around the Hugging Face Transformers Trainer. The project README lists <a href=\"https://huggingface.co/docs/trl/sft_trainer\"><code>SFTTrainer</code></a>, <a href=\"https://huggingface.co/docs/trl/grpo_trainer\"><code>GRPOTrainer</code></a>, <a href=\"https://huggingface.co/docs/trl/dpo_trainer\"><code>DPOTrainer</code></a>, <a href=\"https://huggingface.co/docs/trl/kto_trainer\"><code>KTOTrainer</code></a> and <a href=\"https://huggingface.co/docs/trl/reward_trainer\"><code>RewardTrainer</code></a>, plus a CLI (<code>trl sft</code>, <code>trl dpo</code>, <code>trl kto</code>). The trainers natively support distributed training with Accelerate DDP, DeepSpeed ZeRO and FSDP, and the README states full integration with PEFT for quantization and LoRA/QLoRA. That TRL+PEFT combination is the recommended core of this stack.</p>\n<p>YAML- and UI-oriented SFT/LoRA frameworks (<a href=\"/entries/axolotl/\">Axolotl</a>, <a href=\"/entries/llama-factory/\">LLaMA-Factory</a>, <a href=\"/entries/unsloth/\">Unsloth</a>) are compared separately in <a href=\"/compare/fine-tuning-frameworks/\">Fine-tuning frameworks compared</a>. Distributed agent RL engines such as <a href=\"/entries/verl/\">verl</a> and <a href=\"/entries/skyrl/\">SkyRL</a> are compared in <a href=\"/compare/agent-reinforcement-learning-stacks/\">Agent RL stacks compared</a>. Use those pages to choose a framework family. Use this stack when you want the Hugging Face trainer layer first, then a documented path to scale-out or asynchronous agentic RL.</p>\n<p><a href=\"/entries/openrlhf/\">OpenRLHF</a> and <a href=\"/entries/areal/\">AReaL</a> are the scale-out options in this stack. They are alternatives, not a requirement to run both at once.</p>\n<h2>What each project documents</h2>\n<p><strong>TRL</strong> (PyPI and GitHub latest <strong>1.17.1</strong>, Apache-2.0, Python &gt;=3.11; docs at <a href=\"https://huggingface.co/docs/trl/index\">huggingface.co/docs/trl/index</a>) is the trainer library. Besides the trainers above, the <a href=\"https://huggingface.co/docs/trl/sft_trainer\">SFT trainer documentation</a> has a section on training adapters with PEFT (<code>peft_config=LoraConfig()</code>) and notes that you can pass an already-loaded <code>PeftModel</code> without <code>peft_config</code>. The <a href=\"https://huggingface.co/docs/trl/peft_integration\">PEFT integration page</a> documents QLoRA by combining <code>peft_config</code> with <code>quantization_config</code> (<code>BitsAndBytesConfig</code>). A separate &quot;Tool Calling with SFT&quot; section, together with the <a href=\"https://huggingface.co/docs/trl/dataset_formats\">dataset formats</a> page, specifies that each example should include conversation messages (with <code>tool_calls</code> and <code>tool</code> role messages) and a <code>tools</code> column of JSON schemas. The README also mentions optional Unsloth kernels; if you want Unsloth as a user-facing fine-tuning product, use the frameworks comparison rather than this stack.</p>\n<p><strong>PEFT</strong> (PyPI and GitHub latest <strong>0.21.2</strong>, Apache-2.0, Python &gt;=3.10; docs at <a href=\"https://huggingface.co/docs/peft/index\">huggingface.co/docs/peft/index</a>) is the adapter library. The README quickstart wraps a causal LM with <code>LoraConfig</code> (<code>r</code>, <code>lora_alpha</code>, <code>task_type=TaskType.CAUSAL_LM</code>) and <code>get_peft_model</code>. In this stack you typically let TRL apply that config through <code>SFTTrainer</code> rather than wrapping the model yourself, unless you are continuing an existing <code>PeftModel</code>.</p>\n<p><strong>OpenRLHF</strong> (PyPI and GitHub latest <strong>0.11.2</strong>, Apache-2.0 in the repository <code>LICENSE</code>, Python &gt;=3.10; docs at <a href=\"https://openrlhf.readthedocs.io/en/latest/\">openrlhf.readthedocs.io/en/latest/</a>) is the Ray-based scale-out RLHF option. The README describes a Ray scheduler that can place Actor, Reward, Reference and Critic models on separate GPUs, a Hybrid Engine, generation with vLLM, and training with DeepSpeed ZeRO-3, deepcompile, AutoTP and RingAttention, on Hugging Face Transformers models. Agent execution uses <code>AgentExecutorBase</code> with <code>SingleTurnExecutor</code> and <code>MultiTurnExecutor</code>; the README says algorithms are decoupled from execution mode. Async RLHF is documented via <code>--train.async_enable</code>, and custom or multi-turn agents via <code>--train.agent_func_path</code>. The algorithm table lists <strong>PPO</strong> (default), <strong>REINFORCE++</strong> (<code>reinforce</code>), <strong>RLOO</strong> (<code>rloo</code>), <strong>GRPO</strong> (<code>group_norm</code>), <strong>Dr. GRPO</strong> (<code>dr_grpo</code>) and <strong>FlashREINFORCE</strong> (<code>flash_reinforce</code>); the surrounding text also names <strong>REINFORCE++-baseline</strong>. The project additionally documents SFT, DPO/IPO/cDPO and reward-model scripts. Project wording such as &quot;the first RLHF framework&quot; and claims about generation-time share or model scale are vendor statements and are not used as facts here.</p>\n<p><strong>AReaL</strong> is the asynchronous / agentic RL option. The project README presents it as a large-scale asynchronous reinforcement learning system, &quot;built on a fully asynchronous RL training paradigm,&quot; with a documented <a href=\"https://areal-ai.io/docs/en/tutorial/agentic_rl.html\">agentic RL</a> tutorial and an <a href=\"https://areal-ai.io/docs/en/algorithms/async.html\">asynchronous RL</a> guide. The README says all listed RL algorithms support asynchronous and synchronous versions by setting <code>max_head_offpolicyness=0</code>. The <strong>support matrix</strong> (checked on the <code>main</code> README on 12 November 2026) lists algorithms <strong>GRPO, GSPO, PPO, DAPO, LitePPO, Dr.GRPO, REINFORCE++, RLOO, SAPO, IcePop, KPop, M2PO, DPO</strong>, plus reward modelling, SFT and distillation; training backends <strong>Megatron</strong>, <strong>PyTorch FSDP</strong> and <strong>PyTorch Archon</strong>; and inference backends <strong>vLLM</strong> and <strong>SGLang</strong>. Documented agentic examples include a general agent workflow, tool-integrated reasoning, a search agent, coding-agent RL, and integrations with agent frameworks. Official docs live at <a href=\"https://areal-ai.io/docs/en/intro.html\">areal-ai.io/docs/en/intro.html</a>. Do not use the older <code>areal-project.github.io</code> documentation URLs, which redirect, or treat the stale documentation URL still listed on PyPI as the current docs location.</p>\n<p>Pin AReaL carefully: the latest <strong>GitHub release</strong> on 12 November 2026 is <strong>v2.1.1</strong> (published 4 November 2026). The latest <strong>PyPI</strong> package <code>areal</code> is still <strong>1.0.4</strong> (uploaded 10 June 2026, Apache-2.0, Python &gt;=3.11,&lt;3.13). That wheel predates the README&#39;s AReaL 2.0 notes (1 July 2026) and there is no 2.x series on PyPI. Install from the project&#39;s current docs, not from an assumption that <code>pip install areal</code> matches <code>main</code>.</p>\n<h2>A minimal SFT + LoRA example</h2>\n<p>This sample is copied from the official <a href=\"https://huggingface.co/docs/trl/sft_trainer\">SFT trainer</a> section &quot;Train adapters with PEFT.&quot; It is the documented way to combine TRL and PEFT for LoRA SFT:</p>\n<pre><code class=\"language-python\">from datasets import load_dataset\nfrom trl import SFTTrainer\nfrom peft import LoraConfig\n\ndataset = load_dataset(&quot;trl-lib/Capybara&quot;, split=&quot;train&quot;)\n\ntrainer = SFTTrainer(\n    &quot;Qwen/Qwen3-0.6B&quot;,\n    train_dataset=dataset,\n    peft_config=LoraConfig(),\n)\n\ntrainer.train()</code></pre>\n<p>The same page documents passing a trainable <code>PeftModel</code> instead of <code>peft_config</code> when you continue an existing adapter. For QLoRA, the <a href=\"https://huggingface.co/docs/trl/peft_integration\">PEFT integration page</a> shows combining <code>peft_config</code> with <code>quantization_config</code>. For tool-use SFT, keep this trainer wiring and follow the tool-calling dataset format on the <a href=\"https://huggingface.co/docs/trl/dataset_formats\">dataset formats</a> page rather than inventing a schema. The CLI equivalent for plain SFT (without the PEFT argument shown above) is documented in the TRL README as <code>trl sft</code>.</p>\n<h2>When to add OpenRLHF or AReaL</h2>\n<p>Stay on TRL+PEFT while you are doing single-process or Accelerate-managed SFT, LoRA, DPO or GRPO and the official trainers cover the method. Move to <strong>OpenRLHF</strong> when you want that library&#39;s Ray + vLLM + DeepSpeed layout, its single-turn or multi-turn agent executors, or its async flag. Move to <strong>AReaL</strong> when you want the project&#39;s fully asynchronous training loop and the agentic RL examples and backends in its support matrix (including Megatron or SGLang, which OpenRLHF&#39;s README does not present as first-class in the same way). For other agent RL engines, start from <a href=\"/compare/agent-reinforcement-learning-stacks/\">Agent RL stacks compared</a> instead of forcing them into this stack.</p>\n<h2>Deprecated: do not start with torchtune</h2>\n<p><a href=\"/entries/torchtune/\">torchtune</a> is listed here only so it is not mistaken for a current recommendation. Its README states that it is no longer actively maintained and that development wound down in 2025, pointing at GitHub issue <a href=\"https://github.com/meta-pytorch/torchtune/issues/2883\">#2883</a> (&quot;The future of torchtune&quot;), which was still open on 12 November 2026. PyPI remains at <strong>0.6.1</strong> (BSD-3-Clause, uploaded 7 April 2025). The repository is not archived. Do not start a new post-training stack with it.</p>\n<h2>Licences and versions (12 November 2026)</h2>\n<p>Checked on PyPI, GitHub latest releases, and each repository&#39;s licence file:</p>\n<table><thead><tr><th scope=\"col\">Package</th><th scope=\"col\">Licence file</th><th scope=\"col\">PyPI / GitHub latest</th><th scope=\"col\">Requires Python</th></tr></thead><tbody><tr><td>trl</td><td>Apache License 2.0 (<code>LICENSE</code>)</td><td>1.17.1 / v1.17.1</td><td>&gt;=3.11</td></tr><tr><td>peft</td><td>Apache License 2.0 (<code>LICENSE</code>)</td><td>0.21.2 / v0.21.2</td><td>&gt;=3.10</td></tr><tr><td>openrlhf</td><td>Apache License 2.0 (<code>LICENSE</code>)</td><td>0.11.2 / v0.11.2</td><td>&gt;=3.10</td></tr><tr><td>areal</td><td>Apache License 2.0 (<code>LICENSE</code>)</td><td>PyPI 1.0.4; GitHub latest v2.1.1</td><td>PyPI 1.0.4: &gt;=3.11,&lt;3.13</td></tr></tbody></table>\n<p>OpenRLHF&#39;s PyPI metadata does not set a licence field; the Apache-2.0 text is from the repo <code>LICENSE</code>. Model weights, GPUs, and any hosted inference remain separate from these software licences.</p>\n<h2>What we ran, and what we did not</h2>\n<p>We did <strong>not</strong> install TRL, PEFT, PyTorch, OpenRLHF or AReaL, and we did not run the official SFT+LoRA sample. That sample loads <code>Qwen/Qwen3-0.6B</code> and the <code>trl-lib/Capybara</code> training split; a useful CPU-only smoke test was not feasible here because PyTorch was not already present, installing it is a large download, and the documented example is not a tiny CPU model. No fine-tune was run, and no speed, memory or accuracy figure from project READMEs or blogs was reproduced.</p>"
}
