Index Agentica

Screen tool and web output for prompt injection before your agent reads it

Put a check between untrusted tool results and the model: Llama Prompt Guard 2 for injection attempts, LlamaFirewall's scanners for hidden text and generated code, and Llama Guard for content safety. Includes a tested screening wrapper and the limits of each layer.

Type
Guide
Author
Agentica Author
Published
Last verified
Difficulty
intermediate
Time
40 min

Prerequisites

Why screen tool output

An agent that browses, reads email or opens files puts text written by strangers into its own context. The Prompt Guard 2 model card describes the risk: prompt injections "manipulate untrusted third-party and user data in the context window to make a model execute unintended instructions." This guide adds one check at that boundary. Each tool result is scored before the model sees it, and a flagged result is withheld or sent to a person.

That check is one layer. Classifiers miss attacks, so keep the other layers too: give tools as little power as the task needs, ask a person before anything irreversible, and test the whole agent with AgentDojo.

Which tool does what

ToolWhat it checksRuns whereUse it on tool output for
Llama Prompt Guard 2 22M / 86MText that "explicitly attempts to override prior instructions" (binary benign/malicious)Locally, CPU or GPUThe main injection check
LlamaFirewall 1.0.3Wraps Prompt Guard 2 plus hidden-ASCII, regex, CodeShield and AlignmentCheck scannersLocally, except AlignmentCheck (a hosted LLM by default)Hidden Unicode text, known strings, unsafe generated code, auditing the agent's next action
Llama Guard 4 12BContent safety: MLCommons hazard categories S1 to S13, plus S14 Code Interpreter AbuseLocally, one GPUHarmful content in inputs or outputs, not injection
HOL GuardAgent actions and, per its README, "prompts and tool results" for prompt injection and sensitive contentHooks in the agent harnessCoding agents where you can't edit the loop

Llama Guard is a content-safety classifier, not an injection detector. Its own model card says it "may be susceptible to adversarial attacks or prompt injection attacks" and points to Prompt Guard 2 "for detecting prompt attacks". Use it if you also need to filter harmful content. The rest of this guide is about injection.

What Prompt Guard 2 detects, and what it misses

From the model card:

Meta's reported numbers:

ModelAUC (English)Recall at 1% FPR (English)AUC (multilingual)Latency, A100, 512 tokensAgentDojo APR at 3% utility reduction
Prompt Guard 2 86M0.99897.5%0.99592.4 ms81.2%
Prompt Guard 2 22M0.99588.7%0.94219.3 ms78.4%
Prompt Guard 10.98721.2%0.98392.4 ms67.6%

None of these figures is 100%, and they are Meta's own measurements on its own test sets. Plan for misses.

The weights are gated. The Hugging Face repositories need an approved access request under the Llama 4 Community License. We had no approved account for this guide, so we did not download the model. Everything below was run, but the classifier was a stand-in. The last section, How we tested, has the details.

Get the model onto the machine once

Request access on the model page. After approval, download the files to a fixed directory, from a machine where you are logged in:

hf auth login
hf download meta-llama/Llama-Prompt-Guard-2-22M --local-dir ./models/Llama-Prompt-Guard-2-22M

Your agent then loads from that directory and needs no Hugging Face token or network access at run time.

A screener for tool output

This module loads the classifier with local_files_only=True, so it can't download anything or prompt for a login. It splits the text into 512-token windows that overlap by 64 tokens, scores each window and keeps the highest score. The cookbook's inference utilities also take the maximum over chunks. The module also counts characters from the Unicode Tags block, which render as nothing but can carry ASCII text. Input that needs more than max_chunks windows is flagged instead of being partly scored. The result holds numbers and flags only, never the text.

"""Screen untrusted tool or web output with Llama Prompt Guard 2 before an agent reads it.

Loads the model from local files only: it never downloads and never asks for a
Hugging Face login. Results never include the screened text.
"""
import unicodedata

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

TAG_CHARS = range(0xE0000, 0xE0080)  # Unicode Tags block: invisible "hidden ASCII"


class Screener:
    def __init__(self, model_path, threshold=0.9, window=512, overlap=64, max_chunks=64):
        self.tok = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
        self.model = AutoModelForSequenceClassification.from_pretrained(
            model_path, local_files_only=True).eval()
        labels = {v.upper(): int(k) for k, v in self.model.config.id2label.items()}
        if "MALICIOUS" not in labels:
            raise ValueError(f"no MALICIOUS label in {self.model.config.id2label}")
        self.bad = labels["MALICIOUS"]
        self.threshold = threshold
        self.window = window
        self.body = window - len(self.tok("")["input_ids"])  # leave room for [CLS]/[SEP]
        self.step = self.body - overlap
        self.max_chunks = max_chunks

    def chunks(self, text):
        ids = self.tok(text, add_special_tokens=False)["input_ids"]
        starts = range(0, max(len(ids) - self.body, 0) + self.step, self.step)
        return [ids[s:s + self.body] for s in starts] or [[]]

    @torch.no_grad()
    def score(self, text, batch=16):
        best = 0.0
        parts = self.chunks(text)
        for i in range(0, len(parts), batch):
            texts = [self.tok.decode(p) for p in parts[i:i + batch]]
            enc = self.tok(texts, truncation=True, max_length=self.window,
                           padding=True, return_tensors="pt")
            probs = torch.softmax(self.model(**enc).logits, dim=-1)[:, self.bad]
            best = max(best, probs.max().item())
        return best, len(parts)

    def screen(self, text):
        hidden = sum(ord(c) in TAG_CHARS for c in text)
        fmt = sum(unicodedata.category(c) == "Cf" for c in text) - hidden
        n = len(self.chunks(text))
        if n > self.max_chunks:  # fail closed rather than score part of it
            return {"flagged": True, "score": None, "chunks": n, "too_long": True,
                    "tag_chars": hidden, "other_format_chars": fmt}
        score, n = self.score(text)
        flagged = score >= self.threshold or hidden > 0
        return {"flagged": flagged, "score": round(score, 4), "chunks": n,
                "too_long": False, "tag_chars": hidden, "other_format_chars": fmt}

The 0.9 threshold matches LlamaFirewall's default for its PromptGuard scanner. Tune it on a sample of your real tool output, because false positives are what make people switch a guard off. other_format_chars (zero-width spaces and similar) is reported but not blocked, since ordinary pages contain them.

Wrap every tool that returns outside text:

import logging
from screen_tool_output import Screener

log = logging.getLogger("screen")
screener = Screener("./models/Llama-Prompt-Guard-2-22M")

def guarded(tool_name, output):
    verdict = screener.screen(output)
    log.info("screened %s: %s", tool_name, verdict)  # the verdict holds no tool text
    if verdict["flagged"]:
        return f"[{tool_name} output withheld: flagged as possible prompt injection, score {verdict['score']}]"
    return output

Return the placeholder to the model so it knows the call happened. Keep the original in a quarantine store that a person can review. Don't put it in the same log as the verdict.

LlamaFirewall: what works on tool output

LlamaFirewall assigns scanners per message role. ToolMessage has the role Role.TOOL. With no configuration, tool messages get CodeShield and PromptGuard. When several scanners run, any BLOCK wins.

This configuration ran on CPU with no gated model and no API key:

from llamafirewall import LlamaFirewall, Role, ScanDecision, ScannerType, ToolMessage

fw = LlamaFirewall({Role.TOOL: [ScannerType.HIDDEN_ASCII, ScannerType.REGEX]})
result = fw.scan(ToolMessage(content=tool_output))
if result.decision != ScanDecision.ALLOW:
    ...  # withhold or ask a person

What we saw with llamafirewall 1.0.3:

Gotchas with ScannerType.PROMPT_GUARD in 1.0.3, the latest release on PyPI (May 2025):

AlignmentCheck is different. Instead of looking at the tool output, it asks an LLM whether the agent's latest action still serves the user's original request. That can catch injections a classifier misses. In 1.0.3 the firewall builds it with fixed defaults: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 at https://api.together.xyz/v1, with the key in TOGETHER_API_KEY. It sends the whole trace, user request included, to that endpoint. Check that this is acceptable for your data before you turn it on. We did not run AlignmentCheck.

Put it together

  1. Mark which tools return outside content: web fetch, search, email, files, other agents and MCP servers you don't run yourself.
  2. Screen each result with the screener, plus LlamaFirewall's hidden-ASCII check or your own patterns. Withhold flagged results and keep them for review.
  3. Limit what a successful injection can do: read-only credentials, allowlisted destinations and a person approving sends, payments and deletes. The guardrail frameworks comparison covers policy layers that check tool calls.
  4. For coding agents, HOL Guard works through the agent's own hooks and MCP proxies. Its README notes that coverage "depends on the events each agent exposes", so check its support matrix for your harness.
  5. Measure. Run AgentDojo with and without the screener, and add attacks from the red-teaming tools. A guard you haven't measured is a guess.

How we tested

Run on 2026-11-03 on CPU (Python 3.13.5, torch 2.15.0+cpu, llamafirewall 1.0.3, codeshield 1.0.1).

Directory entries in this guide

Related

Sources

Machine-readable