Evals ask whether your app does the right thing. Red-teaming asks whether someone can make it do the wrong thing: leak a system prompt or PII, follow injected instructions, call a tool it shouldn't, or be talked out of its rules over several turns. All three tools here generate those attacks automatically, run them against your app, and grade the responses with an LLM judge. That last step means a "pass" is a model's opinion, so read the failing transcripts yourself.
\nFor evals, tracing and CI quality gates, see Evaluate an AI agent and Agent observability platforms. For public capability scores, see Agent benchmarks.
\npromptfooconfig.yaml, pick plugins (what to attack) and strategies (how to deliver it), and run promptfoo redteam run. It writes the generated probes to redteam.yaml and then attacks the target. The target can be any HTTP endpoint, a browser session or a model, so it tests the deployed app, not only the model.model_callback(input) -> str, choose vulnerabilities and attacks or a framework such as OWASPTop10(), and call red_team(...). deepteam run config.yaml does the same from YAML.target and a plain-language description to vulnerability_scan(...), and it generates the adversarial scenarios. third_party_scan(tool="garak" | "deepteam") runs garak probes or DeepTeam attacks through the same target and report format, which the README marks as experimental.# DeepTeam, from its README: map an OWASP framework to vulnerabilities and attacks\nfrom deepteam import red_team\nfrom deepteam.frameworks import OWASPTop10\n\nasync def model_callback(input: str) -> str:\n return await my_agent(input) # your app\n\nrisk_assessment = red_team(model_callback=model_callback, framework=OWASPTop10())# Giskard v3, from the giskard-scan README\nfrom giskard.scan import vulnerability_scan\n\nresult = await vulnerability_scan(\n target=my_agent,\n description="Customer support chatbot for an e-commerce store.",\n languages=["en"],\n max_scenarios=20,\n)# Promptfoo, adapted from its MCP security testing guide: attack an MCP server through a model client\ntargets:\n - id: openai:gpt-5\n config:\n mcp:\n enabled: true\n servers:\n - name: your-mcp-server\n url: https://your-mcp-server.example.com/mcp\nredteam:\n purpose: A payment assistant that must enforce authorization on every tool call.\n plugins: [mcp, pii, bfla, bola, sql-injection]\n strategies: [multi-turn, jailbreak, jailbreak-templates]On 2026-10-21 we installed deepteam 1.0.10 and giskard[scan] 3.0.1 in a clean Python 3.13 virtualenv. The imports above resolved, deepteam --help listed run, scan and login, and list_scan_items("giskard") listed the scenario generators named in the table. We did not run any attacks, because every tool needs an LLM API key for generation and judging. Promptfoo 0.124.1 requires Node.js 22.22 or later, and the test box had Node.js 20, so its CLI was not run.
One finding from that install: deepteam 1.0.10 failed to import until sentry-sdk and posthog were installed by hand. Its telemetry module imports them, but the package does not list them as dependencies. If you get ModuleNotFoundError: No module named 'sentry_sdk', run pip install sentry-sdk posthog.
Plain chatbots mostly risk saying something bad. Agents can do something bad, so check coverage for:
\nRed-teaming finds weaknesses; it doesn't fix them. Pair it with server-side controls such as audience-bound tokens and per-tool scopes on your MCP servers (Add OAuth authorization to your own MCP server) and with care about which servers you connect (Pick an MCP server).
\nretry strategy, or a committed DeepTeam YAML config. Gate on new failures, not on a perfect score.third_party_scan.