Index Agentica

Discover a site's agent interfaces

A probe-then-pick workflow for finding the best machine interface a website offers (MCP, A2A, OpenAPI, Agent Skills, llms.txt, NLWeb or WebMCP), with a read-only Python script tested against live sites on 2026-11-01.

Type
Skill
Author
Agentica Author
Published
Last verified
Version
1.1
License
MIT
Compatibility
Needs Python 3.9 or later (standard library only) and HTTPS access to the site you probe. No credentials. By default it contacts only the origin you name (plus any --mcp URLs) and refuses non-public addresses. WebMCP tools can only be called from a WebMCP-capable browser, so the probe just reports hints for them.

Download

Install by unzipping into your agent's skills directory, e.g. for Claude Code: curl -sLO https://indexagentica.com/skills/discover-agent-interfaces.zip && unzip -o discover-agent-interfaces.zip -d ~/.claude/skills/

Files

Description

Find out how an agent can use a website or service before scraping it: probe a domain for an AI Catalog, MCP Server Cards and endpoints, an A2A Agent Card, OAuth protected-resource metadata, an RFC 9727 API catalog, an Agent Skills index, llms.txt and markdown page versions, NLWeb and WebMCP hints. Use when the user names a site, product or domain and wants the agent to search it, call it, connect to it or pull its docs, or asks what agent interfaces a site has. Includes a read-only standard-library Python probe.

SKILL.md

Discover a site's agent interfaces

Use this skill when the user wants you to work with a website or service ("search Hugging Face for...", "what can I do with Vercel from here?", "connect to example.com"). Many sites now publish files and endpoints meant for agents. If you check for them first, you can use a real API instead of scraping HTML.

The workflow has three steps: probe, pick, hand off. Every check below reads a published file or endpoint. Each one is defined by a spec or a draft, linked in the source list.

1. Probe

python3 scripts/probe_site.py https://huggingface.co
python3 scripts/probe_site.py https://developers.cloudflare.com/agents/ --follow-cross-origin --max-mcp 3
python3 scripts/probe_site.py https://example.com --mcp https://api.example.com/mcp --json
Which requests go to which hosts

By default the script talks only to the target origin (the scheme, host and port of the URL you give) and to the origin of each --mcp URL you pass. Everything else the site points at (catalog entry URLs, server-card remotes, resource_metadata URLs in WWW-Authenticate, redirect targets on another origin) is printed under Not fetched, with where it came from. After you or the user have checked who runs those hosts, --follow-cross-origin fetches them as well.

On the target origin, the script sends GET requests for these locations:

LocationWhat it is
/.well-known/ai-catalog.jsonAI Catalog (specVersion "1.0"). This is the domain-level index the MCP Server Card extension tells clients to try first. Each entry has a URN identifier, a media type and either a url or inline data. The script reads MCP Server Card and A2A Agent Card entries that are inline or on the target origin.
/.well-known/mcp/server-card.jsonThe location some sites still use from earlier Server Card drafts. The final extension doc rejects a .well-known card path and puts cards at <streamable-http-url>/server-card.
/.well-known/agent-card.json, then /.well-known/agent.jsonA2A Agent Card. The second path is the pre-0.3 name. The script sends A2A-Version: 1.0, because some servers pick the card shape from that header.
/.well-known/oauth-protected-resourceOAuth Protected Resource Metadata (RFC 9728). MCP 2026-07-28 requires it for protected servers.
/.well-known/api-catalogAPI catalog (RFC 9727, a Standards Track RFC): a linkset of APIs and their OpenAPI or other descriptions.
/.well-known/agent-skills/index.jsonAgent Skills discovery index (Cloudflare-led draft, v0.2.0): downloadable SKILL.md files or archives, each with a sha256 digest.
llms.txt at each path level, most specific first, then /llms.txtllms.txt v2 (August 2026). The script checks that the file opens with an H1 and counts its H2 sections and [name](url) list links.
The page itself<link rel="alternate" type="text/markdown"> (the markdown version of the page), rel="describedby" (the llms.txt that covers it), any Link: header, and WebMCP hints.

For each MCP endpoint (<origin>/mcp unless you set --no-mcp-guess, each --mcp URL, and each server-card remote that is allowed by the rules here), the script sends three requests to that endpoint's host:

  1. A GET for <endpoint>/server-card.
  2. One JSON-RPC server/discover POST, with MCP-Protocol-Version: 2026-07-28 and Mcp-Method: server/discover.
  3. If that POST fails without a 401, one initialize POST using 2025-11-25. This one can open a session on an older server; nothing else is sent on it.

If an endpoint answers 401, the script fetches the resource_metadata URL from WWW-Authenticate, under the same rules. With --try-nlweb it sends one GET /ask query to the target origin. This is off by default, because each query may cost the site an LLM call. The script never calls a tool and never downloads an Agent Skills artifact. --max-mcp (default 6) caps how many endpoints get POSTs.

Headers and addresses

2. Pick

Choose the interface that matches the task. Prefer structured interfaces over prose.

  1. An MCP endpoint, for tools.
  2. MCP 2026-07-28 (stateless) endpoint means server/discover answered. You get supportedVersions, capabilities and serverInfo.
  3. needs auth means a 401 came back. The script prints resource_metadata and, if that URL is on an allowed origin, the authorization servers. Your client runs the OAuth flow, and you don't paste tokens.
  4. not fetched means the endpoint is on another origin, or resolved to a non-public address. Check who runs it before rerunning with --follow-cross-origin.
  5. initialize handshake means the server predates 2026-07-28. Modern clients still connect to it.
  6. An A2A agent, for delegating a whole task to another agent.
  7. Pick the first entry in supportedInterfaces whose protocolBinding (JSONRPC, GRPC or HTTP+JSON) and protocolVersion your client supports.
  8. A v1 card no longer has a top-level url. If you see url, preferredTransport or additionalInterfaces, it is a 0.3-style card.
  9. Send A2A-Version on every request. An empty header is treated as 0.3.
  10. securitySchemes tells you which credential the agent expects. If the card sets capabilities.extendedAgentCard, more skills are available after you authenticate.
  11. An OpenAPI description from the API catalog or AI Catalog, for plain HTTP calls. Read the description, then call only the operations the user asked for.
  12. Agent Skills, for instructions the site maintains about itself.
  13. Before you use an artifact, check that its bytes match digest.
  14. If $schema is a URI you don't recognize, warn the user and don't process the index. The draft requires this.
  15. Read any scripts before running them. Installing a skill is the user's decision.
  16. llms.txt or markdown page versions, for reading docs.
  17. Read or search the llms.txt, then follow only the links you need.
  18. By convention, the ## Optional section holds secondary links you can skip.
  19. Some files are large: indexagentica.com/llms.txt was 138 KB with 626 links on 2026-11-01. Search them instead of loading the whole file.
  20. NLWeb (/ask, also served over /mcp), for natural-language search of a site's content.
  21. query is required. mode is list, summarize or generate, and streaming defaults to true.
  22. Results come back as Schema.org objects.
  23. WebMCP, for tools that run inside the page.
  24. Pages register these with document.modelContext.registerTool() in a secure context, or declare them with <form toolname="..." tooldescription="..."> (the declarative API is still at the explainer stage).
  25. Only a browser agent with WebMCP support can call them. The W3C Community Group draft is dated 27 October 2026, and Chrome 149 and Edge 150 offer it as an origin trial.
  26. The probe only sees what is in the HTML it fetches, so tools registered by external scripts won't show up.

If the probe finds nothing, fall back to the site's documented public API, then to reading pages.

Two protocols you won't find on a website:

The user configures either one locally, so don't look for them on a domain.

3. Hand off

Report to the user:

  1. What was found, with URLs.
  2. Which interface you will use, and why.
  3. Anything that needs their action, such as a sign-in, an API key or a payment.

Safety

What was tested (2026-11-01)

These are live results from the script (v1.1), as the sites served them on 2026-11-01:

Directory entries in this skill

Related

Sources

Machine-readable