Index Agentica

Research agent stack

An agent that searches the web and the scholarly record, reads full sources, runs analysis code in a sandbox and cites what it found, with tracing so you can audit how it got there.

Type
Stack
Author
Agentica Author
Published
Last verified

Use case

Answer open research questions with cited sources by searching the web and academic indexes, reading full texts, running analysis in a sandbox and logging every step for review.

Components

RoleComponentWhy
Agent harnessClaude Agent SDKGives you Claude Code's agent loop, tools and context management as a Python or TypeScript library, with MCP support, so the research loop runs in your own app or CI.
Web searchExa APISearch built for agents on Exa's own index, with page contents in the same call. A hosted MCP server at mcp.exa.ai/mcp works without an API key to start.
Web search (alternative)Tavily APISearch plus extract, crawl and map endpoints returning LLM-ready content; a good second source when you want to cross-check results across indexes.
Page readingFirecrawl APIScrapes pages to clean Markdown, with browser actions for dynamic pages, plus crawl and map endpoints; open source (AGPL-3.0) if you need to self-host.
Quick page readingJina ReaderPrefix any URL with r.jina.ai to get Markdown. Use an API key; anonymous requests are rate-limited and can be refused from cloud networks.
Scholarly metadataOpenAlex APIOpen catalog of works, authors, institutions and citations with CC0 data. A free account's API key includes $1 of usage per day; paid plans add more.
Papers and citationsSemantic Scholar Academic Graph APIPaper search, citation graphs and author data. Most endpoints work without a key on a shared, throttled pool; a free key gives higher limits.
PreprintsarXiv APISearch and metadata for arXiv preprints. The terms allow one request every three seconds on a single connection, so queue and cache calls.
BrowserPlaywright MCPA real browser for pages that block simple fetches or need clicking; works from accessibility snapshots rather than screenshots.
Analysis sandboxE2BRuns the agent's pandas or plotting code in a disposable microVM, with the network turned off once data is loaded.
Tracing and evalsLangfuseOpen-source tracing of every model call and tool call (MIT outside its enterprise directories), so you can check which sources an answer really came from and build evals.

How the pieces fit

A research agent runs the same loop over and over: plan the question, search, read, take notes, check, write. The stack above maps one component to each step.

  1. Search wide, then deep. Start with a web search API (Exa, with Tavily, Brave Search or Parallel as alternatives) for recent and general sources, and the scholarly APIs (OpenAlex, Semantic Scholar, arXiv) for peer-reviewed work and citation trails. Search results are snippets; don't let the agent cite a snippet.
  2. Read the source. Fetch full pages as Markdown with Firecrawl or Jina Reader, and fall back to a real browser (Playwright MCP) when a page needs JavaScript or interaction.
  3. Compute in a sandbox. When the question needs numbers, have the agent write analysis code and run it in E2B, not on your machine. Load the data, then cut the network. The sandboxing guide explains why.
  4. Trace everything. Send every model call and tool call to Langfuse. For a research agent the trace is the audit trail: it shows which fetched document each claim came from.

The harness here is the Claude Agent SDK; the OpenAI Agents SDK or LangGraph work the same way if you prefer them. If your harness speaks MCP, most of these components can be connected as MCP servers instead of custom tools; see the remote MCP guide.

Practical limits to plan for

To choose between the search APIs, see the web search APIs comparison.

Directory entries in this stack

Related

Sources

Machine-readable