OmniMemEval
MemTensor's evaluation framework for memory backends (LoCoMo, LongMemEval, BEAM, PersonaMem v2, HaluMem) and memory-equipped agents on AgentBench domains.
Tags
Description
Two tracks: User Memory Evaluation drives any memory backend through add() and search() and scores answers with configurable answer/eval LLMs; Agent Memory Evaluation runs agent runtimes with memory plugins (OpenClaw) on AgentBench-style domains (reasoning, information retrieval, knowledge work, code, software engineering). Ships data-preparation scripts, env templates per backend (e.g. MemOS) and published result snapshots. Third-party benchmark data keeps its upstream licenses. From the makers of MemOS.
Related
- MemOS: Open-source memory operating system for LLMs and agents with a unified API for adding, retrieving, editing and deleting graph-structured, multi-modal memory.
- Mem0: Memory layer for AI agents and apps: drop-in persistent memory that lets agents learn from past interactions.
- AgentBench: Comprehensive benchmark evaluating LLMs as agents (ICLR'24); the current repo is a function-calling version (AgentBench FC) with containerized tasks.
- OpenClaw: Open-source personal AI assistant that runs on your own machine behind one Gateway and talks to you over WhatsApp, Telegram, Slack, Discord, iMessage and 20+ other channels.
Guides & comparisons
- Memory servers for coding agents compared: m3 Memory vs agentmemory vs Basic Memory vs MCP Memory Service (Comparison): Four self-hosted memory servers that give Claude Code, Codex, Cursor and other MCP clients one shared memory across sessions and tools. Covers install, storage, search, how memories get captured, hosted options and license.
Sources
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)