Feature Request: Release Evaluation Code / Benchmark Framework

[zcg-joker]

Background

Thank you for the excellent work on MemoryArena! The benchmark design for interdependent multi-session agentic tasks is very inspiring, and the dataset on HuggingFace is already helpful for research.

Current Gap

Currently, the repository provides:

• ✅ Project website and paper
• ✅ Dataset via HuggingFace (ZexueHe/memoryarena)

However, there is no publicly available evaluation code / benchmark framework to:

• Run agents on the MemoryArena tasks
• Implement the Memory-Agent-Environment loop described in the paper
• Reproduce the results reported in the paper
• Compare different memory mechanisms under the same protocol

Why This Matters

1. Reproducibility: Code would help the community validate and build upon your findings
2. Adoption: Lower barrier for researchers to use MemoryArena in their work
3. Fair comparison: Standardized evaluation pipeline ensures consistent metrics across methods

Additional Context

The appendix of the paper describes the evaluation protocol, but having executable code would greatly accelerate adoption and reduce implementation ambiguity.

Thank you for considering this request!

[zihan12ai]

+1 to this request.

It's been a few months since this issue was raised with no updates, and the broken code link on the project website still redirects to the GitHub homepage. Given how critical the evaluation framework is for testing the Memory-Agent-Environment loop described in the paper, waiting is no longer a viable option.

I have decided to stop waiting and am currently building a lightweight, open-source evaluation harness from scratch, utilizing the multi-session dataset provided on Hugging Face. My goal is to create a clean, executable pipeline to properly benchmark structured memory mechanisms (specifically dynamic task-tree routing) against these interdependent tasks.

If anyone else is blocked by this missing code and is interested in collaborating, or if you want a reliable pipeline to test your own agentic memory architectures on the MemoryArena dataset, feel free to reach out. I will share the repository link here once the core evaluation loop is stable.