Add Shared-Memory Governance Benchmark (SMGB): dataset + evaluation#33
Open
mrrahman1517 wants to merge 1 commit into
Open
Conversation
…et + evaluation
Single-user memory benchmarks (LongMemEval, LoCoMo, DMR) only score recall and
cannot tell a governed shared-memory system from a naive one. SMGB adds a public,
system-agnostic benchmark for the governance of shared agent memory:
authorization-filtered retrieval, private->shared promotion, tenant isolation,
conflict/supersession, scope-aware deletion, and provenance.
Ground truth is derived deterministically from a scope + policy graph (no LLM
judge), so scoring is reproducible and free. A run is just {query_id: [ranked
ids]}, so any memory system can be scored.
- benchmarks/governance/: schema, deterministic policy, system-agnostic scorer,
loader with hand-label validation, and a CLI (run_governance.py).
- data/seed_scenarios.jsonl: 6 scenarios / 14 queries across all 7 axes,
including two private->shared promotion cases, tenant isolation, supersession,
deletion, and provenance. Reproducible via data/_generate_seed.py.
- Reference runners (oracle / naive_shared / naive_global) demonstrate the point:
all three score mean_recall 1.000, but only the leakage/isolation/stale axes
separate them (naive_shared: 13 leaks; naive_global: 2 isolation violations).
- benchmarks/tests/test_governance_{policy,scorer,seed}.py: 16 tests.
- Docs/shared_memory_governance_benchmark.md: formal design + verified gap
analysis (PiSAs 2607.05318, ArgusFleet 2606.24535).
Self-contained: introduces the benchmarks/ package scaffold. A companion
multi-agent consistency harness is tracked separately.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds the Shared-Memory Governance Benchmark (SMGB) to evaluate shared-memory governance (authorization, isolation, promotion, supersession, deletion, provenance) in a deterministic, system-agnostic way, complementing the existing single-user recall-focused benchmarks.
Changes:
- Introduces a new
benchmarks.governancepackage: schema + deterministic policy + scorer + dataset loader + CLI runner. - Checks in a seed JSONL dataset (6 scenarios / 14 queries) plus a reproducible generator script.
- Adds unit/integrity tests for policy labeling, scorer discrimination, and seed dataset validity; documents the benchmark design in
Docs/.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| Docs/shared_memory_governance_benchmark.md | Design + gap analysis write-up for SMGB. |
| Docs/README.md | Adds SMGB design doc to the documentation index. |
| benchmarks/init.py | Adds top-level benchmarks package marker. |
| benchmarks/governance/init.py | Exposes SMGB public API surface. |
| benchmarks/governance/README.md | Usage docs, dataset format, and quickstart commands. |
| benchmarks/governance/schema.py | Dataclass schema + time parsing + axis/event enums. |
| benchmarks/governance/policy.py | Deterministic authorization/label derivation logic. |
| benchmarks/governance/scorer.py | Scoring + reporting + reference runners. |
| benchmarks/governance/loader.py | JSONL loader + referential integrity + hand-label validation. |
| benchmarks/governance/run_governance.py | CLI entrypoint for validation + scoring + JSON output. |
| benchmarks/governance/data/seed_scenarios.jsonl | Seed dataset (scenarios/queries). |
| benchmarks/governance/data/_generate_seed.py | Reproducible seed dataset generator script. |
| benchmarks/tests/init.py | Test package marker. |
| benchmarks/tests/test_governance_policy.py | Unit tests for deterministic policy labeling. |
| benchmarks/tests/test_governance_scorer.py | Unit tests for scorer + reference runners. |
| benchmarks/tests/test_governance_seed.py | Integrity tests for checked-in seed dataset. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+111
to
+122
| def summary(self) -> dict[str, Any]: | ||
| return { | ||
| "system": self.system, | ||
| "k": self.k, | ||
| "queries": len(self.per_query), | ||
| "mean_recall": self.mean_recall, | ||
| "leak_rate": self.leak_rate, | ||
| "total_leaks": self.total_leaks, | ||
| "isolation_violations": self.isolation_violations, | ||
| "stale_leaks": self.stale_leaks, | ||
| "provenance_accuracy": self.provenance_accuracy, | ||
| } |
Comment on lines
+206
to
+210
| run: Run = {} | ||
| for query in scenario.queries: | ||
| labels = compute_query_labels(scenario, query) | ||
| run[query.id] = list(labels.must_retrieve)[:k] | ||
| return run |
Comment on lines
+92
to
+107
| # The core check: hand annotations must agree with derived policy. | ||
| labels = compute_query_labels(scenario, q) | ||
| if q.must_retrieve is not None: | ||
| hand = set(q.must_retrieve) | ||
| if hand != set(labels.must_retrieve): | ||
| problems.append( | ||
| f"{sid}: query {q.id!r} must_retrieve {sorted(hand)} != " | ||
| f"policy-derived {sorted(labels.must_retrieve)}" | ||
| ) | ||
| if q.must_not_retrieve is not None: | ||
| hand_not = set(q.must_not_retrieve) | ||
| stray = hand_not - set(labels.forbidden) | ||
| if stray: | ||
| problems.append( | ||
| f"{sid}: query {q.id!r} must_not_retrieve {sorted(stray)} are actually allowed" | ||
| ) |
Comment on lines
+62
to
+66
| for scope_id in principal.scopes: | ||
| if scope_id not in scenario.scopes: | ||
| problems.append( | ||
| f"{sid}: principal {principal.id!r} references unknown scope {scope_id!r}" | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Single-user memory benchmarks (LongMemEval, LoCoMo, DMR — the ones in PR #31) only score recall. They can't tell a governed shared-memory system from a naive one. This PR adds the Shared-Memory Governance Benchmark (SMGB): a public, system-agnostic benchmark for the governance of shared agent memory.
The one-table argument
Reference runners on the seed set (
python -m benchmarks.governance.run_governance --reference):All three baselines score
mean_recall = 1.000— a recall-only benchmark would rate them identical. Only the leakage / isolation / stale-propagation axes reveal thatnaive_sharedleaks 13 facts andnaive_globalalso breaches tenant isolation. Recall alone cannot distinguish governed memory from naive; SMGB can.Design highlights
{query_id: [ranked memory_ids]}. Any memory system can be scored; no SDK or network needed.as_of; promotion / supersession / deletion are timeline events. The same question before vs after a promotion has different correct answers.Seven axes
utility,leakage,isolation,promotion,conflict(supersession),deletion,provenance.What's included
benchmarks/governance/—schema.py,policy.py(deterministic core),scorer.py,loader.py,run_governance.py(CLI), packageREADME.md.benchmarks/governance/data/seed_scenarios.jsonl— 6 scenarios / 14 queries covering all axes (incl. the two private->shared promotion cases from the whiteboard), reproducible via_generate_seed.py.benchmarks/tests/test_governance_{policy,scorer,seed}.py— 16 tests.Docs/shared_memory_governance_benchmark.md— formal design + verified gap analysis, for team circulation.Verification
python -m pytest benchmarks/tests -q-> 16 governance tests passruff check benchmarks/governance-> clean--reference,--validate-only,--json-outall verifiedScope
Self-contained and based directly on
main(introduces thebenchmarks/package scaffold). A companion multi-agent consistency harness — durability/staleness of shared writes under concurrency, per Golab et al. — is tracked separately and can land later; this PR does not depend on it.Open questions for review
Scenario coverage priorities, scope taxonomy, whether to ship the live Cosmos adapter in v1, provenance-chain depth, and public naming — see the "Open questions" section of the design doc.