Add Shared-Memory Governance Benchmark (SMGB): dataset + evaluation#32
Closed
mrrahman1517 wants to merge 1 commit into
Closed
Add Shared-Memory Governance Benchmark (SMGB): dataset + evaluation#32mrrahman1517 wants to merge 1 commit into
mrrahman1517 wants to merge 1 commit into
Conversation
…et + evaluation
Single-user memory benchmarks (LongMemEval, LoCoMo, DMR) only score recall and
cannot tell a governed shared-memory system from a naive one. SMGB adds a public,
system-agnostic benchmark for the governance of shared agent memory:
authorization-filtered retrieval, private->shared promotion, tenant isolation,
conflict/supersession, scope-aware deletion, and provenance.
Ground truth is derived deterministically from a scope + policy graph (no LLM
judge), so scoring is reproducible and free. A run is just {query_id: [ranked
ids]}, so any memory system can be scored.
- benchmarks/governance/: schema, deterministic policy, system-agnostic scorer,
loader with hand-label validation, and a CLI (run_governance.py).
- data/seed_scenarios.jsonl: 6 scenarios / 14 queries across all 7 axes,
including two private->shared promotion cases, tenant isolation, supersession,
deletion, and provenance. Reproducible via data/_generate_seed.py.
- Reference runners (oracle / naive_shared / naive_global) demonstrate the point:
all three score mean_recall 1.000, but only the leakage/isolation/stale axes
separate them (naive_shared: 13 leaks; naive_global: 2 isolation violations).
- benchmarks/tests/test_governance_{policy,scorer,seed}.py: 16 tests.
- Docs/shared_memory_governance_benchmark.md: formal design + verified gap
analysis (PiSAs 2607.05318, ArgusFleet 2606.24535).
Self-contained: introduces the benchmarks/ package scaffold. A companion
multi-agent consistency harness is tracked separately.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mrrahman1517
force-pushed
the
feature/shared-memory-governance-benchmark
branch
from
f3040e3 to
813d691
Compare
mrrahman1517
changed the base branch from
feature/multi-agent-shared-memory
to
main
Contributor
There was a problem hiding this comment.
Pull request overview
Adds the Shared-Memory Governance Benchmark (SMGB) to evaluate governed shared-memory systems on governance failure modes (leakage, tenant isolation, promotion timing, supersession/staleness, deletion, provenance) that recall-only single-user benchmarks cannot detect.
Changes:
- Introduces
benchmarks/governance/with a deterministic policy layer, system-agnostic scorer, dataset loader/validator, and CLI runner. - Adds a checked-in SMGB seed dataset (
seed_scenarios.jsonl) plus a reproducible generator script. - Adds unit + dataset integrity tests and documentation (design doc + package README), and links the design doc from
Docs/README.md.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| Docs/shared_memory_governance_benchmark.md | Adds the formal SMGB design, gap analysis, axes, and reference-benchmark motivation. |
| Docs/README.md | Links the SMGB design doc under “Research & Benchmarks”. |
| benchmarks/init.py | Adds a top-level benchmarks package marker. |
| benchmarks/governance/init.py | Exposes the SMGB public API (loader/policy/scorer/schema + reference runners). |
| benchmarks/governance/README.md | Documents SMGB purpose, quick start, axes, scoring model, and dataset format. |
| benchmarks/governance/schema.py | Defines dependency-free dataclasses + parsing/constants for scenarios/queries/memories/events. |
| benchmarks/governance/policy.py | Implements deterministic authorization/currentness labeling from scope + timeline. |
| benchmarks/governance/scorer.py | Implements system-agnostic scoring + reference runners and report aggregation. |
| benchmarks/governance/loader.py | Loads JSONL scenarios and validates referential integrity + hand labels vs policy. |
| benchmarks/governance/run_governance.py | Adds CLI for validation, reference runner scoring, and external run scoring. |
| benchmarks/governance/data/seed_scenarios.jsonl | Adds the seed dataset (6 scenarios / 14 queries) covering all axes. |
| benchmarks/governance/data/_generate_seed.py | Provides a reproducible generator for the committed seed dataset. |
| benchmarks/tests/init.py | Test package marker. |
| benchmarks/tests/test_governance_policy.py | Adds unit tests for promotion, isolation, supersession/history, deletion, label complements. |
| benchmarks/tests/test_governance_scorer.py | Adds unit tests for scorer discrimination, abstention semantics, k cutoff behavior. |
| benchmarks/tests/test_governance_seed.py | Adds integrity tests for the checked-in seed dataset and reference baseline discrimination. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+177
to
+188
| """Fraction of retrieved targets whose claimed author matches ground truth.""" | ||
| targets = labels.must_retrieve & retrieved_set | ||
| if not targets: | ||
| return 0, len(labels.must_retrieve) | ||
| claims = (provenance or {}).get(query.id, {}) | ||
| correct = 0 | ||
| for mem_id in targets: | ||
| gold_author = scenario.memories[mem_id].provenance.author | ||
| claimed = claims.get(mem_id, {}).get("author") | ||
| if claimed is not None and claimed == gold_author: | ||
| correct += 1 | ||
| return correct, len(labels.must_retrieve) |
Comment on lines
+204
to
+210
| def oracle_run(scenario: Scenario, k: int = 10) -> Run: | ||
| """Perfect system: return exactly the authorized-and-relevant target set.""" | ||
| run: Run = {} | ||
| for query in scenario.queries: | ||
| labels = compute_query_labels(scenario, query) | ||
| run[query.id] = list(labels.must_retrieve)[:k] | ||
| return run |
Comment on lines
+213
to
+222
| def oracle_provenance(scenario: Scenario) -> ProvenanceClaims: | ||
| """Gold provenance claims matching :func:`oracle_run` (all authors correct).""" | ||
| claims: ProvenanceClaims = {} | ||
| for query in scenario.queries: | ||
| labels = compute_query_labels(scenario, query) | ||
| claims[query.id] = { | ||
| mem_id: {"author": scenario.memories[mem_id].provenance.author} | ||
| for mem_id in labels.must_retrieve | ||
| } | ||
| return claims |
Comment on lines
+223
to
+240
| def from_dict(d: dict[str, Any]) -> "Query": | ||
| axis = d.get("axis", "utility") | ||
| if axis not in QUERY_AXES: | ||
| raise ValueError(f"unknown query axis {axis!r}; expected {sorted(QUERY_AXES)}") | ||
| mr = d.get("must_retrieve") | ||
| mnr = d.get("must_not_retrieve") | ||
| return Query( | ||
| id=d["id"], | ||
| principal=d["principal"], | ||
| as_of=parse_time(d["as_of"]), | ||
| axis=axis, | ||
| question=d.get("question", ""), | ||
| relevant=tuple(d.get("relevant", ()) or ()), | ||
| temporal_mode=d.get("temporal_mode", "current"), | ||
| expected_answer=d.get("expected_answer"), | ||
| must_retrieve=tuple(mr) if mr is not None else None, | ||
| must_not_retrieve=tuple(mnr) if mnr is not None else None, | ||
| ) |
Comment on lines
+62
to
+66
| for scope_id in principal.scopes: | ||
| if scope_id not in scenario.scopes: | ||
| problems.append( | ||
| f"{sid}: principal {principal.id!r} references unknown scope {scope_id!r}" | ||
| ) |
Comment on lines
+68
to
+72
| for mem in scenario.memories.values(): | ||
| if mem.scope not in scenario.scopes: | ||
| problems.append(f"{sid}: memory {mem.id!r} initial scope {mem.scope!r} unknown") | ||
| if mem.tenant not in scenario.tenants: | ||
| problems.append(f"{sid}: memory {mem.id!r} tenant {mem.tenant!r} not in tenants") |
Comment on lines
+77
to
+78
| if ev.type == "promote" and (not ev.to_scope or ev.to_scope not in scenario.scopes): | ||
| problems.append(f"{sid}: promote event has unknown to_scope {ev.to_scope!r}") |
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Single-user memory benchmarks (LongMemEval, LoCoMo, DMR — the ones in PR #31) only score recall. They can't tell a governed shared-memory system from a naive one. This PR adds the Shared-Memory Governance Benchmark (SMGB): a public, system-agnostic benchmark for the governance of shared agent memory.
The one-table argument
Reference runners on the seed set (
python -m benchmarks.governance.run_governance --reference):All three baselines score
mean_recall = 1.000— a recall-only benchmark would rate them identical. Only the leakage / isolation / stale-propagation axes reveal thatnaive_sharedleaks 13 facts andnaive_globalalso breaches tenant isolation. Recall alone cannot distinguish governed memory from naive; SMGB can.Design highlights
{query_id: [ranked memory_ids]}. Any memory system can be scored; no SDK or network needed.as_of; promotion / supersession / deletion are timeline events. The same question before vs after a promotion has different correct answers.Seven axes
utility,leakage,isolation,promotion,conflict(supersession),deletion,provenance.What's included
benchmarks/governance/—schema.py,policy.py(deterministic core),scorer.py,loader.py,run_governance.py(CLI), packageREADME.md.benchmarks/governance/data/seed_scenarios.jsonl— 6 scenarios / 14 queries covering all axes (incl. the two private->shared promotion cases from the whiteboard), reproducible via_generate_seed.py.benchmarks/tests/test_governance_{policy,scorer,seed}.py— 16 tests.Docs/shared_memory_governance_benchmark.md— formal design + verified gap analysis, for team circulation.Verification
python -m pytest benchmarks/tests -q-> 16 governance tests passruff check benchmarks/governance-> clean--reference,--validate-only,--json-outall verifiedScope
Self-contained and based directly on
main(introduces thebenchmarks/package scaffold). A companion multi-agent consistency harness — durability/staleness of shared writes under concurrency, per Golab et al. — is tracked separately and can land later; this PR does not depend on it.Open questions for review
Scenario coverage priorities, scope taxonomy, whether to ship the live Cosmos adapter in v1, provenance-chain depth, and public naming — see the "Open questions" section of the design doc.