Skip to content

Retire strict_provenance runtime guardrail (#1488)#1489

Merged
dimitri-yatsenko merged 3 commits into
masterfrom
retire/strict-provenance
Jul 15, 2026
Merged

Retire strict_provenance runtime guardrail (#1488)#1489
dimitri-yatsenko merged 3 commits into
masterfrom
retire/strict-provenance

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

Closes #1488.

Summary

Removes the opt-in dj.config["strict_provenance"] flag (introduced in 2.3.0, #1425) and its runtime interception machinery. Comprehensive enforcement of the make() contract is fundamentally a code-inspection problem, not a runtime-interception one — the shipped guardrail was best-effort by construction (documented bypasses via existence/count idioms, restriction-by-table, and any access outside the Python client) and could not be made sound in-process. Enforcement moves to deploy-time static/agentic inspection of make() bodies on the platform, backed by governed, commit-pinned execution. The open-source docs document the make() reproducibility contract itself (the rules) rather than a runtime flag.

Kept (and stabilized)

"Read from self.upstream, write to self and Parts" remains the documented convention — and it is exactly what makes a make() statically verifiable. These are cleanly separable from the removed enforcement code: self._upstream = Diagram.trace(self & key) still runs unconditionally.

Changes

  • Delete src/datajoint/provenance.py — the entire module (enforcement context, read/write gates, per-row key-consistency check) existed only to back the flag.
  • settings.py — drop the strict_provenance config field and its DJ_STRICT_PROVENANCE env-var mapping.
  • expression.py — remove the read gate from QueryExpression.cursor.
  • table.py — remove the write gate and the per-row key-consistency check from the insert path.
  • autopopulate.py — remove the per-make() strict context push/pop; keep the upstream-trace construction and the upstream property (docstring reworded to drop the flag reference and point at the AutoPopulate spec).
  • Delete tests/integration/test_strict_provenance.py.

Why removal, not deprecation

The flag is off by default, has no schema/data/query-result impact, and has no known adoption (no platform customer enabled it). Removing it before it gains use is cheaper than carrying a deprecation cycle for a feature nobody depends on.

Docs

The strict_provenance documentation (2.3 release notes, provenance spec §3) lives in the separate datajoint-docs repo and is handled there — the reproducible make() contract is being documented in reference/specs/autopopulate.md (datajoint-docs #197), and the "trinity" reframed to the trace + self.upstream pair.

Verification

  • datajoint imports cleanly (2.3.0); strict_provenance key removed; Diagram.trace and self.upstream intact; datajoint.provenance no longer importable.
  • No dangling references to the flag or the removed helpers anywhere in src//tests/.
  • ruff, ruff-format, and codespell pre-commit hooks pass. (Full integration test run requires a live database, not available in the authoring environment.)

Remove the opt-in `dj.config["strict_provenance"]` flag (introduced in
2.3.0, #1425) and its runtime interception machinery. Enforcement of the
make() reproducibility contract moves to deploy-time static/agentic
inspection of make() bodies on the platform; the open-source docs
document the contract itself rather than a runtime flag.

Kept intact: `Diagram.trace()` (#1423) and `self.upstream` (#1424) — the
upstream query API and the idiomatic "read from self.upstream, write to
self" pattern that makes a make() statically verifiable.

Changes:
- Delete src/datajoint/provenance.py (enforcement context + read/write/
  key-consistency gates — the whole module supported only the flag).
- settings.py: drop the strict_provenance config field and its
  DJ_STRICT_PROVENANCE env-var mapping.
- expression.py: remove the read gate from QueryExpression.cursor.
- table.py: remove the write gate and per-row key-consistency check from
  the insert path.
- autopopulate.py: remove the per-make() strict context push/pop; keep
  the self._upstream = Diagram.trace(self & key) construction and the
  `upstream` property (docstring reworded to drop the flag reference).
- Delete tests/integration/test_strict_provenance.py.

Removal (not deprecation) is safe: the flag is off by default, has no
schema/data/query-result impact, and has no known adoption.
DataJoint's own docs use the relational workflow model's vocabulary
(lineage, reproducibility); provenance is a platform-level compatibility
concern documented separately.
@dimitri-yatsenko dimitri-yatsenko added the enhancement Indicates new improvements label Jul 13, 2026
dimitri-yatsenko added a commit that referenced this pull request Jul 14, 2026
The strict_provenance read/write-gate surface these tests pinned is being
retired in #1489 (which deletes provenance.py). Remove test_strict_provenance_limits.py
and the now-moot provenance.py docstring edit so this PR is coherent with #1489;
retain test_cascade_integrity.py (companion to #1484).

@MilagrosMarin MilagrosMarin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dimitri-yatsenko — clean removal, 608 lines out and 8 in. Verified provenance.py and test_strict_provenance.py are fully deleted, no dangling references (grep -rn for the removed symbols returns nothing), and the pieces this PR keeps — Diagram.trace() at diagram.py:391 and AutoPopulate.upstream at autopopulate.py:95 — are intact and still wire Diagram.trace(self & dict(key)) unconditionally in _populate_one. The gates come out of expression.cursor and table.insert/_insert_rows without collateral refactoring, the config field and env-var mapping both go, and the docstring/README/test-comment wording follows the reframing away from "provenance" as a runtime term.

The "removal, not deprecation" argument holds: off by default, no schema/data/query-result impact, no known adoption. This is the same reasoning that landed on the docs side in #183's reframing — the runtime guardrail was best-effort by construction and couldn't be made sound in-process.

One thing for the release notes (correcting my earlier phrasing): dj.config["strict_provenance"] = True will now raise a KeyError("Setting 'strict_provenance' not found") at assignment time, not a silent no-op. The path is _ConfigProxy.__setitem__Config.__setitem__ at settings.py:959-966, which checks hasattr(self, key) and raises KeyError when the field is absent. Cleaner error than I initially wrote, but the substance stands: 2.3.0 shipped the flag as public API, so the 2.3.1 release notes should still say "removed" explicitly — a matter of release-notes transparency, not adopter protection (the PR body's "no known adoption" is fair on that front).

Approving.

@dimitri-yatsenko dimitri-yatsenko merged commit 6a3d17f into master Jul 15, 2026
9 checks passed
@dimitri-yatsenko dimitri-yatsenko deleted the retire/strict-provenance branch July 15, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Indicates new improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retire the opt-in strict_provenance runtime guardrail; enforce the make() contract by deploy-time code inspection

2 participants