Retire strict_provenance runtime guardrail (#1488)#1489
Conversation
Remove the opt-in `dj.config["strict_provenance"]` flag (introduced in 2.3.0, #1425) and its runtime interception machinery. Enforcement of the make() reproducibility contract moves to deploy-time static/agentic inspection of make() bodies on the platform; the open-source docs document the contract itself rather than a runtime flag. Kept intact: `Diagram.trace()` (#1423) and `self.upstream` (#1424) — the upstream query API and the idiomatic "read from self.upstream, write to self" pattern that makes a make() statically verifiable. Changes: - Delete src/datajoint/provenance.py (enforcement context + read/write/ key-consistency gates — the whole module supported only the flag). - settings.py: drop the strict_provenance config field and its DJ_STRICT_PROVENANCE env-var mapping. - expression.py: remove the read gate from QueryExpression.cursor. - table.py: remove the write gate and per-row key-consistency check from the insert path. - autopopulate.py: remove the per-make() strict context push/pop; keep the self._upstream = Diagram.trace(self & key) construction and the `upstream` property (docstring reworded to drop the flag reference). - Delete tests/integration/test_strict_provenance.py. Removal (not deprecation) is safe: the flag is off by default, has no schema/data/query-result impact, and has no known adoption.
DataJoint's own docs use the relational workflow model's vocabulary (lineage, reproducibility); provenance is a platform-level compatibility concern documented separately.
There was a problem hiding this comment.
Thanks @dimitri-yatsenko — clean removal, 608 lines out and 8 in. Verified provenance.py and test_strict_provenance.py are fully deleted, no dangling references (grep -rn for the removed symbols returns nothing), and the pieces this PR keeps — Diagram.trace() at diagram.py:391 and AutoPopulate.upstream at autopopulate.py:95 — are intact and still wire Diagram.trace(self & dict(key)) unconditionally in _populate_one. The gates come out of expression.cursor and table.insert/_insert_rows without collateral refactoring, the config field and env-var mapping both go, and the docstring/README/test-comment wording follows the reframing away from "provenance" as a runtime term.
The "removal, not deprecation" argument holds: off by default, no schema/data/query-result impact, no known adoption. This is the same reasoning that landed on the docs side in #183's reframing — the runtime guardrail was best-effort by construction and couldn't be made sound in-process.
One thing for the release notes (correcting my earlier phrasing): dj.config["strict_provenance"] = True will now raise a KeyError("Setting 'strict_provenance' not found") at assignment time, not a silent no-op. The path is _ConfigProxy.__setitem__ → Config.__setitem__ at settings.py:959-966, which checks hasattr(self, key) and raises KeyError when the field is absent. Cleaner error than I initially wrote, but the substance stands: 2.3.0 shipped the flag as public API, so the 2.3.1 release notes should still say "removed" explicitly — a matter of release-notes transparency, not adopter protection (the PR body's "no known adoption" is fair on that front).
Approving.
Closes #1488.
Summary
Removes the opt-in
dj.config["strict_provenance"]flag (introduced in 2.3.0, #1425) and its runtime interception machinery. Comprehensive enforcement of themake()contract is fundamentally a code-inspection problem, not a runtime-interception one — the shipped guardrail was best-effort by construction (documented bypasses via existence/count idioms, restriction-by-table, and any access outside the Python client) and could not be made sound in-process. Enforcement moves to deploy-time static/agentic inspection ofmake()bodies on the platform, backed by governed, commit-pinned execution. The open-source docs document the make() reproducibility contract itself (the rules) rather than a runtime flag.Kept (and stabilized)
Diagram.trace()(Implement Diagram.trace() for upstream restriction propagation #1423)self.upstream(Add self.upstream property for pre-restricted ancestor access in make() #1424)"Read from
self.upstream, write toselfand Parts" remains the documented convention — and it is exactly what makes amake()statically verifiable. These are cleanly separable from the removed enforcement code:self._upstream = Diagram.trace(self & key)still runs unconditionally.Changes
src/datajoint/provenance.py— the entire module (enforcement context, read/write gates, per-row key-consistency check) existed only to back the flag.settings.py— drop thestrict_provenanceconfig field and itsDJ_STRICT_PROVENANCEenv-var mapping.expression.py— remove the read gate fromQueryExpression.cursor.table.py— remove the write gate and the per-row key-consistency check from the insert path.autopopulate.py— remove the per-make()strict context push/pop; keep the upstream-trace construction and theupstreamproperty (docstring reworded to drop the flag reference and point at the AutoPopulate spec).tests/integration/test_strict_provenance.py.Why removal, not deprecation
The flag is off by default, has no schema/data/query-result impact, and has no known adoption (no platform customer enabled it). Removing it before it gains use is cheaper than carrying a deprecation cycle for a feature nobody depends on.
Docs
The
strict_provenancedocumentation (2.3 release notes, provenance spec §3) lives in the separatedatajoint-docsrepo and is handled there — the reproduciblemake()contract is being documented inreference/specs/autopopulate.md(datajoint-docs #197), and the "trinity" reframed to thetrace+self.upstreampair.Verification
datajointimports cleanly (2.3.0);strict_provenancekey removed;Diagram.traceandself.upstreamintact;datajoint.provenanceno longer importable.src//tests/.ruff,ruff-format, andcodespellpre-commit hooks pass. (Full integration test run requires a live database, not available in the authoring environment.)