Retire strict_provenance docs; drop provenance terminology#199
Conversation
Match datajoint-python's removal of the strict_provenance runtime guardrail, and align the docs with the decision to keep provenance terminology out of DataJoint's own documentation. DataJoint implements the relational workflow model — data integrity, consistency, lineage, and reproducibility; provenance is a platform-level compatibility concern documented separately. - Rename reference/specs/provenance.md -> trace.md; rewrite it as the spec for the upstream read surface (Diagram.trace + self.upstream), removing the strict_provenance section entirely and reframing the "read/write boundary" in lineage/reproducibility terms. - about/whats-new-23.md: reframe the "provenance trinity" as the trace + self.upstream pair; drop the strict_provenance subsection and adoption step; retarget links to trace.md and the make() contract. - mkdocs.yaml: nav "Provenance Trace & Strict Provenance" -> "Upstream Trace". - Sweep "provenance" from the conceptual and tutorial pages (index, about/index, relational-workflow-model, normalization, comparison-to-workflow-languages, schema-as-workflow-specification, data-pipelines, faq, and three tutorial notebooks), replacing it with "lineage"/"reproducibility"/"traceability" as fits each context.
Add mkdocs-redirects so reference/specs/provenance/ (renamed to reference/specs/trace/ when the provenance framing was retired) keeps resolving instead of 404-ing.
The strict_provenance runtime guardrail is being retired (datajoint-python #1489), and its spec page is removed in #199 (deletes provenance.md, adds trace.md). Drop this PR's strict_provenance content so it stays coherent: - remove the 'Provenance-safe make() (strict provenance)' how-to section (run-computations.md) - revert the strict_provenance enforcement-limit edits (provenance.md, deleted by #199) - reword the autopopulate.md upstream-only note to drop the runtime-flag claim Retain the cascade U3/limitations, Diagram.trace() docs, and self.upstream section.
MilagrosMarin
left a comment
MilagrosMarin
left a comment
There was a problem hiding this comment.
Thanks @dimitri-yatsenko — nicely-scoped counterpart to datajoint-python#1489. Verified end-to-end.
The retirement side is clean: provenance.md (441 lines, the strict_provenance spec) becomes trace.md (301 lines, focused on the Diagram.trace + self.upstream pair). grep for strict_provenance, assert_read_allowed, assert_write_allowed, contextvars.ContextVar, and "Enforcement model" in the new file all return empty — the flag content is fully purged, not just relabeled. The what's-new-23.md reframe from "provenance trinity" to "trace + self.upstream pair" drops the "Enable strict_provenance=True in staging" migration step cleanly. mkdocs.yaml nav renames "Provenance Trace & Strict Provenance" → "Upstream Trace" and adds the redirects plugin (with pip_requirements.txt picking up mkdocs-redirects) so the old /specs/provenance/ URL doesn't 404. Cross-link to autopopulate.md#43-the-make-reproducibility-contract resolves against the anchor introduced in the stacked #198.
The terminology sweep across the conceptual pages reads well. "Provenance" mostly becomes "lineage" in contexts about tracing derivation ("Queries trace provenance" → "Queries trace lineage", "Rows carrying provenance to their inputs" → "traceable to their inputs"), and "provenance and lineage" (previously redundant) collapses to "lineage and reproducibility". Section labels in the Codd/Chen/RWM comparison table follow: "Provenance" → "Data lineage". Consistent choice per context.
One residual worth calling out: grep -rni "provenance" src/ returns one hit — src/images/dj-platform.svg:155 still lists "Reproducibility • Provenance • Integrity • Collaboration" in the platform-architecture text label. This is defensible under the PR's own framing (the platform's provenance-compatibility features live separately in datajoint-provenance), and the SVG is describing platform-side capabilities rather than framework docs. But the PR body's claim "no 'provenance' terminology remains in tracked source (src/)" is a slight overclaim — the SVG is tracked source. Either update the SVG to match the doc-side terminology reset, or tighten the PR-body claim to note the platform-diagram exception. Small.
Sequencing note (already flagged in the PR body): stacked on #198. Merge #198 first; GitHub retargets this one to main automatically. Verified this PR's diff isolates cleanly against upstream/docs/197-make-contract — 16 files, 341 insertions, 496 deletions.
Approving.
|
Thanks @MilagrosMarin for the thorough end-to-end review — especially catching the |
Re-applies the changes from #199, which was marked merged but never landed on main: #198 and #199 were merged seconds apart, and #199's squash was computed against a stale base (main before #198 propagated), so GitHub could not advance main to it and the retire-provenance changes were silently dropped. This branch re-applies exactly #199's 16-file change set on top of the current main (which now includes #198): deletes provenance.md, adds trace.md, and completes the terminology sweep.
…s-refs (#195) * docs: post-audit reconciliation — provenance limits, cascade U3/limitations, trace cross-refs Follow-up to the comprehensive 2.3 post-release audit of part integrity, provenance, and cascade/trace/upstream across code and docs. provenance.md: - Enforcement limits: add ungated deletes; extend the existence/count-idiom bullet to Aggregation/Union results. - Write enforcement: update1 rows (gated like insert; exact error messages). - Key consistency: INSERT ... SELECT server-side exception; update1 mismatch message. - §2: own Parts not reachable via self.upstream (read as self.PartName); merge/master-part boundary note (trace does not descend into an ancestor Master's Parts). - Config wording: flag read from connection._config; soften 'provenance guarantee' to 'provenance model'. cascade.md: - U3 corrected to child.proj(*attr_map.keys()) (and Worked Example 1 step 3). - F1/U1 triggers: non-empty attribute set required; empty set takes rule 3. - New Limitations section: single-FK-path upward walk, materialization memory cost (preview pays it too), empty-match sentinel, enforce post-check table-level granularity. - Refresh stale trace paragraph: trace shipped in 2.3, reuses U1/U2/U3. diagram.md: - Add Diagram.trace() section cross-linking provenance.md §1. - Correct 'enforce' preview claim (preview never errors; enforcement is Table.delete()'s post-check); note part_integrity value validation. autopopulate.md: - De-stale §4.3 ('a future release may enforce' -> strict_provenance shipped in 2.3, opt-in). - New §4.3.1 documenting self.upstream (lifecycle, laziness, tripartite availability, ancestors-only scope). run-computations.md: - New task-oriented section: provenance-safe make() — self.upstream usage, staged strict_provenance rollout, what the guardrail does/doesn't catch. whats-new-23.md: 'provenance guarantee' -> 'provenance model'. * docs: drop strict_provenance prose, keep cascade + trace reconciliation The strict_provenance runtime guardrail is being retired (datajoint-python #1489), and its spec page is removed in #199 (deletes provenance.md, adds trace.md). Drop this PR's strict_provenance content so it stays coherent: - remove the 'Provenance-safe make() (strict provenance)' how-to section (run-computations.md) - revert the strict_provenance enforcement-limit edits (provenance.md, deleted by #199) - reword the autopopulate.md upstream-only note to drop the runtime-flag claim Retain the cascade U3/limitations, Diagram.trace() docs, and self.upstream section.
2 participants
Documentation counterpart to datajoint/datajoint-python#1489 (retire the
strict_provenanceruntime guardrail), and alignment with the decision to keep provenance terminology out of DataJoint's own documentation.DataJoint implements the relational workflow model — data integrity, consistency, lineage, and reproducibility. "Provenance" is associated with a different class of systems; DataJoint's provenance compatibility features live in the platform and are documented separately (
datajoint-provenance). So the public docs use the model's native vocabulary: lineage, reproducibility, traceability.Changes
Retire
strict_provenance:reference/specs/provenance.md→reference/specs/trace.mdand rewrite it as the spec for the upstream read surface (Diagram.trace+self.upstream) — thestrict_provenancesection (§3), config docs, examples, and references are removed; the "read/write boundary" is reframed in lineage/reproducibility terms.about/whats-new-23.md: reframe the "provenance trinity" as thetrace+self.upstreampair, drop thestrict_provenancesubsection and adoption step, retarget links.mkdocs.yaml: nav "Provenance Trace & Strict Provenance" → "Upstream Trace".Drop "provenance" terminology across the conceptual and tutorial pages (
index,about/index,relational-workflow-model,normalization,comparison-to-workflow-languages,schema-as-workflow-specification,data-pipelines,faq, and three tutorial notebooks) — replaced with "lineage" / "reproducibility" / "traceability" per context.Verification
grepconfirms no "provenance" terminology remains in the framework documentation, and no links point to the oldspecs/provenance.mdpath. The sole remaining occurrence undersrc/is the platform-architecture label inimages/dj-platform.svg("Reproducibility • Provenance • Integrity • Collaboration") — intentionally kept: it describes the DataJoint platform, which builds provenance capabilities, whereas the framework docs describe the open-source core's relational workflow model, where "lineage" is the apt term.llms-full.txtis a generated, untracked build artifact — regenerated on build.)Published URL
reference/specs/provenance/becomesreference/specs/trace/. This PR addsmkdocs-redirects(and themkdocs-redirectsbuild dependency) so the old URL redirects to the new page instead of 404-ing.