docs: post-audit reconciliation — cascade U3/limitations + trace cross-refs#195
Conversation
…ations, trace cross-refs
Follow-up to the comprehensive 2.3 post-release audit of part integrity,
provenance, and cascade/trace/upstream across code and docs.
provenance.md:
- Enforcement limits: add ungated deletes; extend the existence/count-idiom
bullet to Aggregation/Union results.
- Write enforcement: update1 rows (gated like insert; exact error messages).
- Key consistency: INSERT ... SELECT server-side exception; update1 mismatch
message.
- §2: own Parts not reachable via self.upstream (read as self.PartName);
merge/master-part boundary note (trace does not descend into an ancestor
Master's Parts).
- Config wording: flag read from connection._config; soften 'provenance
guarantee' to 'provenance model'.
cascade.md:
- U3 corrected to child.proj(*attr_map.keys()) (and Worked Example 1 step 3).
- F1/U1 triggers: non-empty attribute set required; empty set takes rule 3.
- New Limitations section: single-FK-path upward walk, materialization memory
cost (preview pays it too), empty-match sentinel, enforce post-check
table-level granularity.
- Refresh stale trace paragraph: trace shipped in 2.3, reuses U1/U2/U3.
diagram.md:
- Add Diagram.trace() section cross-linking provenance.md §1.
- Correct 'enforce' preview claim (preview never errors; enforcement is
Table.delete()'s post-check); note part_integrity value validation.
autopopulate.md:
- De-stale §4.3 ('a future release may enforce' -> strict_provenance shipped
in 2.3, opt-in).
- New §4.3.1 documenting self.upstream (lifecycle, laziness, tripartite
availability, ancestors-only scope).
run-computations.md:
- New task-oriented section: provenance-safe make() — self.upstream usage,
staged strict_provenance rollout, what the guardrail does/doesn't catch.
whats-new-23.md: 'provenance guarantee' -> 'provenance model'.
The strict_provenance runtime guardrail is being retired (datajoint-python #1489), and its spec page is removed in #199 (deletes provenance.md, adds trace.md). Drop this PR's strict_provenance content so it stays coherent: - remove the 'Provenance-safe make() (strict provenance)' how-to section (run-computations.md) - revert the strict_provenance enforcement-limit edits (provenance.md, deleted by #199) - reword the autopopulate.md upstream-only note to drop the runtime-flag claim Retain the cascade U3/limitations, Diagram.trace() docs, and self.upstream section.
dimitri-yatsenko
changed the title
MilagrosMarin
left a comment
MilagrosMarin
left a comment
There was a problem hiding this comment.
Thanks @dimitri-yatsenko — good post-audit reconciliation. Noting 324e64e already dropped the strict_provenance prose (matching your review-push note on the batch), so this PR is now the cascade/trace reconciliation part only. Verified.
The U3 correction matches shipped code exactly (src/datajoint/diagram.py: parent_item = child_ft.proj(*attr_map.keys())), and Worked Example 1's walk-through now reflects it. The F1/U1 "non-empty" precondition closes a gap I would have flagged: the old rule table left the empty-attrs case ambiguous; the "empty attribute set takes rule 3" note resolves it.
New Limitations section is honest and matches shipped behavior: single FK path (nx.shortest_path picking one arm), materialization cost bounded by distinct master rows (same reason preview pays the cost too), empty-match sentinel (matches the [False] branch in _propagate_part_to_master), and enforce granularity (table-level, not row-level — with the false-negative/false-positive framing spelled out). The part_integrity="enforce" table row now cross-references it.
diagram.md — good correction: "Error if parts would be deleted before masters" was misleading (the preview itself never errors) and now correctly attributes enforcement to Table.delete()'s post-check. Diagram.trace() documented alongside cascade() with the same shape. autopopulate.md §4.3.1 covers the four self.upstream semantic properties (lifecycle, laziness, tripartite availability, ancestors-only scope) matching what I verified when reviewing #1473.
Sequencing note — worth confirming beyond the PR body's own flag. The retained cascade.md / diagram.md / autopopulate.md references to provenance.md will resolve at runtime via #199's redirect (the anchor slugs #1-diagramtracetable_expr and #2-selfupstream-inside-make survive the rename since trace.md retains those section headers). But mkdocs build --strict resolves links against source-tree file paths, not redirects — so the provenance.md references would fail linkcheck the moment #199 merges. Cleanest path is to repoint the three refs to trace.md as part of merging #199, or land them simultaneously. Either way, the flag in the PR body is on the money.
Approving.
…nance-cascade # Conflicts: # src/reference/specs/autopopulate.md
…de' into docs/post-audit-provenance-cascade
…nance-cascade # Conflicts: # src/about/whats-new-23.md
2 participants
Post-2.3 audit reconciliation for the cascade and trace specs.
Contents
cascade.md— corrects the U3 upward rule (child.proj(*attr_map.keys())), notes the enforce post-check is table-level, and adds a Limitations section (single FK path, materialization cost, empty-match sentinel, enforce granularity).diagram.md— documentsDiagram.trace()(the upstream mirror ofcascade()) and thepart_integritypreview semantics.autopopulate.md— adds the §4.3.1self.upstream(pre-restricted ancestor access) section.whats-new-23.md— minor wording.Scope note
Originally this PR also documented the
strict_provenanceruntime guardrail (a how-to section inrun-computations.mdand enforcement-limit edits inprovenance.md). That guardrail is being retired in datajoint-python #1489, and its spec page is removed in #199 (which deletesprovenance.mdand addstrace.md). Those parts have been dropped here to stay coherent.cascade.md/diagram.md/autopopulate.mdstill cross-link toprovenance.md(#1-diagramtracetable_expr,#2-selfupstream-inside-make). Those anchors resolve againstmaintoday, but #199 deletesprovenance.mdin favor oftrace.md— so after #199 merges these links must be repointed totrace.md. Recommend landing #199 first (or repointing as part of it).