From cd90aa337177676c89892ee0c624b5b1ca8c319d Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Wed, 15 Jul 2026 11:16:45 -0500 Subject: [PATCH] docs: fold 2.3.1 changes into What's New in 2.3; fix cross-link gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - whats-new-23.md: add a 'Changes in 2.3.1' section (strict_provenance removed, Python 3.14 support, codec-driven GC reference-discovery fix) so 2.3.0->2.3.1 upgraders learn the flag is gone and the range extended. - autopopulate.md: link the fan-out mention to the Fan-Out Ingestion page. - computation-model.md: add Upstream Trace, Fan-Out Ingestion, and Comparison to Provenance Systems to 'See also'. Does not touch mkdocs datajoint_version (still '2.2') — bumping it is gated on re-executing the tutorial notebooks (the version-banner guard), a release step. --- src/about/whats-new-23.md | 8 ++++++++ src/explanation/computation-model.md | 3 +++ src/reference/specs/autopopulate.md | 2 +- 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/src/about/whats-new-23.md b/src/about/whats-new-23.md index 73403422..123d6fa2 100644 --- a/src/about/whats-new-23.md +++ b/src/about/whats-new-23.md @@ -6,6 +6,14 @@ DataJoint 2.3 adds a first-class **upstream read surface** — `Diagram.trace` a > **Citation:** Yatsenko D, Nguyen TT. *DataJoint 2.0: A Computational Substrate for Agentic Scientific Workflows.* arXiv:2602.16585. 2026. [doi:10.48550/arXiv.2602.16585](https://doi.org/10.48550/arXiv.2602.16585) +## Changes in 2.3.1 + +2.3.1 is a patch release on the 2.3 line. If you are upgrading from **2.3.0**: + +- **`strict_provenance` removed.** The opt-in `dj.config["strict_provenance"]` runtime guardrail that shipped in 2.3.0 has been retired. Comprehensively checking the `make()` read/write contract across every access path is a code-inspection problem rather than a runtime one, so it is validated at review or deploy time rather than by an in-process flag. The flag was off by default with no known adoption, so existing pipelines are unaffected. The ergonomic read surface — `Diagram.trace` and `self.upstream` — is unchanged, and the rules are documented as the [make() reproducibility contract](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract). +- **Python 3.14 support.** `requires-python` now spans `>=3.10,<3.15`; CI exercises both ends of the range (3.10 and 3.14). +- **Garbage-collection fix.** `gc.collect()` now discovers codec-referenced files correctly, closing a path where live custom-codec and schema-addressed object-store files (``, ``) could be misclassified as orphans and deleted. See [Clean Up Object Storage](../how-to/garbage-collection.md). + ## Overview A computed row is reproducible only when `make(self, key)` reads only from its declared upstream dependencies and writes only to `self` (and its Parts). DataJoint 2.3 makes that read/write boundary easy to follow — and the resulting data lineage easy to query — with two features designed as a unit: diff --git a/src/explanation/computation-model.md b/src/explanation/computation-model.md index 1f1dd1fd..0c7440c7 100644 --- a/src/explanation/computation-model.md +++ b/src/explanation/computation-model.md @@ -451,6 +451,7 @@ MyTable.populate(key, max_calls=1) **Specifications** - [AutoPopulate](../reference/specs/autopopulate.md) — normative spec for `key_source`, the [make() reproducibility contract](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract), the tripartite pattern, and job reservation. +- [Upstream Trace](../reference/specs/trace.md) — `Diagram.trace` and `self.upstream`, the read surface for declared upstream inside `make()`. - [Cascade](../reference/specs/cascade.md) — restriction propagation and master–part integrity for cascading deletes. - [Diagram](../reference/specs/diagram.md) — the dependency graph that `populate()` and `key_source` are computed from. @@ -463,3 +464,5 @@ MyTable.populate(key, max_calls=1) - [Relational workflow model](relational-workflow-model.md) — how computation fits DataJoint's data model. - [Data pipelines](data-pipelines.md) — the pipeline abstraction that auto-populated tables extend. +- [Fan-out ingestion](fan-out-ingestion.md) — one ingestion step populating several entry-point tables, and where origin is recorded. +- [Comparison to provenance systems](comparison-to-provenance-systems.md) — how DataJoint's structural lineage relates to dedicated provenance tooling. diff --git a/src/reference/specs/autopopulate.md b/src/reference/specs/autopopulate.md index 23fcc7cd..7ae06b2f 100644 --- a/src/reference/specs/autopopulate.md +++ b/src/reference/specs/autopopulate.md @@ -310,7 +310,7 @@ The value of an auto-populated table is that every one of its rows is a *reprodu 3. **Read only upstream, restricted to the job.** A `make(key)` fetches only from tables upstream of the target — the declared ancestors reachable by following the foreign keys embedded in the primary key — and only the rows reachable from the current `key`. This key-restricted set of ancestors is the entity's *upstream cone*, exposed as `self.upstream` (see [§4.4](#44-accessing-source-data)). Reading anything else — a table that is not an ancestor, or entities under a different `key` — breaks the contract and makes the result depend on data the pipeline does not track. -4. **Write the result to `self` and its Parts.** The derived result is inserted into `self` and the table's Part tables, atomically, as one unit. A `make()` *may* also write to other destinations — other Manual tables, files, external systems (the *fan-out* pattern) — but such writes leave the pipeline's dependency graph, so the `make()` must record the source identity at each destination (as it would at any Manual-table entry point) to keep that data traceable. +4. **Write the result to `self` and its Parts.** The derived result is inserted into `self` and the table's Part tables, atomically, as one unit. A `make()` *may* also write to other destinations — other Manual tables, files, external systems (the [fan-out ingestion pattern](../../explanation/fan-out-ingestion.md)) — but such writes leave the pipeline's dependency graph, so the `make()` must record the source identity at each destination (as it would at any Manual-table entry point) to keep that data traceable. 5. **No other result-affecting input.** Anything that changes *what* is computed must enter through a declared upstream table (typically a Lookup or Manual parameter table), so it is part of the dependency graph. `make_kwargs` forwarded through `populate(make_kwargs=...)` may *orchestrate* a run (batching, parallelism, logging) but must never *parameterize* the result — see [§4.6](#46-additional-make-arguments).