docs: computation-model corrections + full make() reproducibility contract (incl. tripartite)

src/explanation/computation-model.md

@@ -41,6 +41,19 @@ The `make(self, key)` method:
 2. **Computes** results for that entity
 3. **Inserts** results into the table
 
+Those three steps are the basic mechanics. Beyond them, a well-behaved `make()`
+observes the full **make() reproducibility contract** — five rules that keep every
+result reproducible and `populate()` safely parallel:
+
+1. **Populate-only** — rows are produced only by `make()` through `populate()`, never inserted directly.
+2. **One entity per call, in isolation** — a `make(key)` computes exactly the entity named by `key` (plus its Part rows) and shares no state across calls.
+3. **Read only the upstream cone** — fetch only declared ancestors, restricted to the current `key` (exposed as `self.upstream`).
+4. **Write only to `self` and its Parts** — atomically, as one unit; any fan-out write elsewhere must record the source identity.
+5. **No other result-affecting input** — anything that changes *what* is computed must enter through a declared upstream table.
+
+The full contract — with rationale and the enforcement model — is specified in the
+[AutoPopulate reference §4.3, "The make() reproducibility contract"](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract).
+
 DataJoint guarantees:
 
 - `make()` is called once per upstream entity
@@ -50,8 +63,7 @@ DataJoint guarantees:
 ### Why the contract matters
 
 These guarantees hold because a well-behaved `make()` observes a small set of
-rules — the **make() reproducibility contract** (specified in full in the
-[AutoPopulate reference](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract)).
+rules — the **make() reproducibility contract** listed above.
 The organizing idea is a single **read/write boundary**: a `make(key)` reads only
 from its declared upstream dependencies, restricted to the current `key`, and
 writes only to `self` and its Part tables. Because each call sees a fixed,
@@ -63,9 +75,11 @@ of declared inputs — which is exactly what makes results reproducible and
 This boundary is why the auto-populated tiers split into two:
 
 - **Computed** tables derive entirely from other pipeline tables. Every input is
-  itself tracked under referential integrity, so a Computed result is
-  reproducible from within the pipeline alone — re-running `make()` on the same
-  upstream data yields the same result.
+  itself tracked under referential integrity, so a Computed result is fully
+  traceable within the pipeline — re-running `make()` derives it from the same
+  declared inputs. It is bitwise-identical only if the computation is
+  deterministic; stochastic computations are allowed (see the
+  [reproducibility contract](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract)).
 - **Imported** tables read a source the pipeline does *not* track (a file, an
   instrument, an API). They cannot be reproduced from the pipeline alone, so an
   Imported `make()` is responsible for recording the source's identity (path,
@@ -243,15 +257,19 @@ class SignalAverage(dj.Computed):
         raw_signal = (RawSignal & key).fetch1("signal")
         return (raw_signal,)
 
-    def make_compute(self, key, fetched):
-        """Step 2: Perform computation (outside transaction)"""
-        (raw_signal,) = fetched
+    def make_compute(self, key, raw_signal):
+        """Step 2: Perform computation (outside transaction).
+
+        The tuple returned by make_fetch is unpacked into positional args here.
+        """
         avg = raw_signal.mean()
         return (avg,)
 
-    def make_insert(self, key, fetched, computed):
-        """Step 3: Insert results (inside brief transaction)"""
-        (avg,) = computed
+    def make_insert(self, key, avg):
+        """Step 3: Insert results (inside brief transaction).
+
+        The tuple returned by make_compute is unpacked into positional args here.
+        """
         self.insert1({**key, "avg_signal": avg})
 ```
 
@@ -260,15 +278,14 @@ class SignalAverage(dj.Computed):
 DataJoint executes the three parts with verification:
 
 ```
-fetched = make_fetch(key)              # Outside transaction
-computed = make_compute(key, fetched)  # Outside transaction
+fetched = make_fetch(key)              # a tuple; outside transaction
+computed = make_compute(key, *fetched) # tuple unpacked; outside transaction
 
 <begin transaction>
-fetched_again = make_fetch(key)        # Re-fetch to verify
-if fetched != fetched_again:
-    <rollback>                         # Inputs changed—abort
+if make_fetch(key) != fetched:         # re-fetch and hash-verify inputs
+    <rollback>                         # inputs changed—abort
 else:
-    make_insert(key, fetched, computed)
+    make_insert(key, *computed)        # computed tuple unpacked
     <commit>
 ```
 
@@ -277,6 +294,44 @@ referential integrity is preserved by re-fetching and verifying inputs before
 insertion. If upstream data changed during computation, the job is cancelled
 rather than inserting inconsistent results.
 
+### Phase responsibilities
+
+**The simple rule to follow: `make_fetch` only fetches, `make_compute` only
+computes, and `make_insert` only inserts.** Keep each phase to its named job and
+your table is always within the contract — no further reasoning needed.
+
+These are part of the make() reproducibility contract: the framework does **not**
+enforce them at runtime — they are rules the pipeline author must follow, and
+against which a pipeline should be validated (at review or deploy time). The
+precise requirements are narrower than the one-job-per-phase rule above, which
+matters when a computation doesn't fit the clean split:
+
+- **`make_fetch(key)` must not insert, and must be bitwise reproducible.** It
+  fetches the entity's inputs (and may do *deterministic* computation), then
+  returns them. It runs *outside* the transaction and is re-run *inside* it, where
+  its output is **hash-verified** against the first call to catch inputs that
+  changed mid-computation — so the two calls must return byte-identical data, and
+  it must have no write side effects.
+- **`make_compute(key, *fetched)` must neither fetch nor insert, but need not be
+  deterministic.** It runs outside the transaction and depends only on the values
+  `make_fetch` returned (no database access). Because its output is inserted once
+  and never re-verified, it *may* use stochastic functions (random initialization,
+  sampling) — unlike `make_fetch`, it need not be bitwise reproducible.
+- **`make_insert(key, *computed)` inserts the result** into `self` and its
+  Part tables. It *always* runs inside the transaction, so it may additionally
+  fetch data or compute there — those reads and the write are covered by the same
+  transaction, so this does not break the model.
+
+So the two requirements the contract places on the split are **(a) `make_fetch`
+must not insert** and **(b) `make_compute` must neither fetch nor insert**
+(because it runs outside the transaction) — again, observed by the author and
+checked by validation, not by the runtime. The make() reproducibility contract
+still holds overall — reads come from the upstream cone and writes go only to
+`self` and its Parts. And
+because `make_fetch` performs no writes and `make_compute` touches no database,
+both can be called and tested **directly and safely** (see
+[Best Practices](#best-practices)).
+
 ### Benefits
 
 | Aspect | Standard `make()` | Three-Part Pattern |
@@ -319,28 +374,69 @@ def make(self, key):
         process_chunk(row['data'])
 ```
 
-### 3. Use Transactions for Multi-Row Inserts
+### 3. Don't Open a Transaction Inside `make()`
+
+`make()` already runs inside a transaction: `populate()` opens one per key before
+calling `make()` and commits it only if `make()` returns without error. Everything
+a `make()` inserts — multiple rows, and inserts into Part tables — is therefore
+already atomic. It all commits together, or rolls back together if `make()` raises.
+No explicit transaction is needed:
 
 ```python
 def make(self, key):
     results = compute_multiple_results(key)
+    # Already atomic — make() runs inside a transaction managed by populate()
+    self.insert(results)
+```
+
+Do **not** open your own transaction inside `make()`. DataJoint does not support
+nested transactions, so starting one while `make()`'s transaction is already active
+raises an error:
 
-    # All-or-nothing insertion
+```python
+def make(self, key):
+    # WRONG — a transaction is already in progress, so this raises an error
     with dj.conn().transaction:
-        self.insert(results)
+        self.insert(compute_multiple_results(key))
 ```
 
-### 4. Test with Single Keys First
+### 4. Test on One Entity with `populate()`, Not `make()` Directly
+
+To try a computation on a single entity, restrict `populate()` and cap the call
+count. This runs the entity through the real machinery — the per-key transaction,
+error handling, and (if enabled) job reservation:
 
 ```python
-# Test make() on one key
+# Compute just one pending entity, end-to-end
 key = (Scan - Segmentation).fetch1('KEY')
-Segmentation().make(key)
+Segmentation.populate(key, max_calls=1, display_progress=True)
+```
 
-# Then populate all
-Segmentation.populate()
+Do **not** call `make()` directly (e.g. `Segmentation().make(key)`) to test. It
+bypasses `populate()`: it runs **outside** the per-key transaction, so a partial
+or failed `make()` is not rolled back and can leave the table inconsistent; it
+also skips job reservation and error capture, and writes to the database as an
+uncontrolled side effect rather than as a managed, atomic unit.
+
+If your table uses the [three-part make](#the-three-part-make-model), you *can*
+test the fetch and compute steps directly and safely: `make_fetch(key)` performs
+no inserts and `make_compute(key, *fetched)` is pure, so neither writes to the
+database. Reserve `populate()` for exercising the insert step (`make_insert`):
+
+```python
+# Safe: neither call writes to the database
+fetched = MyTable().make_fetch(key)
+computed = MyTable().make_compute(key, *fetched)
+# Then exercise the full path (including make_insert) through populate()
+MyTable.populate(key, max_calls=1)
 ```
 
+!!! note "Future: a no-insert debug mode"
+    Calling `make()` directly could become a safe way to dry-run a computation
+    once DataJoint adds a dedicated test/debug mode that runs `make()` without
+    inserting. That is a planned future capability, not current behavior — today,
+    a direct `make()` call really does write to the database.
+
 ## Summary
 
 1. **`make(key)`** — Computes one entity at a time
@@ -349,3 +445,21 @@ Segmentation.populate()
 4. **Three-part make** — For long computations without long transactions
 5. **Cascade deletes** — Maintain workflow integrity
 6. **Error handling** — Robust retry mechanisms
+
+## See also
+
+**Specifications**
+
+- [AutoPopulate](../reference/specs/autopopulate.md) — normative spec for `key_source`, the [make() reproducibility contract](../reference/specs/autopopulate.md#43-the-make-reproducibility-contract), the tripartite pattern, and job reservation.
+- [Cascade](../reference/specs/cascade.md) — restriction propagation and master–part integrity for cascading deletes.
+- [Diagram](../reference/specs/diagram.md) — the dependency graph that `populate()` and `key_source` are computed from.
+
+**How-to guides**
+
+- [Run computations](../how-to/run-computations.md) — practical `populate()` usage, restrictions, and options.
+- [Distributed computing](../how-to/distributed-computing.md) — parallel and multi-worker populate with job reservation.
+
+**Related concepts**
+
+- [Relational workflow model](relational-workflow-model.md) — how computation fits DataJoint's data model.
+- [Data pipelines](data-pipelines.md) — the pipeline abstraction that auto-populated tables extend.

src/reference/specs/autopopulate.md

@@ -302,7 +302,7 @@ These requirements are the surface of a broader rule set — the make() reproduc
 
 ### 4.3 The make() reproducibility contract
 
-The value of an auto-populated table is that every one of its rows is a *reproducible* result: re-running `make()` on the same upstream data yields the same output. That guarantee rests on a small set of rules — the **make() reproducibility contract** — that every `make()` should observe. The rules apply to both `dj.Computed` and `dj.Imported` tables.
+The value of an auto-populated table is that every one of its rows is a *reproducible* result — **fully traceable to its declared inputs**: the row is derived solely from its declared, key-restricted upstream inputs plus the recorded `make()` code, with nothing entering from outside the dependency graph. That guarantee rests on a small set of rules — the **make() reproducibility contract** — that every `make()` should observe. The rules apply to both `dj.Computed` and `dj.Imported` tables.
 
 1. **Populate-only.** Rows are produced only by `make()`, invoked through `populate()` — never inserted directly. Direct inserts into an auto-populated table are rejected at runtime.
 
@@ -316,9 +316,14 @@ The value of an auto-populated table is that every one of its rows is a *reprodu
 
 In one line: **read from `self.upstream`, write to `self`.** The read/write boundary is what makes each computed result self-contained and reproducible — every row was produced by a specific `make()` call over a specific, declared, key-restricted set of upstream entities.
 
+**Tripartite extension.** When `make()` is written as the [tripartite pattern](#45-tripartite-make-pattern) (`make_fetch` / `make_compute` / `make_insert`), the contract extends to the phases: `make_fetch` must not insert, `make_compute` must neither fetch nor insert (it runs outside the transaction), and `make_insert` — which always runs inside the transaction — performs the write and may also fetch or compute. The simple convention is one job per phase; see [§4.5](#45-tripartite-make-pattern) for details.
+
 !!! note "How the contract is upheld"
     The contract has historically been a **convention** — observed by discipline, not enforced. DataJoint 2.3 adds an ergonomic read surface that makes it easy to follow and to inspect: `self.upstream` gives each `make()` its key-restricted upstream cone directly, and [`Diagram.trace`](diagram.md) walks the same ancestry for any row after the fact. The framework does not itself enforce the contract at runtime; observing it remains the pipeline author's responsibility.
 
+!!! note "Reproducibility means tracked derivation, not bitwise determinism"
+    The contract guarantees that every row is **derived only from its declared, key-restricted upstream inputs** — its derivation is fully tracked — not that `make()` is bitwise deterministic. **Stochastic computations are fully allowed.** Optimization and machine-learning algorithms are frequently non-deterministic, and DataJoint's model does not oppose this: re-running such a `make()` produces an equally valid, equally-traceable result, just not a byte-identical one. Bitwise reproducibility holds *only if the computation itself is reproducible* — for that, the pipeline author must control the sources of nondeterminism (fix random seeds, pin library versions, and so on). In the [tripartite pattern](#45-tripartite-make-pattern) this is why nondeterministic work belongs in `make_compute` (run once, never re-verified) and never in `make_fetch` (re-run and hash-verified, so it must be bitwise reproducible).
+
 ### 4.4 Accessing Source Data
 
 ```python
@@ -361,6 +366,14 @@ DataJoint 2.3 introduced `self.upstream` (and the underlying [`Diagram.trace`](d
 
 For long-running computations, use the tripartite pattern to separate fetch, compute, and insert phases. This enables better transaction management for jobs that take minutes or hours.
 
+**Phase contract.** The simple convention is one job per phase: `make_fetch` only fetches, `make_compute` only computes, and `make_insert` only inserts. The precise requirements the [make() reproducibility contract](#43-the-make-reproducibility-contract) places on the split — observed by the author and checked by validation, not enforced at runtime — are:
+
+- **`make_fetch(key)` must not insert, and must be bitwise reproducible.** It fetches the entity's inputs (and may perform *deterministic* computation), then returns them. It runs *outside* the transaction and is re-run *inside* it, where its output is **hash-verified** against the first call to detect inputs that changed mid-computation — so the two calls must return byte-identical data. It must also have no write side effects.
+- **`make_compute(key, *fetched)` must neither fetch nor insert, but need not be deterministic.** It runs outside the transaction and depends only on the values `make_fetch` returned (no database access). Because its output is inserted once and never re-verified, it *may* use stochastic functions (random initialization, sampling, non-deterministic solvers) — unlike `make_fetch`, its result need not be bitwise reproducible.
+- **`make_insert(key, *computed)` inserts the result** into `self` and its Part tables. It *always* runs inside the transaction, so it may additionally fetch or compute there without breaking the model.
+
+Because `make_fetch` performs no writes and `make_compute` touches no database, both can be called and tested directly and safely; only `make_insert` (and full `populate()`) writes.
+
 **Method-based tripartite:**
 
 ```python
@@ -373,19 +386,22 @@ class HeavyComputation(dj.Computed):
     """
 
     def make_fetch(self, key, **kwargs):
-        """Fetch all required data (runs in transaction).
+        """Fetch all required data (runs outside the transaction; re-run and
+        hash-verified inside it).
 
         kwargs are passed from populate(make_kwargs={...}).
+        Return a tuple — it is unpacked into make_compute's positional args.
         """
-        return (Recording & key).fetch1('raw_data')
+        return ((Recording & key).fetch1('raw_data'),)
 
     def make_compute(self, key, data):
         """Perform computation (runs outside transaction)."""
-        # Long-running computation - no database locks held
-        return heavy_algorithm(data)
+        # Long-running computation - no database locks held.
+        # Return a tuple — unpacked into make_insert's positional args.
+        return (heavy_algorithm(data),)
 
     def make_insert(self, key, result):
-        """Insert results (runs in transaction)."""
+        """Insert results (runs inside the transaction)."""
         self.insert1({**key, 'result': result})
 ```