Skip to content

Add a complete, executed worked example of the tripartite make() #205

Description

@dimitri-yatsenko

Summary

Add a complete, executed worked example of the three-part (tripartite) makemake_fetch / make_compute / make_insert — as a tutorial. Today the pattern is documented in prose (Computation Model, autopopulate.md §4.5) but there is no end-to-end, runnable example.

Why

  • The calling convention is a footgun. The framework invokes make_compute(key, *fetched_data) and make_insert(key, *computed_result) (the returned tuples are unpacked; make_insert receives only the computed result, not fetched). This is easy to get wrong — the docs themselves shipped an incorrect example recently. A copy-pasteable, correct template is the best antidote.
  • The semantics are subtle. make_fetch runs outside the transaction and is re-run and hash-verified inside it (so it must be bitwise-reproducible); make_compute runs once and is never re-verified (so it may be stochastic); make_insert runs inside the transaction. These boundaries are hard to grasp abstractly and land better as concrete, running code.
  • Real need. It's the recommended pattern for long-running computations (minutes–hours) that can't hold a transaction open — a common ML / signal-processing case.

What it should show

  • All three phases with the correct signatures and tuple return/unpack convention.
  • The re-fetch/verify behavior, ideally including the failure mode: inputs change mid-compute → rollback, no insert.
  • A make_compute that is legitimately non-deterministic (e.g. a seeded vs. unseeded stochastic step), to make "reproducibility = tracked derivation, not bitwise determinism" concrete.
  • How populate(make_kwargs=...) forwards to make_fetch.

Recommended form

  • Executed, not prose — a tutorial notebook run in CI (or a doctest), so it stays correct as the API evolves. A non-executed snippet would drift out of sync again (exactly the failure we just fixed).
  • Placement: Tutorials → Advanced (e.g. "Long computations with the three-part make"), or How-To → Computation. Pair with, and cross-link, autopopulate.md §4.5 and the Computation Model.

Notes

  • Scope/reach is modest (most pipelines use plain make(); tripartite is advanced), but the value for its audience is high and the reliability win — a tested example that can't drift — is the main argument.
  • Simulating a genuinely "long" computation cheaply in CI is the one awkwardness; a trivial compute with a comment noting where the real work would go is acceptable.

Targeted for a future release, not v2.3.1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions