First-class ingestion lineage, separate from the FK graph

[dimitri-yatsenko]

Summary

Roadmap capability from the strict-provenance adoption analysis (#1474 discussion, @ttngu207): a first-class record of where data came from, kept separate from the FK graph.

Why

DataJoint has one graph, used for two different things:

• what is derived from what — durable, invariant, belongs in the FK graph;
• how a row arrived — transient, operational (a file today, an API next quarter, a sync after that).

Forcing the second into the FK graph (the "landing-table" pattern: entities downstream of a ParsedFile ancestor) marries the slow-moving domain model to fast-moving infrastructure — every ingestion change becomes a schema migration, and historical rows permanently carry whichever ingestion ancestor existed at creation time. That is why the strict-provenance write boundary has no clean answer for the common fat ingestion make() (one file parsed into many top-level entity tables): every compliant restructuring corrupts the domain model or abandons DataJoint orchestration.

Shape

• Append-only ingestion-lineage records, per row or per insert batch: (entity key, source descriptor, timestamp, actor) — multiple sources over an entity's lifetime, zero schema changes.
• Queryable alongside (not through) the FK graph; surfaced in row-lineage views.
• Candidate capture points: the gated insert path (already intercepts every write under strict mode), staged_insert1, and populate job metadata.

Related use cases to fold in

• Spyglass Exports (discussion Cascading Restrictions #1232, @CBroz1): monitor fetch/restrict/join calls, log accessed leaves, drive an upward trace for mysqldump/DANDI publication — the same "record what was touched, separate from the FK graph" mechanism from the read side.
• Platform row-lineage / compute-provenance work (branch identity ↔ per-row code-version tagging) — ingestion lineage is the missing third leg beside code-version and FK derivation.

[dimitri-yatsenko]

Decision: do not implement — this is outside the domain of DataJoint's core framework.

Recording how a row arrived — the external source a value came from and the operational metadata around ingestion — is provenance about data entering the pipeline from outside its derivation graph. A first-class, separate ingestion-lineage store as proposed here is effectively an extrinsic provenance system, and DataJoint's core deliberately stays neutral on that. Many pipelines already run alongside a dedicated provenance/governance system that captures this in its own terms.

Within DataJoint, the accepted approach is the fan-out ingestion pattern: one ingestion make() may populate several Manual/Imported entry-point tables that carry no FK dependency on the ingesting table, and each records its own source identity — exactly as at any manual data-entry point. The origin record lives at the entry-point tables, not in a separate log the framework maintains.

The distinction between DataJoint's structural, within-pipeline lineage and dedicated provenance systems — and the fact that capturing external-origin provenance is left to the systems built for it — is documented in the new Comparison to Provenance Systems explainer (datajoint/datajoint-docs#204).

Closing as not planned. The read-side Spyglass Exports use case (#1232) is a separate mechanism and remains tracked there.

[dimitri-yatsenko]

The accepted alternative is now documented: Fan-Out Ingestion — one ingestion step populating several entry-point tables that carry no FK back to the loader, each recording its own origin. It goes live when datajoint/datajoint-docs#204 merges.

[CBroz1]

Thanks for writing this up. The core-neutrality argument for keeping extrinsic provenance out of the framework is reasonable. A few things come to mind when I think about the match between fan-out ingestion and the Export use case.

1. Retrofit -> schema migration. Fan-out applies at ingestion time. Existing databases would need restructuring and re-ingest to adopt it. I don't foresee Spyglass doing that anytime soon, as much as I would like to revise core tables.

2. Axis. Fan-out groups by ingestion source; an Export is scoped by project, a subset of downstream rows. FAIR data reuse means that the same file routinely feeds multiple projects, so project membership can't be derived from origin. For us, it's read-side monitoring of user figure creation.

If Export is ever to be targeted for inclusion in DataJoint, it might need to do the read-side monitoring, or allow the unioning of multiple upstream restriction cascades.