Extend Python support to 3.14; test min/max in CI#1490
Conversation
datajoint-python declared support for 3.10–3.13 but CI (a single pixi environment) only exercised 3.13, and 3.14 — released 2025-10 — was excluded. This closes the ceiling gap and makes the declared range actually tested. - pyproject.toml: bump requires-python and the pixi python dependency to ">=3.10,<3.15" (adds 3.14); add per-version PyPI classifiers for 3.10–3.14. - pyproject.toml: add version-pinned pixi test environments (test-py310, test-py314) so CI can run the suite on both ends of the range. Each solves independently, since a solve-group cannot span Python versions. - .github/workflows/test.yaml: run the containerized test job as a matrix over [test-py310, test-py314]. Note: 3.10 reaches end-of-life 2026-10; dropping it (min → 3.11) is a follow-up, kept out of this change to avoid a breaking bump.
MilagrosMarin
left a comment
MilagrosMarin
left a comment
There was a problem hiding this comment.
Thanks @dimitri-yatsenko — nice audit catch. The EOL alignment and pixi feature-based envs are exactly right, and splitting 3.10's eventual drop as a follow-up (given its 2026-10 EOL) keeps this PR clean.
Two small observations, neither blocking:
Middle versions are unexercised. Testing 3.10 and 3.14 covers the endpoints where compat issues usually cluster, but 3.11–3.13 aren't touched. Reasonable trade-off for CI-time — just worth being explicit that a middle-version-specific regression would land with users first.
The unit-tests job leans on pixi's solver picking the highest Python. PR body says it "resolves to 3.14," which is pixi's default behavior today, but the reliance is implicit. If a future pixi version shifts its default strategy, unit-tests could quietly slide to a lower version. Adding an explicit py314 pin to that env would make the intent enforced rather than emergent.
Approving.
test_insecure_connection asserted Ssl_cipher == '', but a MySQL 8 server with TLS configured (the datajoint/mysql:8.0 test image) can negotiate TLS during connection setup even when the client requests no SSL. Whether the cipher ends up empty depends on the resolved client stack, so the assertion passed on the py3.13 solve but failed on the py3.10/py3.14 matrix added in this PR. Assert what DataJoint actually controls: use_tls=False sends no client-side SSL config and the connection is usable.
Parallel populate pickles the table (and its connection's Dependencies) to worker processes, which then reload dependencies and reconnect. Dependencies held an `itertools.count` (_node_alias_count) that is not picklable. This was latent on Linux through Python 3.13 because multiprocessing defaulted to `fork` (which inherits rather than pickles); Python 3.14 defaults to `forkserver`, which pickles the initializer args and surfaced the failure (`TypeError: cannot pickle 'itertools.count' object` in test_multi_processing). Add __getstate__/__setstate__ to drop the counter on pickle and rebuild it on load — workers reload dependencies anyway, so the value need not be carried.
|
Re-requesting review — two commits landed after the earlier approval, both fixing CI failures the new py3.10/py3.14 matrix surfaced (no change to the pyproject/CI matrix that was already reviewed):
py3.10 is green; py3.14 is still running. Flagging @MilagrosMarin 's earlier suggestion (explicit |
The real cause of the py3.14 failure is Python 3.14 changing the multiprocessing default start method from fork to forkserver. Parallel populate hands the live table and its open DB connection to workers by process inheritance (fork); under forkserver the payload is pickled instead, which cannot carry a live connection and deadlocked the job (after first surfacing as 'cannot pickle itertools.count'). Pin both Pool call sites to a fork context (fork where available, platform default otherwise). Reverts the earlier Dependencies __getstate__/__setstate__ pickle workaround, which addressed only the symptom. py3.10 (fork default) already passed; this makes py3.14 use the same proven path.
2 participants
Motivation
An audit of supported Python versions against the official EOL schedule found two gaps:
>=3.10) ✅Two problems: (1) the ceiling lagged a full release behind — 3.14 has been out since Oct 2025 but
requires-python = ">=3.10,<3.14"blocked it; and (2) although the package claimed 3.10–3.13, the pixi-based test job resolved a single interpreter (3.13), so the declared floor was never exercised.Changes
pyproject.tomlrequires-pythonand the pixipythondependency →>=3.10,<3.15(adds 3.14).Programming Language :: Python), so the supported range is discoverable on PyPI.test-py310andtest-py314(each solves independently — a solve-group can't span Python versions)..github/workflows/test.yaml— run the containerized test job as a matrix over[test-py310, test-py314], so both ends of the supported range are validated on every push/PR.Fixes surfaced by the 3.14 matrix
Running the full suite on 3.14 exposed two pre-existing issues (latent because CI only ever ran 3.13); both are fixed here:
test(tls)—test_insecure_connectionassertedSsl_cipher == "", but thedatajoint/mysql:8.0test image negotiates TLS during connection setup even when the client requests none, so the cipher is not reliably empty. Now pins what DataJoint controls (use_tls=Falsesends no client-side SSL config; the connection is usable) rather than a server-negotiated outcome.fix(populate)—test_multi_processinghung on 3.14. Python 3.14 changed the multiprocessing default start method fromforktoforkserver; parallel populate hands the live table and its open DB connection to workers by process inheritance (fork), whichforkserverbreaks by pickling the payload instead. BothPoolcall sites are now pinned to aforkcontext.Notes / follow-ups
target-version/mypypython_version) is a small breaking bump left as a follow-up rather than folded in here.unit-testsjob (fast feedback) resolves to the highest supported version (3.14); the full matrix covers min+max. (Reviewer suggestion: make this an explicitpy314pin — deferred as a follow-up.)testmatrix job has notimeout-minutes, so the 3.14 hang ran ~1h before it was cancelled during review — worth adding a job timeout as a follow-up so future hangs fail fast.pixi.lockis re-solved in CI (locked: false), so no committed lock update is needed; the PR's own CI run validates that all dependencies resolve on both 3.10 and 3.14.Verification
test-py310,test-py314) pass, along withlintandunit-tests.