feat: uv python env + DREDge paramset seeds#2
Open
tabedzki wants to merge 2 commits into
Open
Conversation
getPythonEnv resolved the EnvAutoPipeGUI conda env by calling `conda env list` and slicing the raw output between the 1st and 2nd occurrence of the env name. That assumed the name appears exactly twice, which breaks when a name is a prefix of another (iblenv vs iblenv2) and on newer conda's table output format. Replace it with `uv run --project <repo> python`, so uv resolves and syncs the environment declared in pyproject.toml on demand. MATLAB no longer needs to know where the interpreter lives. uv is located across PATH and the standard install dirs on Windows and macOS/Linux, and bootstrapped via Astral's installer if absent. py_env is now quoted, so a repo path containing spaces survives system(). The IBL atlas GUI still uses conda; that env is not owned by this repo, but its lookup now matches the Name column instead of slicing raw output. Pin datajoint<2.0: 2.x renamed the config file to datajoint.json and ignores the dj_local_conf.json this repo ships, so it cannot connect. Also fix read_params.py, which crashed before writing any .mat file: savemat cannot serialize None, and the suite2p imaging paramset (idx 6) has nulls nested in its params blob. Because both modalities are savemat'd together, one bad imaging row broke params.mat for electrophysiology too, making fillParams fail. Coerce None -> [] and stop shadowing the `dict` builtin with the loop variable. Assisted-by: ClaudeCode:claude-opus-4.8
Seed JSON payloads for registering DREDge as a preprocessing method (alongside
CatGT) and a Kilosort4 paramset with internal drift correction disabled, so KS4
does not re-correct motion after DREDge.
- preprocess_paramset_dredge.json: {preset:"dredge", device, motion_kwargs,
job_kwargs} — matches the run_dredge consumer in BrainCogsEphysSorters.
- process_paramset_kilosort4_no_drift.json: nblocks=0 (the actual KS4 drift
off-switch); do_correction kept as an inert, KS2/KS3-friendly key.
Registered via the existing upload-paramset form; no MATLAB UI change is needed
(method dropdowns populate dynamically from DataJoint).
Companion to BrainCOGS/BrainCogsEphysSorters#5 and BrainCOGS/U19-pipeline-python#94.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Summary
Two related pieces of the DREDge / uv rollout for the ephys pipeline:
uvinstead of a named conda env (existing commits on this branch).Paramset seeds (
Original_Params_DB/)preprocess_paramset_dredge.json—{preset:"dredge", device, motion_kwargs, job_kwargs}, matching therun_dredgeconsumer in feat: DREDge motion correction (CatGT→DREDge→KS4) + uv/py3.14 env BrainCogsEphysSorters#5.process_paramset_kilosort4_no_drift.json—nblocks=0(the real KS4 drift off-switch);do_correctionkept as an inert, KS2/KS3-friendly key.These register through the existing upload-paramset form — no MATLAB UI change is needed, because the method dropdowns populate dynamically from DataJoint. To enable DREDge for a recording: insert a
PreClusterMethod('dredge')row + aPreClusterParamSetfrom the dredge seed, then order the step list catgt → dredge.Linkage
Relates to BrainCOGS/U19-pipeline-python#53 (uv across projects — this is the RecordingProcessJobGUI slice).
Companion to BrainCOGS/BrainCogsEphysSorters#5 (DREDge implementation) and BrainCOGS/U19-pipeline-python#94 / #97 (DataJoint param inserts + slurm prefetch).
🤖 Generated with Claude Code