feat(ctl): add --dependencies flag to infrahubctl marketplace get#1118
feat(ctl): add --dependencies flag to infrahubctl marketplace get#1118minitriga wants to merge 10 commits into
Conversation
Deploying infrahub-sdk-python with
|
| Latest commit: |
693a924
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e392208f.infrahub-sdk-python.pages.dev |
| Branch Preview URL: | https://marketplace-dependencies-ihs.infrahub-sdk-python.pages.dev |
github-actions
Bot
added
the
type/documentation
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## stable #1118 +/- ##
==========================================
+ Coverage 82.32% 82.43% +0.11%
==========================================
Files 138 138
Lines 12042 12186 +144
Branches 1805 1833 +28
==========================================
+ Hits 9913 10045 +132
- Misses 1573 1579 +6
- Partials 556 562 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Resolve a collection's dependencies via the marketplace API and download each schema individually. Prerequisite collections are grouped into their own directories and standalone dependency schemas land in the output root, with transitive, cycle-safe resolution. Collection members (requested or prerequisite) download strictly; loose and transitively-discovered schema dependencies soft-fail with a note so one missing dependency does not abort the bundle. Referenced kinds the marketplace cannot resolve are reported without downloading. Refs: IHS-246, #1117
minitriga
force-pushed
the
marketplace-dependencies-ihs-246
branch
from
cfb9019 to
e02ae4f
Compare
Extend --dependencies beyond collections: `marketplace get <schema> --dependencies` now downloads the requested schema plus its transitive dependency schemas (soft-fail, deduped, cycle-safe) into the output root. The requested schema downloads strictly; a dependency sharing its name in another namespace is disambiguated into a namespace subdirectory rather than overwriting it. Refs: IHS-246, #1117
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Avoid duplicating a schema across the output tree when resolving dependencies: before writing, reconcile against schemas already present (e.g. a dependency at the root vs. a copy under a collection directory from a prior download). An already-present schema is kept by default, or overwritten in place with the new `-y`/`--yes` flag; an interactive terminal is prompted. Only files present before the run are considered, and the check is skipped entirely without --dependencies and in --stdout mode, so existing behavior is unchanged. Refs: IHS-246, #1117
Collapse the latest-version dependency lookup in _read_schema_dependencies to a single expression, and extract the duplicated "unresolved dependencies" report line into a shared _print_unresolved helper. No behavior change. Refs: IHS-246, #1117
When resolving a single schema's --dependencies, place each dependency under the directory of the collection it belongs to (via /collections/for-schema), mirroring a collection download; dependencies in no collection stay at the output root. The existing existing-file reconciliation (keep / --yes overwrite, no cross-directory duplicates) continues to apply. Refs: IHS-246, #1117
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
There was a problem hiding this comment.
This whole implementation looks very complicated to live in the SDK. Im wondering if thats something we need to have in the marketplace itself in case we need to display it there. Would it make sense to have it as an endpoint in that case so the SDK has as little logic as possible and minimize requests over the network?
There was a problem hiding this comment.
Agree the resolution logic is heavy for the SDK — dependency resolution really is the marketplace's job. The ideal is the marketplace exposing an endpoint that returns the full closure so the SDK just downloads what it's given (one request, no walk). This overlaps with gmazoyer's point on the dependencies payload above. My preference is to land the flag now since the UI already advertises --dependencies, and open a follow-up to move resolution server-side once we confirm what the collection endpoint already returns. Happy to file that issue — or hold this PR if you'd rather do it in one go. What do you prefer?
There was a problem hiding this comment.
This is now done — resolution moved server-side. The marketplace exposes .../dependencies endpoints that return the resolved closure, so the SDK holds no resolution logic anymore: it fetches the grouped result and downloads it. Much thinner, one request instead of a walk. Thanks for pushing on this.
| schemas: list[dict[str, Any]] = [ | ||
| {"namespace": str(entry["namespace"]), "name": str(entry["name"]), "version": None} | ||
| for entry in dep.get("schemas") or [] | ||
| if isinstance(entry, dict) and entry.get("namespace") and entry.get("name") | ||
| ] |
There was a problem hiding this comment.
The backend's dependencies already has the id and the full transitive list (that's what show reads). Dropping the id forces the re-walk plus a for-schema call per dep. Can we keep the id and use this list directly?
There was a problem hiding this comment.
Done in 9c1f1bc. The marketplace now resolves the closure server-side and exposes it at GET /collections/{ns}/{name}/dependencies and GET /schemas/{ns}/{name}/dependencies, returning the full transitive set grouped by source (prerequisite collections with their members + standalone schemas, plus unresolved_kinds/hidden_count). The SDK now just calls that once and downloads each group — the client-side walk (_resolve_dependency_closure, per-schema re-reads, _walk_collection_graph) and the for-schema lookups are gone. Net ~250 fewer lines. Verified end-to-end against the live marketplace.
…ents Two review fixes for marketplace `get --dependencies`: - Versioned single-schema downloads now resolve the requested version's dependencies. `_read_schema_dependencies` selects the matching version from the schema-detail payload instead of always reading `latest_version`, and `_resolve_dependency_closure` threads each item's version through the walk. - Names from the marketplace (schema/namespace/collection) are validated with `_safe_segment` before being used as directory or file components, so a value like `../evil` is refused rather than escaping the output directory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ncies-ihs-246 # Conflicts: # infrahub_sdk/ctl/marketplace.py # tests/unit/ctl/test_marketplace_app.py
… endpoints
The marketplace now resolves dependency closures server-side and exposes them
at GET /collections/{ns}/{name}/dependencies and GET /schemas/{ns}/{name}/dependencies,
returning the full transitive set grouped by source (prerequisite collections
with their members, plus standalone schemas) along with unresolved kinds and a
hidden count.
Replace the client-side walk with a single call to those endpoints:
- Delete the per-schema detail re-walk (_read_schema_dependencies,
_resolve_dependency_closure), the collection-graph walk (_walk_collection_graph,
_fetch_collection_payload, _collection_dependency_targets), and the per-dependency
for-schema owning-collection lookup (_owning_collection_name).
- _download_collection_tree / _download_schema_tree now fetch the resolved closure
once and download each group into its directory, reusing the existing dedup,
disambiguation, soft/strict, and reconciliation logic.
- Forward --version to the schema dependencies endpoint so a pinned schema
resolves its own version's dependencies (resolved server-side).
- Report dependencies hidden by visibility (hidden_count).
The on-disk layout, flags, and reporting are unchanged; only the resolution
mechanism moved to the marketplace. Verified end-to-end against the live
marketplace for collection, single-schema, and versioned downloads.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update — dependency resolution moved server-sidePushed
These return the full transitive closure grouped by source (prerequisite collections with their members embedded, standalone schemas, plus Deleted the entire client-side walk — Also folded in the two earlier review fixes:
On-disk layout, flags, and output text are unchanged. Verified end-to-end against the live marketplace (collection, single-schema, and versioned downloads); unit tests rewritten to mock the new endpoints (65 passing), ruff/ty/mypy clean. |
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
_download_collection_tree used len(members) as the dependency-count baseline, but total_written only counts members actually written — kept pre-existing members (no --yes) could make dependency_count negative or wrong in the report. Use the count returned by the members _download_schema_set call as the baseline, matching how the schema tree already handles it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
3 participants
Why
Downloading a marketplace collection only fetched its own member schemas. The schemas those members reference (defined in other collections or as standalone schemas) were left out, so the bundle wouldn't load into Infrahub without manually hunting down each missing dependency. The marketplace UI already advertises
infrahubctl marketplace get <ns>/<name> --collection --dependencies, but the SDK didn't implement the flag — so anyone copying that command hit an error.Goal: ship the
--dependenciesflag so a collection downloads with everything it needs in one command.Non-goals: changing the no-flag behavior; using the marketplace's ZIP
?dependencies=trueendpoint (we resolve via the API and download per-schema for a consistent on-disk layout).Closes #1117
What changed
Behavioral:
infrahubctl marketplace get <ns>/<name> --collection --dependenciesnow downloads the collection's members plus the schemas it depends on.schemas/<collection>/directory; standalone dependency schemas land in the output root. Resolution is transitive, cycle-safe, and deduplicated.What stayed the same:
--dependencies, collection and single-schema downloads are byte-for-byte unchanged.--dependencieson a single schema is a no-op with an informational note.How to review
Focus on
infrahub_sdk/ctl/marketplace.py— the dependency walk and the per-collection layout. Tests intests/unit/ctl/test_marketplace_app.pymirror cases C1–C9 plus dedup/cycle and strict-vs-soft. The.mdxdoc anddev/specs/artifacts are generated/planning.How to test
```bash
uv run pytest tests/unit/ctl/test_marketplace_app.py -q
uv run infrahubctl marketplace get infrahub/routing-protocols --collection --dependencies -o ./schemas
find ./schemas -type f
```
Expected: members under
schemas/routing-protocols/, thebase-schemasprerequisite underschemas/base-schemas/. All 8 live marketplace collections were downloaded successfully (0 failures) during development.Impact & rollout
Checklist
changelog/1117.added.md)dev/specs/)Summary by cubic
Adds the
--dependenciesand-y/--yesflags toinfrahubctl marketplace getto resolve and download collections and single schemas with their transitive dependencies via Marketplace/.../dependencies, in a deduped layout that mirrors the UI. Closes IHS-246.New Features
... --collection --dependenciesdownloads members plus prerequisite collections and standalone dependency schemas. Layout: requested collection under./<collection>/; each prerequisite collection in its own directory; standalone dependencies in the output root. Same-name conflicts are disambiguated by namespace; resolution is server-side and deduped.... --dependenciesdownloads the schema plus its transitive dependencies; dependencies are grouped under their owning collection directory, or in the output root if none. Same-name conflicts across namespaces are disambiguated into namespace subdirectories.-y/--yes(TTY prompts; non-interactive keeps). Reconciliation only considers files present before the run and is skipped with--stdout. Without--dependencies, behavior is unchanged.Bug Fixes
--versionwith--dependenciesresolves the pinned version’s dependencies..., separators, NUL), preventing output-dir escape.--yes).Written for commit 693a924. Summary will update on new commits.