Switch skill CI validation workflows to vally lint#2030
Draft
aaronpowell wants to merge 6 commits into
Draft
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the downloaded skill-validator binary with px @microsoft/vally-cli lint in the external plugin quality gates pipeline: - Remove downloadSkillValidator() and SKILL_VALIDATOR_ARCHIVE_URL constant - Replace �uildSkillValidatorArgs() + unSkillValidatorGate() with �uildVallyLintArgs() + unVallyLintGate() that run px vally-cli lint per resolved skill directory (falling back to the full plugin root when no specific skill paths can be resolved from plugin.json) - Rename result keys skill_validator_status / skill_validator_output to �ally_lint_status / �ally_lint_output throughout both �ng/external-plugin-quality-gates.mjs and �ng/external-plugin-intake.mjs - Update PR comment markdown to show 'vally lint' instead of 'skill-validator' - Update CONTRIBUTING.md prose references accordingly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aaronpowell
changed the title
Replace the npx-spawned vally-cli process with a direct call to the
@microsoft/vally core library in the external plugin quality gates scripts:
- Add @microsoft/vally as a devDependency in package.json
- Import runLint and LintConsoleReporter from @microsoft/vally
- Replace runVallyLintGate() process spawn with async API call:
- runLint({ rootPath }) returns structured LintResults
- LintConsoleReporter with a Writable capture stream collects
text output without printing to stdout
- Make runExternalPluginQualityGates() async (propagated to
runExternalPluginPrQualityGates() and both main entry points)
- Use Promise.all in runExternalPluginPrQualityGates() for parallel
plugin checks
- Fix remaining skill_validator_status reference in pr-quality-gates
summary string (now vally-lint=...) and YAML workflow table header
- Add 'npm install @microsoft/vally' step to both calling workflows
This removes a layer of indirection (Node -> npx -> CLI -> library)
and replaces it with a direct in-process library call, which is faster,
more reliable, and gives structured access to lint results.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the external
skill-validatorbinary (downloaded nightly from GitHub releases) with@microsoft/vally-cli(vally lint) for all skill static-analysis CI checks.Why vally lint?
npx, no curl+tar+chmod per runvally lintenforces the full Agent Skills spec: valid frontmatter types, file-reference integrity (valid-refs), orphan-file detection, and a 500-lineSKILL.mdcapChanges
Skill PR gate & comment (
skill-check.yml,skill-check-comment.yml)actions/setup-node+npx @microsoft/vally-cli lintvally-lint-results/vally-output.txtNightly quality report (
skill-quality-report.yml)SKILL.md, not.agent.md)External plugin quality gates (
eng/external-plugin-quality-gates.mjs)downloadSkillValidator()andSKILL_VALIDATOR_ARCHIVE_URLbuildSkillValidatorArgs()+runSkillValidatorGate()withbuildVallyLintArgs()+runVallyLintGate(): runsnpx @microsoft/vally-cli lintper resolved skill directory, falling back to the full plugin root when no specific skill paths are foundskill_validator_status→vally_lint_status,skill_validator_output→vally_lint_outputExternal plugin intake comments (
eng/external-plugin-intake.mjs)normalizeQualityGateResultdefaults updated to new key namesDocumentation (
CONTRIBUTING.md)skill-validator check --pluginupdated tovally lintBehavioural notes
vally lintexit code but does not fail the workflow (same as before)vallytreats prose absolute paths like/spec/as broken file references (valid-refs: absolute-path). Skills with such documentation text may show new warningsvallyenforces a 500-line cap onSKILL.md. Many existing skills exceed this; the gate is warn-only so this doesn't block PRs but does surface in the quality reportvally lintonly processesSKILL.md;.agent.mdlinting remains out of scope for this change