feat: smooth agent BYO-LLM onboarding — self-healing remediation + self-documenting MCP tools + first-time-setup docs#302
Merged
Conversation
…lf-documenting MCP tools + first-time-setup docs loopctl is agent-native and strictly BYO: a tenant provisions its OWN Anthropic + OpenAI embedding keys once, at onboarding, via set_llm_config (role :user). This makes that a smooth, discoverable, self-service experience a stranger agent can complete from docs + tool descriptions + self-healing errors alone. Self-healing remediation (server-side): - New Loopctl.Llm.Remediation builder emits a consistent, machine-readable, secret-free object (action/missing/mcp_tool/example/api/docs/message), distinguishing the missing credential — Anthropic api_key (ingest/ classify/merge) vs OpenAI embedding_api_key (embeddings/semantic search). - Wired into the ingest no-key 422 (code no_api_key) and the search/context keyword-only degrade (meta.remediation when fallback_reason == no_embedding_key). Transient/provider fallbacks carry no remediation (a key IS configured). Reuses the ProviderError-sanitized fallback_reason, so no key/body ever leaks. Self-documenting MCP tools: - Rewrote set_llm_config / llm_config descriptions to fully onboard a stranger agent (WHAT/WHY/WHEN, required LOOPCTL_USER_KEY, which key powers what, partial-merge, has_api_key/has_embedding_key status check). - knowledge_ingest/_batch/_search/_context results now LEAD with an ACTION REQUIRED notice built from the server remediation. - Bumped mcp-server 2.33.1 -> 2.34.0 + CHANGELOG; README first-time-setup section + updated tool-table/env rows. Docs + signup hook: - New docs/onboarding-agent-tenant.md (WebAuthn L0 signup -> provision keys -> use ingest/search) as the authoritative sequence. - Added a "Provision your BYO LLM keys" step to the post-signup onboarding checklist (security-neutral; WebAuthn/chain-of-custody untouched). Tests: remediation builder unit test; ingest 422 + search no_embedding_key remediation shape (names set_llm_config + the right missing key, never a secret); node tests for tool descriptions + result remediation surfacing.
Closes the two LOW test-coverage gaps from the enhanced review of #302: 1. /knowledge/context remediation path: the existing fallback test mocked generate_embedding/2 -> {:error, :service_unavailable}, which sanitizes to the generic "embedding_error" tag (no remediation). Add a test mocking {:error, :no_api_key} and assert meta.fallback_reason == "no_embedding_key" AND meta.remediation names set_llm_config + missing == ["embedding_api_key"], with the same no-secret-leak guard as the search/ingest tests. 2. The new "Provision your BYO LLM keys" onboarding step had no render assertion. Assert the rendered onboarding checklist contains the step title and set_llm_config (static display text only).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes provisioning BYO LLM keys a smooth, discoverable, self-service experience a stranger agent can complete from docs + tool descriptions + self-healing errors alone. The mechanism already existed (
PATCH /api/v1/tenants/me/llm-config+ theset_llm_configMCP tool, role:user); this closes the onboarding gap around it. No security/role/chain-of-custody changes.Self-healing remediation (server-side)
Loopctl.Llm.Remediation— one shared builder emitting a consistent, machine-readable, secret-free object:action,missing,credential,mcp_tool,example,api,docs,message. Distinguishes the missing credential — Anthropicapi_key(ingest/classify/merge) vs OpenAIembedding_api_key(embeddings/semantic search).code: no_api_key) now carrieserror.remediation(Anthropic). The bare string is gone.meta.remediationonly whenfallback_reason == "no_embedding_key"— transient/provider fallbacks carry none (a key IS configured there, so "configure a key" would be wrong). Derives from the existingProviderError-sanitizedfallback_reason, so no key/body can leak.Self-documenting MCP tools (mcp-server 2.33.1 → 2.34.0)
set_llm_config/llm_configdescriptions to fully onboard a stranger agent: WHAT (BYO, encrypted, never returned), WHY (required before ingest/search; loopctl fronts no cost), WHEN (once, at signup), the requiredLOOPCTL_USER_KEY, which key powers what, partial-merge, and thehas_api_key/has_embedding_keystatus check.knowledge_ingest/_batch/_search/_contextresults now lead with anACTION REQUIREDnotice built from the server remediation, so an agent that skips setup reads the exact next step (callset_llm_config) instead of a bare error.LOOPCTL_USER_KEYenv rows; CHANGELOG entry.Docs + signup hook
docs/onboarding-agent-tenant.md— the authoritative agent-tenant sequence (WebAuthn L0 signup → provision keys → use ingest/search).TenantOnboardingLive). Security-neutral display step — WebAuthn / chain-of-custody untouched. (The only concrete signup surface is this checklist; there is no getting-started API payload, so the doc is the authoritative sequence.)Tests
Loopctl.Llm.Remediationunit test (shapes, missing-key names, no secret,for_fallback_reasonmapping).set_llm_config+api_key,apiis a registered route, no secret).no_embedding_keyfallback carries the embedding remediation; a provider-error fallback carries none.Verification
mix compile --warnings-as-errors,format,credo --strict(0),dialyzer(0), fullmix test— 3673 tests, 0 failures (via the pre-commit hook, 1.19/OTP28).node --test test/*.test.js— 112 tests, 0 failures.