fix(witness): STH persistence + transparent bootstrap-412 retry (#298)

[mkreyman]

Summary

Fixes #298 (and the #299 root cause): a fresh MCP-server process reusing a
long-lived env-var key (LOOPCTL_AGENT_KEY, LOOPCTL_ORCH_KEY, …) failed its
first tool call with 412 witness_bootstrap_already_consumed, because the
witness protocol's one-time bootstrap grace is consumed once per API key and the
MCP server kept the learned Signed Tree Head (STH) in memory only.

Two parts: a client-robustness fix in the MCP server, and a careful,
security-neutral server-side hardening of the witness plug.

Part A — MCP server client robustness

• Cross-process persistence. The learned STH is cached to a small per-server
  state file (default <os-tmp>/loopctl-mcp-sth-<hash>.json, override via
  LOOPCTL_STH_STATE_PATH) so a fresh process loads it and sends
  X-Loopctl-Last-Known-STH on its first request — avoiding the 412 entirely.
  STHs are append-only (one row per (tenant_id, chain_position)), so a persisted
  historical STH still validates; it is never a stale-STH 409 trap.
• Transparent retry-once. Any 412 carrying an x-loopctl-current-sth header
  is caught, the STH is cached, and the SAME request is retried exactly once
  with the real header — so the tool call succeeds instead of surfacing the 412.
  Bounded to a single attempt (never a loop). Safe because the witness plug halts
  before the operation runs, so the rejected request had no side effect (even
  for POSTs).
• Logic extracted to mcp-server/lib/witness-sth.js (shared with the test suite,
  single source of truth); apiCall delegates to it. All file I/O degrades
  gracefully (missing/corrupt/unwritable → in-memory + retry).
• Docs: README "Witness protocol (STH)" section + LOOPCTL_STH_STATE_PATH;
  version bump 2.32.0 → 2.33.0 + CHANGELOG entry.
• Tests: mcp-server/test/witness_sth.test.js — 412-with-STH triggers a single
  transparent retry that succeeds; persisted STH loaded on startup + sent on the
  first request; corrupt/missing/unwritable state file degrades to bootstrap +
  retry; retry bounded to one attempt. 99/99 node tests pass.

Part B — server-side protocol hardening (security-neutral)

In lib/loopctl_web/plugs/validate_witness_header.ex:

• The witness_bootstrap_already_consumed 412 now reuses the STH already read
  (successfully, inside the rescue-guarded block) before the atomic consume — no
  second DB round-trip — so it can never become a 500 and ALWAYS carries a
  valid x-loopctl-current-sth header for the client's one-shot retry. This is
  the linchpin that makes retry-once reliable.
• Adds a machine-readable retry contract (error.remediation.retry +
  current_sth) so future clients don't rediscover the protocol.
• Adds a distinct info log + [:loopctl, :witness, :bootstrap_already_consumed]
  telemetry event so operators can meter consumed-grace 412s separately from the
  divergence-alerting path.

Why this does NOT weaken chain-of-custody

The one-time bootstrap grace (api_keys.sth_bootstrap_consumed_at, custody-03 /
GHSA-36g5-mcrh-rcrm) is unchanged: the atomic conditional UPDATE guarded by
is_nil(sth_bootstrap_consumed_at) still lets exactly one first-request win, and
a consumed key's request is still rejected and halted (no operation runs). We
only make the already-present resync path (the STH was always returned in the
412 header) reliable, self-documenting, and observable. The STH handed back is
already public via GET /api/v1/audit/sth/:tenant_id. No role, RLS, or trust
boundary changed.

Deliberately NOT done (would re-open custody-03): making the grace repeatable,
per-request, unbounded, or letting a client skip witness enforcement. If reducing
412s further required weakening the gate, this PR stops at the safe changes above.

Dispatch-based (v2) clients are already unaffected — each dispatch mints a
fresh ephemeral key that gets its own clean one-time bootstrap. The affected
population is exactly the long-lived-env-key clients (the MCP server), which
Part A fixes.

Verification

• mix precommit green (compile --warnings-as-errors, format, credo --strict,
  deps.audit, dialyzer 0, 3655 tests, 0 failures) — incl. the witness/custody
  suite (validate_witness_header_test.exs, +4 new tests).
• mcp-server node suite: 99/99 pass.

Draft — do not merge.