JS-SAM: cross-platform sandbox, Opus 4.8, and real Phase-3 spin traces#21
JS-SAM: cross-platform sandbox, Opus 4.8, and real Phase-3 spin traces#21jdubray wants to merge 6 commits into
Conversation
_run_helper assumed a POSIX host: it called os.getuid()/os.getgid() (absent on Windows) and mounted the helper and spec at their host paths, which are invalid container destinations for Windows C:\ paths. Decouple host paths from fixed in-container mount points (/opt/js-sam, /work), normalize -v sources to forward slashes, rewrite specPath to the container path, and pick a non-root --user via a hasattr(os, "getuid") guard (host uid:gid on POSIX, fixed 1000:1000 on Windows). Behavior is unchanged on Linux/macOS; the JS-SAM backend now runs on a Windows host against Docker Desktop. Verified by the existing container integration tests plus a real Phase 1/2 sandbox run.
claude-sonnet-4-20250514 retired 2026-06-15, so the `claude` entry
returned not_found. Repoint it to claude-opus-4-8.
Opus 4.7+, Fable 5, and Mythos 5 removed temperature/top_p/top_k and
return a 400 if any are sent. Dropping them from models.yaml is not
enough — the adapter falls back to GenerationConfig defaults and sent
temperature unconditionally, and litellm's drop_params does not yet
cover these newer models. Add _should_omit_sampling_params() (mirroring
_should_omit_top_p) and guard both temperature and top_p with it.
Verified end to end: get_configured_model("claude").generate_direct(...)
succeeds, and a full spin/JS-SAM run passes Phases 1, 2, and 4.
_display_evaluation_results prints ✓/✗ status glyphs, which raise UnicodeEncodeError on consoles whose default encoding can't represent them (e.g. Windows cp1252) — the evaluation succeeds, then the final summary print crashes. Reconfigure sys.stdout/sys.stderr to UTF-8 at startup; a no-op where they already speak it.
"States explored" was hardcoded to 0 for any backend that isn't TLA+: runtime_check only parsed it from TLC output, and the invariant evaluator never set the top-level field at all (so even TLA+ Phase 4 showed 0). The JS-SAM helper already returns stepsExplored — it just wasn't threaded through. Add states_explored to ModelCheckOutcome and populate it from the JS-SAM check response; seed runtime_check's states from the backend outcome (TLA+ still overrides via its own parse); and set the invariant evaluator's top-level states_explored to the sum across cases. Also fix the JS-SAM Phase-4 metadata key (steps_explored -> states_explored) so the evaluator actually reads it. Verified on spin/JS-SAM: Phase 2 reports 326592 steps (depthMax 6), Phase 4 reports 979776 (326592 x 3 invariants).
Adds the trace-capture harness for the spin task so JS-SAM Phase 3
(transition validation) runs against real Asterinas spinlock behavior
rather than being blocked on missing traces.
- scripts/harness/spin/run.sh: clone Asterinas v0.16.0 (LF), apply the
reference instrumentation + the 2-thread ktest patch, build+run under
QEMU-TCG on an ext4 docker volume, parse serial JSON into NDJSON.
- scripts/harness/spin/build_and_test.sh: in-container build+run.
- scripts/harness/spin/parse_traces.py: folds kernel events
(TryAcquireBlocking/AcquireSuccess/AcquireFail/Release) into
(pre, post) windows keyed on the objective {lockHeld, lockHolder}
state; JS-SAM's projection rule ignores model-internal bookkeeping.
- data/patches/spin_2thread_ktest.patch: adds test_spin_2thread, a
2-actor contention scenario (try_lock avoids single-CPU deadlock);
the reference ktests are single-actor.
- data/sys_traces/spin/spin_2thread.ndjson: the captured 8-window trace
set (force-added; traces are normally generated, not checked in).
The v0.16.0 pin (patch doesn't apply to main), LF normalization of the
clone and patches, and the ext4 build volume (host FS lacks fallocate)
are what make this Linux-oriented flow run under Docker Desktop on
Windows. Verified: Phase 3 scores 87.5% (7/8) on the Opus 4.8 spin model.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Documents the first four-phase JS-SAM run on the spin task (Opus 4.8): method, the Phase 3 kernel trace-capture pipeline, results (Phase 3 87.5%, with the try_lock-from-free modeling discrepancy), the cross-platform fixes, limitations, and reproduction steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Qian-Cheng-nju
left a comment
There was a problem hiding this comment.
Reviewed the diff end-to-end — no correctness bugs. I confirmed the parse_traces.py regex and action names (TryAcquireBlocking/AcquireSuccess/AcquireFail/StopSpinning/Release) match what the reference spin_trace.rs instrumentation actually emits, folded the 10 ktest events by hand and got the same 8 windows as the checked-in spin_2thread.ndjson, and verified the states_explored threading is consistent across ModelCheckOutcome / runtime_check / the invariant evaluator. Two minor harness-robustness nits inline; both are non-blocking.
| LOG="${LOG:-$PROJECT_ROOT/artifacts/spin_capture.log}" | ||
|
|
||
| # 1-2. Clone + patch (idempotent: skip if already instrumented). | ||
| if [ ! -e "$SRC/ostd/src/sync/spin_trace.rs" ]; then |
There was a problem hiding this comment.
Small robustness nit on this idempotency guard. The script runs under set -euo pipefail, so if the reference patch applies (creating spin_trace.rs) but the second git apply below fails, the checkout is left half-patched. On the next run this guard sees spin_trace.rs already exists and skips the whole clone+patch block — so test_spin_2thread never gets added, and build_and_test.sh then runs cargo osdk test test_spin_2thread against a test that isn't there. Recovering needs a manual rm -rf "$SRC".
Consider keying the guard on a sentinel written only after both patches succeed, or doing rm -rf "$SRC" on any failure, so a partial apply self-heals on re-run.
| cd ostd | ||
| timeout 1200 cargo osdk test --features tla-trace --target-arch x86_64 \ | ||
| --qemu-args='-accel tcg' test_spin_2thread 2>&1 | ||
| echo "===== DONE rc=$? =====" |
There was a problem hiding this comment.
rc=$? here can't actually report a failing test. With set -e at the top, a non-zero exit from the cargo osdk test call above aborts the script before this echo runs; when the echo is reached, $? is the exit status of the preceding successful command (0). So this line always prints rc=0.
Not harmful in practice — run.sh still catches the non-zero container exit via its || { …; exit 1; } — but the DONE line is misleading. To make it reflect the real result, capture it explicitly, e.g. run the test with set +e around it and rc=$?; echo "===== DONE rc=$rc =====".
Summary
Follow-up to the JS-SAM backend: makes it runnable on a Windows host, refreshes
the default Claude model, fixes two display/reporting defects, and — the main
addition — stands up the Phase 3 (transition validation) trace-capture
harness for
spin, enabling the first complete four-phase JS-SAM run.With these changes,
spin/ JS-SAM / Claude Opus 4.8 scores across all fourphases:
depthMax 6)The one Phase-3 failure is diagnostic: the generated model reproduces blocking
lock()acquisitions but itstry_lockacquire never actually takes the lock —caught only by replaying against transitions captured from the real Asterinas
spinlock.
Changes
js_sam: cross-platform Docker sandbox._run_helperwas POSIX-only(
os.getuid, host-path mounts). Now uses fixed container mount points(
/opt/js-sam,/work), forward-slash-vsources, aspecPathrewrite, anda
hasattr(os, "getuid")-guarded--user. Linux/macOS behavior unchanged;runs on Windows/Docker Desktop.
claude→claude-opus-4-8(Sonnet 4 retired 2026-06-15);new
_should_omit_sampling_params()dropstemperature/top_p/top_kformodels that 400 on them (Opus 4.7+, Fable 5, Mythos 5) — LiteLLM
drop_paramsdoesn't yet cover them.
✓/✗and crashed on aWindows cp1252 console after a successful eval; std streams are reconfigured to
UTF-8.
states_exploredfor non-TLA+ backends. Was hardcoded 0 even though theJS-SAM helper reports
stepsExplored; now threaded throughModelCheckOutcome, and the invariant evaluator's top-level field (previously0 for all backends) is populated.
scripts/harness/spin/*,data/patches/spin_2thread_ktest.patch,data/sys_traces/spin/*). ClonesAsterinas v0.16.0 (the patch doesn't apply to
main), applies the referenceinstrumentation + a new 2-actor
test_spin_2threadktest (try_lockavoids single-CPU deadlock; the reference tests are single-actor), builds and
runs it under QEMU-TCG on an ext4 docker volume, and folds the kernel serial
JSON into JS-SAM
(pre,post)windows keyed on the objective{lockHeld, lockHolder}state.Notes
docs/js_sam_first_experiment.mdis a full write-up (method, capturepipeline, results, limitations, reproduction).
(
scripts/harness/spin/run.sh). The large Asterinas clone stays gitignored.node:20-slimandasterinas/asterinas:0.16.0-20250822images.🤖 Generated with Claude Code