Skip to content

JS-SAM: cross-platform sandbox, Opus 4.8, and real Phase-3 spin traces#21

Open
jdubray wants to merge 6 commits into
specula-org:mainfrom
jdubray:js-sam-crossplatform-phase3
Open

JS-SAM: cross-platform sandbox, Opus 4.8, and real Phase-3 spin traces#21
jdubray wants to merge 6 commits into
specula-org:mainfrom
jdubray:js-sam-crossplatform-phase3

Conversation

@jdubray

@jdubray jdubray commented Jul 1, 2026

Copy link
Copy Markdown

Summary

Follow-up to the JS-SAM backend: makes it runnable on a Windows host, refreshes
the default Claude model, fixes two display/reporting defects, and — the main
addition — stands up the Phase 3 (transition validation) trace-capture
harness
for spin, enabling the first complete four-phase JS-SAM run.

With these changes, spin / JS-SAM / Claude Opus 4.8 scores across all four
phases:

Phase Result
1 compilation_check PASS (0 errors)
2 runtime_check PASS (326,592 states, depthMax 6)
3 transition_validation PASS — 87.5% (7/8) against real kernel traces
4 invariant_verification PASS (3/3)

The one Phase-3 failure is diagnostic: the generated model reproduces blocking
lock() acquisitions but its try_lock acquire never actually takes the lock —
caught only by replaying against transitions captured from the real Asterinas
spinlock.

Changes

  • js_sam: cross-platform Docker sandbox. _run_helper was POSIX-only
    (os.getuid, host-path mounts). Now uses fixed container mount points
    (/opt/js-sam, /work), forward-slash -v sources, a specPath rewrite, and
    a hasattr(os, "getuid")-guarded --user. Linux/macOS behavior unchanged;
    runs on Windows/Docker Desktop.
  • Model refresh. claudeclaude-opus-4-8 (Sonnet 4 retired 2026-06-15);
    new _should_omit_sampling_params() drops temperature/top_p/top_k for
    models that 400 on them (Opus 4.7+, Fable 5, Mythos 5) — LiteLLM drop_params
    doesn't yet cover them.
  • UTF-8 console output. Result summaries printed / and crashed on a
    Windows cp1252 console after a successful eval; std streams are reconfigured to
    UTF-8.
  • states_explored for non-TLA+ backends. Was hardcoded 0 even though the
    JS-SAM helper reports stepsExplored; now threaded through
    ModelCheckOutcome, and the invariant evaluator's top-level field (previously
    0 for all backends) is populated.
  • Spin Phase-3 trace harness (scripts/harness/spin/*,
    data/patches/spin_2thread_ktest.patch, data/sys_traces/spin/*). Clones
    Asterinas v0.16.0 (the patch doesn't apply to main), applies the reference
    instrumentation + a new 2-actor test_spin_2thread ktest (try_lock
    avoids single-CPU deadlock; the reference tests are single-actor), builds and
    runs it under QEMU-TCG on an ext4 docker volume, and folds the kernel serial
    JSON into JS-SAM (pre,post) windows keyed on the objective
    {lockHeld, lockHolder} state.

Notes

  • docs/js_sam_first_experiment.md is a full write-up (method, capture
    pipeline, results, limitations, reproduction).
  • The captured NDJSON is checked in for convenience; the harness regenerates it
    (scripts/harness/spin/run.sh). The large Asterinas clone stays gitignored.
  • Requires the node:20-slim and asterinas/asterinas:0.16.0-20250822 images.

🤖 Generated with Claude Code

kizo-core and others added 6 commits June 30, 2026 21:50
_run_helper assumed a POSIX host: it called os.getuid()/os.getgid()
(absent on Windows) and mounted the helper and spec at their host paths,
which are invalid container destinations for Windows C:\ paths.

Decouple host paths from fixed in-container mount points (/opt/js-sam,
/work), normalize -v sources to forward slashes, rewrite specPath to the
container path, and pick a non-root --user via a hasattr(os, "getuid")
guard (host uid:gid on POSIX, fixed 1000:1000 on Windows). Behavior is
unchanged on Linux/macOS; the JS-SAM backend now runs on a Windows host
against Docker Desktop. Verified by the existing container integration
tests plus a real Phase 1/2 sandbox run.
claude-sonnet-4-20250514 retired 2026-06-15, so the `claude` entry
returned not_found. Repoint it to claude-opus-4-8.

Opus 4.7+, Fable 5, and Mythos 5 removed temperature/top_p/top_k and
return a 400 if any are sent. Dropping them from models.yaml is not
enough — the adapter falls back to GenerationConfig defaults and sent
temperature unconditionally, and litellm's drop_params does not yet
cover these newer models. Add _should_omit_sampling_params() (mirroring
_should_omit_top_p) and guard both temperature and top_p with it.

Verified end to end: get_configured_model("claude").generate_direct(...)
succeeds, and a full spin/JS-SAM run passes Phases 1, 2, and 4.
_display_evaluation_results prints ✓/✗ status glyphs, which raise
UnicodeEncodeError on consoles whose default encoding can't represent
them (e.g. Windows cp1252) — the evaluation succeeds, then the final
summary print crashes. Reconfigure sys.stdout/sys.stderr to UTF-8 at
startup; a no-op where they already speak it.
"States explored" was hardcoded to 0 for any backend that isn't TLA+:
runtime_check only parsed it from TLC output, and the invariant
evaluator never set the top-level field at all (so even TLA+ Phase 4
showed 0). The JS-SAM helper already returns stepsExplored — it just
wasn't threaded through.

Add states_explored to ModelCheckOutcome and populate it from the JS-SAM
check response; seed runtime_check's states from the backend outcome
(TLA+ still overrides via its own parse); and set the invariant
evaluator's top-level states_explored to the sum across cases. Also fix
the JS-SAM Phase-4 metadata key (steps_explored -> states_explored) so
the evaluator actually reads it.

Verified on spin/JS-SAM: Phase 2 reports 326592 steps (depthMax 6),
Phase 4 reports 979776 (326592 x 3 invariants).
Adds the trace-capture harness for the spin task so JS-SAM Phase 3
(transition validation) runs against real Asterinas spinlock behavior
rather than being blocked on missing traces.

- scripts/harness/spin/run.sh: clone Asterinas v0.16.0 (LF), apply the
  reference instrumentation + the 2-thread ktest patch, build+run under
  QEMU-TCG on an ext4 docker volume, parse serial JSON into NDJSON.
- scripts/harness/spin/build_and_test.sh: in-container build+run.
- scripts/harness/spin/parse_traces.py: folds kernel events
  (TryAcquireBlocking/AcquireSuccess/AcquireFail/Release) into
  (pre, post) windows keyed on the objective {lockHeld, lockHolder}
  state; JS-SAM's projection rule ignores model-internal bookkeeping.
- data/patches/spin_2thread_ktest.patch: adds test_spin_2thread, a
  2-actor contention scenario (try_lock avoids single-CPU deadlock);
  the reference ktests are single-actor.
- data/sys_traces/spin/spin_2thread.ndjson: the captured 8-window trace
  set (force-added; traces are normally generated, not checked in).

The v0.16.0 pin (patch doesn't apply to main), LF normalization of the
clone and patches, and the ext4 build volume (host FS lacks fallocate)
are what make this Linux-oriented flow run under Docker Desktop on
Windows. Verified: Phase 3 scores 87.5% (7/8) on the Opus 4.8 spin model.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Documents the first four-phase JS-SAM run on the spin task (Opus 4.8):
method, the Phase 3 kernel trace-capture pipeline, results (Phase 3
87.5%, with the try_lock-from-free modeling discrepancy), the
cross-platform fixes, limitations, and reproduction steps.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@Qian-Cheng-nju Qian-Cheng-nju left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff end-to-end — no correctness bugs. I confirmed the parse_traces.py regex and action names (TryAcquireBlocking/AcquireSuccess/AcquireFail/StopSpinning/Release) match what the reference spin_trace.rs instrumentation actually emits, folded the 10 ktest events by hand and got the same 8 windows as the checked-in spin_2thread.ndjson, and verified the states_explored threading is consistent across ModelCheckOutcome / runtime_check / the invariant evaluator. Two minor harness-robustness nits inline; both are non-blocking.

LOG="${LOG:-$PROJECT_ROOT/artifacts/spin_capture.log}"

# 1-2. Clone + patch (idempotent: skip if already instrumented).
if [ ! -e "$SRC/ostd/src/sync/spin_trace.rs" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small robustness nit on this idempotency guard. The script runs under set -euo pipefail, so if the reference patch applies (creating spin_trace.rs) but the second git apply below fails, the checkout is left half-patched. On the next run this guard sees spin_trace.rs already exists and skips the whole clone+patch block — so test_spin_2thread never gets added, and build_and_test.sh then runs cargo osdk test test_spin_2thread against a test that isn't there. Recovering needs a manual rm -rf "$SRC".

Consider keying the guard on a sentinel written only after both patches succeed, or doing rm -rf "$SRC" on any failure, so a partial apply self-heals on re-run.

cd ostd
timeout 1200 cargo osdk test --features tla-trace --target-arch x86_64 \
--qemu-args='-accel tcg' test_spin_2thread 2>&1
echo "===== DONE rc=$? ====="

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rc=$? here can't actually report a failing test. With set -e at the top, a non-zero exit from the cargo osdk test call above aborts the script before this echo runs; when the echo is reached, $? is the exit status of the preceding successful command (0). So this line always prints rc=0.

Not harmful in practice — run.sh still catches the non-zero container exit via its || { …; exit 1; } — but the DONE line is misleading. To make it reflect the real result, capture it explicitly, e.g. run the test with set +e around it and rc=$?; echo "===== DONE rc=$rc =====".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants