Skip to content

Route vCPU preemption signals via sigwait thread#145

Merged
jserv merged 2 commits into
mainfrom
preemption-signal
Jul 5, 2026
Merged

Route vCPU preemption signals via sigwait thread#145
jserv merged 2 commits into
mainfrom
preemption-signal

Conversation

@jserv

@jserv jserv commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

elfuse preempted a running vCPU by sending a host signal whose handler called hv_vcpus_exit(): SIGUSR2 for the cross-process guest-signal doorbell and SIGALRM for the per-iteration safety timeout. Because the signal landed on the vCPU thread inside hv_vcpu_run, Apple HVF aborted the run with HV_EXIT_REASON_UNKNOWN instead of the clean CANCELED that hv_vcpus_exit() produces for a vCPU caught between runs, leaving the run loop unable to tell a self-directed preemption from a genuine fault.

Block SIGUSR2 and SIGALRM on the main thread before any vCPU thread is created (proc_preempt_init), so every CLONE_THREAD worker and posix_spawn fork-child inherits the block. A dedicated thread sigwaits both and kicks every live vCPU via thread_interrupt_all() from a thread that never runs one, so hv_vcpu_run always returns CANCELED and the run loop treats any HV_EXIT_REASON_UNKNOWN as a hard hypervisor fault. The g_timed_out and g_external_guest_signal handoff flags become _Atomic with release/acquire ordering now that they cross the preempt-thread to vCPU-thread boundary.

Make a guest thread parked in a blocking host syscall reachable by a signal, which hv_vcpus_exit() alone cannot interrupt:

  • attention_raise() pokes the wakeup pipe on every deliverable signal.
  • ppoll/pselect/epoll_pwait break out on signal_pending_interruption.
  • a blocking pipe/stdio read waits in wait_readable_or_interrupted, which returns EINTR only for a guest-visible delivery, skips O_WRONLY and O_NONBLOCK fds, and uses a bounded 200ms recheck; the non-verbose fast-path read diverts a would-block pipe/stdio read to this slow path.
  • fork_child_main creates the wakeup pipe, which it skips via syscall_init.

Make the signal transport race-free: the sender holds flock around its append and the drain holds it across a full read plus ftruncate and no longer unlinks, so a concurrent append is never routed to an orphaned inode. Each process unlinks its own empty file at exit.

signal_pending_interruption() reports whether a guest-visible delivery is pending and excludes ignored and default-ignore signals, so read, poll, select, and epoll no longer wake spuriously on an ignored signal.

Close #77


Summary by cubic

Route vCPU preemption (SIGUSR2/SIGALRM) through a dedicated sigwait thread so hv_vcpu_run reliably returns CANCELED on HVF, and expand interruptible blocking I/O so reads, writes, and socket ops wake only for guest-visible signals. Also harden cross-process signal transport and clean up wakeup-pipe handling.

  • Bug Fixes
    • Preemption: block SIGUSR2/SIGALRM before any vCPU threads; a sigwait thread sets _Atomic handoff flags and kicks vCPUs; HV_EXIT_REASON_UNKNOWN now indicates a hypervisor fault.
    • Blocking I/O: add per-fd can_block and a shared io_wait_fd_or_interrupted; used by read/write/readv/writev and socket recv/accept/connect/send; the fast-path read/write diverts on would-block or probe error; sends probe with MSG_DONTWAIT and retry on EAGAIN; connect flips nonblocking, waits for POLLOUT, checks SO_ERROR; zero-length iovecs still enforce EBADF; zero-length recvmsg returns immediately.
    • Signals and waits: attention_raise pokes the wakeup pipe; ppoll/pselect/epoll_pwait break only on guest-visible deliveries; ignored/default-ignore signals no longer interrupt; wakeup-pipe init is idempotent with read/drain helpers.
    • Signal transport: sender append and receiver drain are serialized with flock; drain reads fully and ftruncate’s; each process unlinks its empty file at exit.
    • Tests: test-poll covers ignored SIGCHLD and that SA_RESTART read handlers run and unblock recv; test-socket adds zero-length/invalid recvmsg cases; test-readv-writev adds an all-empty-iovec access-check.

Written for commit c3de5dc. Summary will update on new commits.

Review in cubic

@jserv jserv requested a review from Max042004 July 3, 2026 20:00
cubic-dev-ai[bot]

This comment was marked as resolved.

@jserv jserv force-pushed the preemption-signal branch from 296957f to a102a20 Compare July 3, 2026 20:19
@Max042004

Max042004 commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

below paths not covered

Sockets

  • read on a socket fd — excluded by the FD_PIPE || FD_STDIO check
  • recv / recvfrom / recvmsg — direct blocking call, src/syscall/net.c:783
  • accept / accept4src/syscall/net.c:348
  • connectsrc/syscall/net.c:546
  • blocking write / send* on a full buffer (no write-side wait helper exists)

FIFO and pty/character devices (both verified): opened_fd_type()
only tests S_ISDIR (src/syscall/fs.c:50), so these fall into FD_REGULAR
and inherit the regular-file exclusion — but Linux makes reads on them
interruptible. open(O_RDONLY) on a FIFO also blocks in the host open()
itself.

Consequences:

  • A guest-level kill cannot reach a process stuck in blocking recv/accept;
    only killing the host process works.
  • The classic alarm(n) + blocking read(sock) timeout pattern never fires.
  • Ctrl-C is ignored by programs using default-blocking sockets (CPython signal
    handling relies on EINTR per PEP 475; same for nc, blocking DB clients).
  • Runtimes using non-blocking fds + epoll/poll (node, Go, nginx) are unaffected,
    and SO_RCVTIMEO passes through (src/syscall/net-abi.c:291), which limits
    the blast radius to default-blocking usage.

jserv added 2 commits July 5, 2026 23:49
elfuse preempted a running vCPU by sending a host signal whose handler
called hv_vcpus_exit(): SIGUSR2 for the cross-process guest-signal
doorbell and SIGALRM for the per-iteration safety timeout. Because the
signal landed on the vCPU thread inside hv_vcpu_run, Apple HVF aborted
the run with HV_EXIT_REASON_UNKNOWN instead of the clean CANCELED that
hv_vcpus_exit() produces for a vCPU caught between runs, leaving the run
loop unable to tell a self-directed preemption from a genuine fault.

Block SIGUSR2 and SIGALRM on the main thread before any vCPU thread is
created (proc_preempt_init), so every CLONE_THREAD worker and posix_spawn
fork-child inherits the block. A dedicated thread sigwaits both and kicks
every live vCPU via thread_interrupt_all() from a thread that never runs
one, so hv_vcpu_run always returns CANCELED and the run loop treats any
HV_EXIT_REASON_UNKNOWN as a hard hypervisor fault. The g_timed_out and
g_external_guest_signal handoff flags become _Atomic with release/acquire
ordering now that they cross the preempt-thread to vCPU-thread boundary.

Make a guest thread parked in a blocking host syscall reachable by a
signal, which hv_vcpus_exit() alone cannot interrupt:
- attention_raise() pokes the wakeup pipe on every deliverable signal.
- ppoll/pselect/epoll_pwait break out on signal_pending_interruption.
- a blocking pipe/stdio read waits in wait_readable_or_interrupted, which
  returns EINTR only for a guest-visible delivery, skips O_WRONLY and
  O_NONBLOCK fds, and uses a bounded 200ms recheck; the non-verbose
  fast-path read diverts a would-block pipe/stdio read to this slow path.
- fork_child_main creates the wakeup pipe, which it skips via syscall_init.

Make the signal transport race-free: the sender holds flock around its
append and the drain holds it across a full read plus ftruncate and no
longer unlinks, so a concurrent append is never routed to an orphaned
inode. Each process unlinks its own empty file at exit.

signal_pending_interruption() reports whether a guest-visible delivery is
pending and excludes ignored and default-ignore signals, so read, poll,
select, and epoll no longer wake spuriously on an ignored signal.

The fork-child reaches syscall_init() indirectly through
fork_ipc_recv_fd_table() before the vCPU loop runs, so it already
creates the process-local wakeup pipe. The explicit wakeup_pipe_init()
in fork_child_main was therefore redundant and, worse, leaked two fds
per fork: the later syscall_init() call overwrote wakeup_pipe_rd and
wakeup_pipe_wr without closing the first pair. Drop the redundant call
and make wakeup_pipe_init() idempotent so any second reach is a no-op
rather than a leak.

Route a fast-path pipe/stdio read to the interruptible slow path on a
probe error (poll() < 0), not only when the fd is not ready, so a
transient poll failure cannot leave the vCPU thread in a blocking host
read() that the preempt thread cannot break.

Replace the three inline wakeup-pipe drain loops in ppoll, pselect, and
epoll_pwait with the wakeup_pipe_drain() helper so the drain logic lives
in one place.

Close epfd on the epoll_create1-success, fork-failure path in
test-poll.c, and document why the SA_RESTART read test accepts EINTR:
elfuse has no transparent syscall restart, so like nanosleep/poll/select
it surfaces EINTR to the guest.

Close #77
vCPU-preemption wait path made only pipe and stdio reads interruptible.
A guest thread parked in a blocking socket recv/accept/connect, or a
fifo/tty read, stayed unreachable by a guest signal: no Ctrl-C, no
alarm+read timeout, no guest kill reaching the target, only killing the
host process worked.

Add a per-fd can_block flag, computed once via fstat in fd_init_entry
(true for anything that is not a regular file or directory), seeded for
the pre-opened stdin/stdout/stderr slots and fork-restored fds. It
replaces the scattered type == FD_PIPE || FD_STDIO checks with one "can
this fd block" predicate, so a fifo or char device classified FD_REGULAR
no longer slips past the wait without a risky reclassification.
type_may_block resolves pipe/socket/synthetic from the type alone, so
only regular-file and stdio slots pay the fstat.

Generalize the wait helper to io_wait_fd_or_interrupted(host_fd, events)
plus an io_block_wait gate, and wire it into read/write/readv/writev, the
syscall.c read/write fast-path divert (POLLOUT for writes too), and the
socket recv/accept/connect/send paths.

The read, write, recv, and accept paths do a single interruptible wait
then the plain host call: a regular file never blocks, and the tiny
window where the buffer refills or a sibling steals the data between the
wait and the call can still block briefly, matching Linux's lack of a
transparent restart here. The send paths additionally loop: they probe
with a per-call MSG_DONTWAIT and re-wait on EAGAIN, so a post-POLLOUT
buffer-full race retries instead of parking the vCPU. MSG_DONTWAIT is
used rather than toggling the fd's O_NONBLOCK because that flag rides the
shared open file description, so an fd toggle would hit a sibling thread
writing the same socket with a spurious EAGAIN. Recv keeps a single wait
so MSG_WAITALL still blocks for the full buffer. connect drives a
nonblocking connect + POLLOUT wait + SO_ERROR and restores the original
flags, falling back to a plain blocking connect if the socket cannot be
flipped.

Zero-length readv/writev still validate fd access mode via io_check_access
so a mismatched-direction fd returns EBADF as on Linux.

Coverage: test-poll gains a "signal interrupts blocking recv" case;
test-socket covers sendmsg/recvmsg, sendmmsg/recvmmsg, sendto/recvfrom,
MSG_TRUNC, and zero-length/invalid-iov edges; test-readv-writev gains an
all-empty-iovec access-check case.
@jserv jserv force-pushed the preemption-signal branch from a102a20 to c3de5dc Compare July 5, 2026 18:10
@jserv jserv merged commit f8a06d4 into main Jul 5, 2026
5 checks passed
@jserv jserv deleted the preemption-signal branch July 5, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Route vCPU preemption signals through a dedicated sigwait thread to eliminate HV_EXIT_REASON_UNKNOWN

2 participants