Route vCPU preemption signals via sigwait thread#145
Merged
Conversation
Collaborator
|
below paths not covered Sockets
FIFO and pty/character devices (both verified): Consequences:
|
elfuse preempted a running vCPU by sending a host signal whose handler called hv_vcpus_exit(): SIGUSR2 for the cross-process guest-signal doorbell and SIGALRM for the per-iteration safety timeout. Because the signal landed on the vCPU thread inside hv_vcpu_run, Apple HVF aborted the run with HV_EXIT_REASON_UNKNOWN instead of the clean CANCELED that hv_vcpus_exit() produces for a vCPU caught between runs, leaving the run loop unable to tell a self-directed preemption from a genuine fault. Block SIGUSR2 and SIGALRM on the main thread before any vCPU thread is created (proc_preempt_init), so every CLONE_THREAD worker and posix_spawn fork-child inherits the block. A dedicated thread sigwaits both and kicks every live vCPU via thread_interrupt_all() from a thread that never runs one, so hv_vcpu_run always returns CANCELED and the run loop treats any HV_EXIT_REASON_UNKNOWN as a hard hypervisor fault. The g_timed_out and g_external_guest_signal handoff flags become _Atomic with release/acquire ordering now that they cross the preempt-thread to vCPU-thread boundary. Make a guest thread parked in a blocking host syscall reachable by a signal, which hv_vcpus_exit() alone cannot interrupt: - attention_raise() pokes the wakeup pipe on every deliverable signal. - ppoll/pselect/epoll_pwait break out on signal_pending_interruption. - a blocking pipe/stdio read waits in wait_readable_or_interrupted, which returns EINTR only for a guest-visible delivery, skips O_WRONLY and O_NONBLOCK fds, and uses a bounded 200ms recheck; the non-verbose fast-path read diverts a would-block pipe/stdio read to this slow path. - fork_child_main creates the wakeup pipe, which it skips via syscall_init. Make the signal transport race-free: the sender holds flock around its append and the drain holds it across a full read plus ftruncate and no longer unlinks, so a concurrent append is never routed to an orphaned inode. Each process unlinks its own empty file at exit. signal_pending_interruption() reports whether a guest-visible delivery is pending and excludes ignored and default-ignore signals, so read, poll, select, and epoll no longer wake spuriously on an ignored signal. The fork-child reaches syscall_init() indirectly through fork_ipc_recv_fd_table() before the vCPU loop runs, so it already creates the process-local wakeup pipe. The explicit wakeup_pipe_init() in fork_child_main was therefore redundant and, worse, leaked two fds per fork: the later syscall_init() call overwrote wakeup_pipe_rd and wakeup_pipe_wr without closing the first pair. Drop the redundant call and make wakeup_pipe_init() idempotent so any second reach is a no-op rather than a leak. Route a fast-path pipe/stdio read to the interruptible slow path on a probe error (poll() < 0), not only when the fd is not ready, so a transient poll failure cannot leave the vCPU thread in a blocking host read() that the preempt thread cannot break. Replace the three inline wakeup-pipe drain loops in ppoll, pselect, and epoll_pwait with the wakeup_pipe_drain() helper so the drain logic lives in one place. Close epfd on the epoll_create1-success, fork-failure path in test-poll.c, and document why the SA_RESTART read test accepts EINTR: elfuse has no transparent syscall restart, so like nanosleep/poll/select it surfaces EINTR to the guest. Close #77
vCPU-preemption wait path made only pipe and stdio reads interruptible. A guest thread parked in a blocking socket recv/accept/connect, or a fifo/tty read, stayed unreachable by a guest signal: no Ctrl-C, no alarm+read timeout, no guest kill reaching the target, only killing the host process worked. Add a per-fd can_block flag, computed once via fstat in fd_init_entry (true for anything that is not a regular file or directory), seeded for the pre-opened stdin/stdout/stderr slots and fork-restored fds. It replaces the scattered type == FD_PIPE || FD_STDIO checks with one "can this fd block" predicate, so a fifo or char device classified FD_REGULAR no longer slips past the wait without a risky reclassification. type_may_block resolves pipe/socket/synthetic from the type alone, so only regular-file and stdio slots pay the fstat. Generalize the wait helper to io_wait_fd_or_interrupted(host_fd, events) plus an io_block_wait gate, and wire it into read/write/readv/writev, the syscall.c read/write fast-path divert (POLLOUT for writes too), and the socket recv/accept/connect/send paths. The read, write, recv, and accept paths do a single interruptible wait then the plain host call: a regular file never blocks, and the tiny window where the buffer refills or a sibling steals the data between the wait and the call can still block briefly, matching Linux's lack of a transparent restart here. The send paths additionally loop: they probe with a per-call MSG_DONTWAIT and re-wait on EAGAIN, so a post-POLLOUT buffer-full race retries instead of parking the vCPU. MSG_DONTWAIT is used rather than toggling the fd's O_NONBLOCK because that flag rides the shared open file description, so an fd toggle would hit a sibling thread writing the same socket with a spurious EAGAIN. Recv keeps a single wait so MSG_WAITALL still blocks for the full buffer. connect drives a nonblocking connect + POLLOUT wait + SO_ERROR and restores the original flags, falling back to a plain blocking connect if the socket cannot be flipped. Zero-length readv/writev still validate fd access mode via io_check_access so a mismatched-direction fd returns EBADF as on Linux. Coverage: test-poll gains a "signal interrupts blocking recv" case; test-socket covers sendmsg/recvmsg, sendmmsg/recvmmsg, sendto/recvfrom, MSG_TRUNC, and zero-length/invalid-iov edges; test-readv-writev gains an all-empty-iovec access-check case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
elfuse preempted a running vCPU by sending a host signal whose handler called hv_vcpus_exit(): SIGUSR2 for the cross-process guest-signal doorbell and SIGALRM for the per-iteration safety timeout. Because the signal landed on the vCPU thread inside hv_vcpu_run, Apple HVF aborted the run with HV_EXIT_REASON_UNKNOWN instead of the clean CANCELED that hv_vcpus_exit() produces for a vCPU caught between runs, leaving the run loop unable to tell a self-directed preemption from a genuine fault.
Block SIGUSR2 and SIGALRM on the main thread before any vCPU thread is created (proc_preempt_init), so every CLONE_THREAD worker and posix_spawn fork-child inherits the block. A dedicated thread sigwaits both and kicks every live vCPU via thread_interrupt_all() from a thread that never runs one, so hv_vcpu_run always returns CANCELED and the run loop treats any HV_EXIT_REASON_UNKNOWN as a hard hypervisor fault. The g_timed_out and g_external_guest_signal handoff flags become _Atomic with release/acquire ordering now that they cross the preempt-thread to vCPU-thread boundary.
Make a guest thread parked in a blocking host syscall reachable by a signal, which hv_vcpus_exit() alone cannot interrupt:
Make the signal transport race-free: the sender holds flock around its append and the drain holds it across a full read plus ftruncate and no longer unlinks, so a concurrent append is never routed to an orphaned inode. Each process unlinks its own empty file at exit.
signal_pending_interruption() reports whether a guest-visible delivery is pending and excludes ignored and default-ignore signals, so read, poll, select, and epoll no longer wake spuriously on an ignored signal.
Close #77
Summary by cubic
Route vCPU preemption (SIGUSR2/SIGALRM) through a dedicated sigwait thread so hv_vcpu_run reliably returns CANCELED on HVF, and expand interruptible blocking I/O so reads, writes, and socket ops wake only for guest-visible signals. Also harden cross-process signal transport and clean up wakeup-pipe handling.
test-pollcovers ignored SIGCHLD and that SA_RESTART read handlers run and unblock recv;test-socketadds zero-length/invalid recvmsg cases;test-readv-writevadds an all-empty-iovec access-check.Written for commit c3de5dc. Summary will update on new commits.