support zcrx

[wokron]

We will soon have device-less mode for zcrx.

for linux 7.1

axboe/liburing#1527

[wokron]

https://lore.kernel.org/io-uring/177430525401.475832.17980460579260076952.b4-ty@b4/T/#m2fb993271a2b63f9b11f32a5a6e93bee25d86ff0

[wokron]

/review

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Possible Issue In the destructor ~ZeroCopyRxBufferPool(), the munmap of rq_ring_.ring_ptr is performed unconditionally, but the mmap that sets rq_ring_.ring_ptr is only called inside register_ifq_(). If the constructor throws an exception after register_ifq_() succeeds but before rq_ring_.ring_ptr is assigned (e.g., after the mmap call but before the assignment completes), the destructor will still run and attempt to munmap an uninitialized or partially initialized pointer. However, the more realistic scenario is that if register_ifq_() throws, the destructor runs and rq_ring_.ring_ptr is zero-initialized, leading to munmap of a null pointer, which is undefined behavior. ~ZeroCopyRxBufferPool() { [[maybe_unused]] int r; if (area_size_ > 0) { assert(area_ptr_ != nullptr); r = munmap(area_ptr_, area_size_); assert(r == 0); } assert(rq_ring_.ring_ptr != nullptr); r = munmap(rq_ring_.ring_ptr, ring_size_); assert(r == 0); // TODO: Unregister ifq } Resource Leak In register_ifq_(), if the mmap call for the refill ring fails (line 226-231), the function throws an exception. However, the io_uring_register_ifq call on line 220 has already succeeded, registering the interface queue with the kernel. There is no mechanism to unregister the ifq on this error path, leaving the kernel with a dangling registration. The TODO comment on line 224 acknowledges this but does not implement the cleanup. void register_ifq_(uint32_t if_idx, uint32_t if_rxq, uint32_t rq_entries, io_uring_zcrx_area_reg &area_reg, size_t page_size) { rq_entries = std::bit_ceil(rq_entries); io_uring_region_desc region_reg = {}; ring_size_ = get_refill_ring_size_(rq_entries, page_size); region_reg.user_addr = 0; region_reg.size = ring_size_; region_reg.flags = 0; io_uring_zcrx_ifq_reg reg = {}; reg.if_idx = if_idx; reg.if_rxq = if_rxq; reg.rq_entries = rq_entries; reg.area_ptr = reinterpret_cast<uint64_t>(&area_reg); reg.region_ptr = reinterpret_cast<uint64_t>(&region_reg); reg.flags = flags_; int r = io_uring_register_ifq(ring_->ring(), &reg); if (r != 0) { throw detail::make_system_error("io_uring_register_ifq", -r); } // TODO: unregister ifq if any exception void *ring_ptr = mmap(nullptr, ring_size_, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, ring_->ring()->ring_fd, static_cast<off_t>(region_reg.mmap_offset)); if (ring_ptr == MAP_FAILED) { throw detail::make_system_error("mmap"); } rq_ring_.khead = (unsigned int *)((char *)ring_ptr + reg.offsets.head); rq_ring_.ktail = (unsigned int *)((char *)ring_ptr + reg.offsets.tail); rq_ring_.rqes = (struct io_uring_zcrx_rqe *)((char *)ring_ptr + reg.offsets.rqes); rq_ring_.rq_tail = 0; rq_ring_.ring_entries = reg.rq_entries; rq_ring_.ring_ptr = ring_ptr; zcrx_id_ = reg.zcrx_id; area_token_ = area_reg.rq_area_token; } Possible Issue In the handle_finish method (line 185-195), the code accesses cqe + 1 to get the io_uring_zcrx_cqe. This assumes that the CQE is followed by an extended CQE structure. This is only valid if the ring was set up with CQE32 enabled (i.e., IORING_SETUP_CQE32). If the user creates a ZeroCopyRxBufferPool without enabling CQE32 on the runtime, this access will read out-of-bounds memory, leading to undefined behavior. The documentation and test do enable CQE32, but the pool constructor does not enforce or check this requirement. ZeroCopyRxBuffer handle_finish(io_uring_cqe *cqe) noexcept { if (cqe->res < 0) { return ZeroCopyRxBuffer(); } io_uring_zcrx_cqe *rcqe = reinterpret_cast<io_uring_zcrx_cqe *>(cqe + 1); void *data = static_cast<char *>(area_ptr_) + (rcqe->off & ~IORING_ZCRX_AREA_MASK); size_t size = static_cast<size_t>(cqe->res); return ZeroCopyRxBuffer(data, size, this); }