Skip to content

Bound the ring walk of ParentConsistentHash::selectParent#13368

Open
masaori335 wants to merge 1 commit into
apache:masterfrom
masaori335:asf-master-consistent-hash-parent
Open

Bound the ring walk of ParentConsistentHash::selectParent#13368
masaori335 wants to merge 1 commit into
apache:masterfrom
masaori335:asf-master-consistent-hash-parent

Conversation

@masaori335

Copy link
Copy Markdown
Contributor

When every parent in a consistent_hash pool is down, selectParent walked the whole hash ring taking the global host_status_rwlock on every hop. The ring holds 1024 replica nodes per parent (num_parents * 1024 nodes) and the chash_lookup() gate withholds wrap_around until the ring is traversed twice, so one all-down selection cost ~2 * num_parents * 1024 HostStatus::getHostStatus() calls (~49k for 24 parents) -- inline ET_NET CPU that starved the loopback health probe and drove the VIP flap in inc-p1s2-260703.

Track the distinct parents examined on each ring: skip the locked getHostStatus read for a parent already seen, and force wrap_around once every distinct parent has been rejected. The expensive locked read is now paid at most once per parent (O(num_parents)); the ring still advances ~O(N*logN) cheap, lock-free hops to reach every distinct parent. Selection order and the retry-window logic are unchanged.

The seen-parent tracking is sized to num_parents (std::vector), not MAX_PARENTS: the parent.config parser does not cap num_parents at MAX_PARENTS, so a fixed [MAX_PARENTS] array would overflow the stack for pools larger than 64.

Add consistent_hash_ring_walk.test.py: an all-down 100-parent pool (marked down via HostStatus, >MAX_PARENTS on purpose) must report "getHostStatus calls: 100", proving the walk reads each parent once instead of walking the full ring.

When every parent in a consistent_hash pool is down, selectParent walked the
whole hash ring taking the global host_status_rwlock on every hop. The ring
holds 1024 replica nodes per parent (num_parents * 1024 nodes) and the
chash_lookup() gate withholds wrap_around until the ring is traversed twice, so
one all-down selection cost ~2 * num_parents * 1024 HostStatus::getHostStatus()
calls (~49k for 24 parents) -- inline ET_NET CPU that starved the loopback
health probe and drove the VIP flap in inc-p1s2-260703.

Track the distinct parents examined on each ring: skip the locked getHostStatus
read for a parent already seen, and force wrap_around once every distinct parent
has been rejected. The expensive locked read is now paid at most once per parent
(O(num_parents)); the ring still advances ~O(N*logN) cheap, lock-free hops to
reach every distinct parent. Selection order and the retry-window logic are
unchanged.

The seen-parent tracking is sized to num_parents (std::vector<bool>), not
MAX_PARENTS: the parent.config parser does not cap num_parents at MAX_PARENTS, so
a fixed [MAX_PARENTS] array would overflow the stack for pools larger than 64.

Add consistent_hash_ring_walk.test.py: an all-down 100-parent pool (marked down
via HostStatus, >MAX_PARENTS on purpose) must report "getHostStatus calls: 100",
proving the walk reads each parent once instead of walking the full ring.
@masaori335 masaori335 added this to the 11.0.0 milestone Jul 7, 2026
@masaori335 masaori335 self-assigned this Jul 7, 2026
Copilot AI review requested due to automatic review settings July 7, 2026 23:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes ParentConsistentHash::selectParent() for the “all parents down” case by bounding the consistent-hash ring walk so each distinct parent’s HostStatus is read at most once per selection, avoiding excessive contention on the global host-status lock in ET_NET threads.

Changes:

  • Add per-selection tracking of “seen” parents to avoid repeated HostStatus::getHostStatus() calls while walking replica nodes on the hash ring.
  • Force wrap-around once all distinct parents in the active ring have been rejected, preventing multi-pass full-ring scans.
  • Add a gold test that constructs a >64-parent all-down consistent-hash pool and asserts bounded getHostStatus call counts.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/proxy/ParentConsistentHash.cc Adds seen-parent tracking and bounded ring-walk logic; emits per-selection getHostStatus call count under parent_select debug.
tests/gold_tests/parent_proxy/consistent_hash_ring_walk.test.py New regression test for all-down large pools; validates bounded getHostStatus calls and 502 behavior.

Comment thread tests/gold_tests/parent_proxy/consistent_hash_ring_walk.test.py
Comment thread tests/gold_tests/parent_proxy/consistent_hash_ring_walk.test.py
@masaori335

Copy link
Copy Markdown
Contributor Author

[approve ci autest 2of4]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants