Skip to content

feat: stream cross-shard ORDER BY results with k-way merge#1114

Open
murex971 wants to merge 3 commits into
pgdogdev:mainfrom
murex971:nagrawal-merge-sort
Open

feat: stream cross-shard ORDER BY results with k-way merge#1114
murex971 wants to merge 3 commits into
pgdogdev:mainfrom
murex971:nagrawal-merge-sort

Conversation

@murex971

@murex971 murex971 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Description

  • Replace full in-memory buffering/sort for eligible cross-shard ORDER BY queries with a streaming k-way merge path.
  • Keep existing buffer/sort/aggregate path unchanged for non-ORDER BY-only cases (e.g. aggregate/distinct flows).

Tests

  • test_order_by_k_way_merge_streams_before_command_complete
  • test_order_by_with_distinct_uses_buffered_path
  • test_order_by_merge_waits_for_all_shards_heads
  • test_order_by_merge_honors_offset_and_limit

Related issues/comments

closes #207

@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.36277% with 32 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pgdog/src/backend/pool/connection/buffer.rs 87.20% 32 Missing ⚠️

📢 Thoughts on this report? Let us know!


read = true;
if let Some(message) = state.forward(message)? {
if let Some(message) = state.forward_from(position, message)? {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: rename position to shard

use super::Aggregates;

#[derive(Debug, Clone)]
struct OrderByMergeState {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would be good to move that to its own file.

@levkk

levkk commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Looks good. We would need quite a few integration tests for this one though before it's in a mergeable state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Query engine] Merge/sort for sorting

2 participants