Skip to content

fix: range-aware passthrough for Scylla manifest UploadPartCopy#122

Merged
ServerSideHannes merged 1 commit into
mainfrom
fix/scylla-manifest-range-passthrough
Jul 8, 2026
Merged

fix: range-aware passthrough for Scylla manifest UploadPartCopy#122
ServerSideHannes merged 1 commit into
mainfrom
fix/scylla-manifest-range-passthrough

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

Summary

  • Root cause of 2026.7.14 still streaming: Scylla sends bytes=0-(~4768MB-1) on .sm_* manifests while source big-Data.db sidecar total_plaintext_size is ~6GB. PR fix: passthrough Scylla manifest UploadPartCopy with full-object range #121 only cleared ranges matching metadata total, so _can_passthrough_part_copy still rejected any range → UPLOAD_PART_COPY_STREAMING → 300s timeout → HAProxy 400.
  • Fix: Large ranged copies from offset 0 that align to internal ciphertext segments now use range-aware passthrough (server-side upload_part_copy per segment, no re-encrypt).
  • Observability: New UPLOAD_PART_COPY_ROUTE log with route, raw_copy_source_range, metadata_total_plaintext_mb, range_plaintext_mb, and passthrough_blocked_reason.

Tests

  • test_scylla_prod_shape_range_smaller_than_metadata_uses_passthrough — 149 internal parts, metadata inflated 1.27× (prod shape), range smaller than metadata total → passthrough not streaming
  • test_prod_shape_passthrough_eligibility_and_route — eligibility + normalize does not strip Scylla range
  • test_segments_for_plaintext_range_* — segment alignment helper
  • test_small_partial_range_blocked_from_passthrough — partial 32MB range still re-encrypts
  • All existing passthrough + copy pipeline unit tests pass (24)

Post-deploy verification

kubectl logs -n s3proxy-python -l app.kubernetes.io/name=s3proxy-python --since=30m \
  | rg 'UPLOAD_PART_COPY_ROUTE|UPLOAD_PART_COPY_PASSTHROUGH'

Expect route=passthrough and UPLOAD_PART_COPY_PASSTHROUGH on *.sm_* manifest keys; UPLOAD_PART_COPY_STREAMING count should drop sharply.

Made with Cursor

PR #121 only normalized ranges that matched metadata total_plaintext_size.
Prod sends bytes=0-(~4768MB-1) while sidecar metadata totals ~6GB, so copies
still routed to UPLOAD_PART_COPY_STREAMING and timed out at 300s.

Allow large ranged copies from offset 0 when they align to internal ciphertext
segments, filter segments for passthrough, and log UPLOAD_PART_COPY_ROUTE with
passthrough_blocked_reason for future diagnosis.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ServerSideHannes ServerSideHannes merged commit 832a13e into main Jul 8, 2026
10 checks passed
@ServerSideHannes ServerSideHannes deleted the fix/scylla-manifest-range-passthrough branch July 8, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant