2026.7.15

Fix Scylla backup manifest UploadPartCopy still streaming after 2026.7.14.

Problem: PR #121 only normalized ranges matching sidecar total_plaintext_size. Scylla Manager sends bytes=0-(~4768MB-1) on .sm_* manifests while source big-Data.db metadata totals ~6GB. Copies still routed to UPLOAD_PART_COPY_STREAMING, queued behind the 2-slot pipeline, and timed out at ~300s → HAProxy 400.

Fix (#122):

• Range-aware passthrough for large copies from offset 0 when the range aligns to internal ciphertext segments
• Server-side upload_part_copy per segment (no decrypt/re-encrypt) instead of streaming
• UPLOAD_PART_COPY_ROUTE log with route, range sizes, and passthrough_blocked_reason for diagnosis

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.15

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.15

Post-deploy: Scylla manifest uploads (*.sm_*) should log UPLOAD_PART_COPY_ROUTE route=passthrough and UPLOAD_PART_COPY_PASSTHROUGH. Upgrade from 2026.7.14 if backups still show 400 @ ~300s on manifest copies.

2026.7.14

Fix Scylla backup manifest UploadPartCopy 400s at ~300s.

Problem: Scylla Manager sends UploadPartCopy with x-amz-copy-source-range: bytes=0-(size-1) on .sm_* manifest parts referencing already-uploaded encrypted SSTs (big-Data.db). s3proxy treated the full-object range as a partial copy, forcing UPLOAD_PART_COPY_STREAMING (decrypt/re-encrypt). Requests queued behind the 2-slot copy pipeline for many minutes while Scylla Manager timed out at ~300s → HAProxy 400.

Fix (#121):

• Normalize full-object CopySourceRange to whole-object copy → UPLOAD_PART_COPY_PASSTHROUGH (server-side ciphertext copy, no re-encrypt)
• Remove streaming pipeline semaphore from passthrough path so manifest copies are not blocked behind slow re-encrypt jobs
• Partial ranges still use streaming re-encrypt as before

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.14

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.14

Post-deploy: Scylla backup manifest uploads (*.sm_*) should log UPLOAD_PART_COPY_PASSTHROUGH and return 200 instead of 400 @ ~300s. Upgrade from 2026.7.13 if Scylla backups are failing on large manifest copies.

2026.7.13

Hotfix for Scylla Manager backup 403s after #116.

Problem: Scylla Manager uploads SSTs via presigned PUT URLs (X-Amz-Signature in query). #116 treated missing x-amz-content-sha256 as deferred header-auth, routing presigned requests through prepare_header_auth → 403 Forbidden.

Fix:

• Skip body preload and deferred signature when query contains X-Amz-Signature (or legacy Signature)
• Presigned large PUTs verify immediately via verifier.verify() and stream without double-buffering

Also in this release:

• E2E tests: generate_presigned_url SigV4 + legacy V2 through full proxy stack
• CI fix: isolated MinIO for heavy copy tests (disk exhaustion)

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.13

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.13

Post-deploy: Scylla backup PUTs should return 200. Upgrade from 2026.7.12 if backups are failing with 403.

2026.7.12

Deferred SigV4 for large Scylla SST PUTs — fixes s3proxy OOM during backups (#116).

Problem: Scylla Manager uploads ~179MB SSTables with SigV4 but without x-amz-content-sha256. s3proxy preloaded the full body for signature verification, then streamed it again for encryption — ~2× body size in memory per request. With concurrent backup nodes in a 1Gi pod, RSS hit ~965MB → OOMKilled (exit 137).

Fix:

• Deferred SigV4 for large PUTs (>8MB) missing x-amz-content-sha256: verify credentials up front, hash body once while streaming encrypt/upload, verify signature before complete
• Single-pass body iterator — no double-buffer from request.body() + request.stream()
• Stop polling is_disconnected() during body streaming (Starlette can drop bytes and corrupt SHA256)
• Dockerfile uses python -m s3proxy.main so TLS settings from Settings apply in prod

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.12

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.12

Post-deploy check: logs should show signature_deferred, not body_loaded body_size=187836908.

2026.7.11

UploadPartCopy passthrough — fixes s3proxy OOM during Scylla backup dedup (#115).

Problem: Dedup copies use UploadPartCopy on encrypted SSTables. The re-encrypt path downloaded, decrypted, and re-encrypted each ~1280MB object (~88MB peak per internal part × concurrent copies). With 13 agents and maxConnPerPod: 20, pods OOMKilled (exit 137) and backups showed Failed ≈ Deduplicated.

Fix:

• Passthrough for eligible copies: encrypted multipart sidecar source, full-object copy (no CopySourceRange), same credential — native server-side upload_part_copy of existing ciphertext with source DEK adoption (no decrypt/re-encrypt amplification)
• Hold governor reservation through upload_part completion (release after ciphertext is handed off)
• Default S3PROXY_MAX_PARALLEL_COPIES=2 pipeline semaphore per pod

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.11

Verify after rollout:

# During Scylla backup dedup: Failed should stop tracking Deduplicated
kubectl exec -n scylla-manager deploy/scylla-manager -- sctool progress <task-id> -c searchengine

# s3proxy OOMKills should drop; passthrough log lines appear
kubectl -n s3proxy-python logs deploy/s3proxy-python | grep UPLOAD_PART_COPY_PASSTHROUGH

Pre-merge gate:

make verify-passthrough

No config changes required. Encryption at rest unchanged — ciphertext is byte-copied, not decrypted to disk.

2026.7.10

Per-part copy memory reservation — fixes sustained 176MB governor hold during Scylla backup (#114).

Problem: #113 fixed O(1) peak sizing (~88MB per copy) but still reserved that amount for the entire copy duration (40 internal parts × ~24s for a 1280MB SSTable). Two concurrent copies held ~176MB continuously (active_mb=176.38), blocking a third and causing MEMORY_BACKPRESSURE even while INTERNAL_PART_UPLOADED events kept firing.

Fix: Streaming UploadPartCopy now acquires/releases governor memory per 32MB internal part (~88MB peak during encrypt only). Memory is free during S3 upload I/O between parts, so concurrent copies interleave within the 192MB budget.

Also: legacy encrypted objects without isec-kid now pass through byte-identically on COPY instead of failing re-encrypt.

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.10

Verify after rollout:

# Governor sawtooth during backup (not flat ~176MB)
kubectl -n s3proxy-python port-forward deploy/s3proxy-python 9090:9090 &
watch -n1 'curl -s localhost:9090/metrics | grep s3proxy_memory_reserved_bytes'

Or run pre-merge gate locally:

make verify-copy-memory

No config changes required. Keep S3PROXY_MEMORY_LIMIT_MB=192.

2026.7.9

Fix Scylla backup 503 storm: server-side copies are now O(1) memory (#113).

Problem: Scylla dedup UploadPartCopy of ~4.7GB SSTables sized internal parts as object_size/20 (~238MB), so each copy peaked at ~535MB — larger than the 192MB governor budget. The limiter clamped the reservation to the whole budget (exclusive slot), which could only be admitted when active_bytes == 0. Under steady backup traffic that never happened → endless MEMORY_BACKPRESSURE (requested_mb == limit_mb) → 503 SlowDown (~50% failure rate observed in prod).

Fix: Copies frame at a fixed 32MB internal part size (crypto.copy_internal_part_size), so copy peak is ~88MB regardless of object size. Copies fit the budget outright, never monopolize it, and run concurrently like normal requests. Large copies allocate internal parts sequentially (safe: prod copies are all partNumber=1, reassembly is metadata-driven).

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.9

Verify after rollout:

kubectl -n s3proxy-python exec deploy/s3proxy-python -- python3 -c "
from s3proxy import crypto; MB=1024**2
print('peak_4767MB:', round(crypto.copy_pipeline_peak(4767*MB)/MB,1), 'MB')
print('has copy_internal:', hasattr(crypto,'copy_internal_part_size'))
"

Expected: peak_4767MB: 88.0, has copy_internal: True

No config changes required. Keep S3PROXY_MEMORY_LIMIT_MB=192 — copies no longer need a higher budget.

2026.7.8

Fix copy starvation in the memory governor (#112).

• A budget-monopolizing server-side COPY (multi-GB manifest — reserves the whole governor budget to run exclusively) could be starved indefinitely by a steady stream of small requests: it backpressured (requested_mb == limit_mb) until it hit the timeout and 503'd.
• Add writer preference: while an exclusive copy is waiting, new non-exclusive requests are held back so active memory drains to 0 and the copy gets its slot (bounded by in-flight request lifetimes).
• Copies still reserve the whole budget and run one-at-a-time; the multi-copy OOM protection (#110) is unchanged.

No config or migration changes.

2026.7.7

Expose HAProxy Prometheus metrics on the bundled frontproxy (#111).

• Enable HAProxy's native /metrics endpoint on s3proxy-python-frontproxy (port 8404)
• Expose haproxy_backend_current_queue, haproxy_server_current_queue, haproxy_frontend_current_sessions, etc.
• Add a Prometheus Operator ServiceMonitor for Thanos scraping (configurable via frontproxy.metrics.*)

Backwards compatible (observability only; S3 traffic path unchanged). No migration.

2026.7.6

Fix copy-path memory governor under-reserve for multi-GB manifest copies (#110).

• Add copy_governor_clamped_reserve() so large server-side copies monopolize the governor budget instead of being crushed to the routine upload peak (~59MB)
• Route UploadPartCopy and CopyObject through reserve_copy_memory() instead of upload-style reserve_memory()
• Tighten copy_pipeline_peak() for large internal parts (add part // 7 slack for aiobotocore signing buffers on multi-GB manifests)
• Shard integration CI into parallel matrix jobs; update branch ruleset required checks accordingly

Backwards compatible (copy-path reservation logic only). No migration.