fix(hub): surface silently swallowed errors in hubsync hook and replication loop#133
Open
omergk28 wants to merge 1 commit into
Open
fix(hub): surface silently swallowed errors in hubsync hook and replication loop#133omergk28 wants to merge 1 commit into
omergk28 wants to merge 1 commit into
Conversation
…cation loop Two hub code paths swallowed errors with no logging surface — the session-start hubsync hook and the cluster replication loop. Operators could not tell a healthy quiet sync from one that never reached the network. Route every silent return through the warn sink with format constants in config/warn, and un-conflate the sync-error check from the empty-result check so only a real error warns; a genuine empty result stays silent. Bound the session-start pull with a 10s deadline (HubSyncTimeout) so a hub that accepts the connection but never responds cannot stall session start; the hook still returns "" on timeout like any other failure. The replication recv loop stays silent on io.EOF (normal stream end) and caller shutdown, and warns on anything else. Both functions keep their signatures and never-block contracts; the logging and the deadline are the only behavior changes. Resubmission of the work from PR ActiveMemory#114, which was swept closed during the DCO history-rewrite of main rather than rejected. Incorporates the final review round: - Drop the memory-file hunk from the PR; the deferred replication warn-cadence follow-up is recorded on main post-merge. - Fix the "next session retries" comments — the throttle marker is stamped unconditionally after the pull, so a cut-off pull retries on the next day's first session, not the next session. - Correct the Sync doc comment: it returns only a status string. - Keep the eof helper strict (==) for the clean-end callers in client.go; broaden to errors.Is inline only at the replication warn-suppression site, so each polarity gets the matching strictness. - Register the warn capture before the black-hole listener in the hung-hub test so LIFO cleanup cannot leak a late warning from a stranded goroutine into the next test. Closes ActiveMemory#100. Spec: specs/fix-hub-silent-error-suppression.md Signed-off-by: Omer Kocaoglu <omergk28@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resubmission of #114, which was swept closed during the DCO history-rewrite of
main(batch-closed alongside #118/#128/#130/#131, immediately before themainforce-push) rather than rejected. The last review was "the code fix is right; one more small push and this merges" — this PR is that push, rebased clean onto the rewrittenmainwith the final review round applied.Two hub code paths swallowed errors with no logging surface — the session-start hubsync hook (
internal/cli/system/core/hubsync) and the cluster replication loop (internal/hub/replicate.go). Operators couldn't tell a healthy quiet sync from one that never reached the network.This wires every silent return through the established
warnsink with format constants inconfig/warn, un-conflates the sync-error check from the empty-result check (only the error warns), and bounds the session-start pull with a 10s deadline (HubSyncTimeout) so a hung-but-reachable hub can't stall session start. Both functions keep their signatures and never-block contracts.Closes #100.
Final review round applied (from #114)
mergeable: dirty.context/TASKS.mdhunk entirely; the deferred replication warn-cadence follow-up (F2) is recorded onmainpost-merge per the standing contributor-PR pattern.sync.go,config/hub/hub.go). Also fixedSync's doc comment — it returns only a status string, not a count.eof()strict; inline the broadeningeof()to strict==(protects the clean-end callersclient.Sync/client.Listenfrom treating a future wrapped dirty-EOF as clean success). Broadened toerrors.Isinline only at thereplicate.gowarn-suppression site — each polarity gets its matching strictness.captureWarnings(t)before the black-hole listener, so LIFO cleanup closes the listener's connections before restoring the process-global warn sink — a strandedSyncgoroutine on the failure path can no longer bleed a late warning into the next test.Deliberate deviation from issue #100's proposed code (unchanged from #114)
Issue #100's sketch warns on every
RecvMsgerror — butio.EOFis the normal end of every sync stream, so that would emit a warning once perReplicateIntervalon perfectly healthy replication. The receive path stays silent forio.EOFand caller shutdown (ctx.Err() != nil) and warns on everything else.TestReplicateOnce_CleanReplicationDoesNotWarnpins this.Tests
TestSync_WarnsOnLoadError,TestSync_WarnsOnDialError,TestSync_WarnsOnPullError— each error path asserts its warning.TestReplicateOnce_WarnsOnDialError,TestReplicateOnce_WarnsOnTransportError— replication error paths assert their warnings.TestSync_NoWarnOnEmptyResult— pins the un-conflation (genuine empty result stays silent).TestSync_WarnsOnHungHub— black-hole TCP listener (accepts, never speaks gRPC) proves the deadline returns rather than hanging. Reverting the deadline tocontext.Background()makes it fail at its 5s guard.TestReplicateOnce_CleanReplicationDoesNotWarn— zero warnings on a real round-trip +lastSeq == 2.TestReplicateOnce_KeepsConsumingAfterAppendError— continue-on-append-failure.make build,make lint(0 issues), andmake test(full suite) all green locally.