Skip to content

fix(hub): surface silently swallowed errors in hubsync hook and replication loop#133

Open
omergk28 wants to merge 1 commit into
ActiveMemory:mainfrom
omergk28:fix/100-hub-silent-errors-resubmit
Open

fix(hub): surface silently swallowed errors in hubsync hook and replication loop#133
omergk28 wants to merge 1 commit into
ActiveMemory:mainfrom
omergk28:fix/100-hub-silent-errors-resubmit

Conversation

@omergk28

@omergk28 omergk28 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Resubmission of #114, which was swept closed during the DCO history-rewrite of main (batch-closed alongside #118/#128/#130/#131, immediately before the main force-push) rather than rejected. The last review was "the code fix is right; one more small push and this merges" — this PR is that push, rebased clean onto the rewritten main with the final review round applied.

Two hub code paths swallowed errors with no logging surface — the session-start hubsync hook (internal/cli/system/core/hubsync) and the cluster replication loop (internal/hub/replicate.go). Operators couldn't tell a healthy quiet sync from one that never reached the network.

This wires every silent return through the established warn sink with format constants in config/warn, un-conflates the sync-error check from the empty-result check (only the error warns), and bounds the session-start pull with a 10s deadline (HubSyncTimeout) so a hung-but-reachable hub can't stall session start. Both functions keep their signatures and never-block contracts.

Closes #100.

Final review round applied (from #114)

Item Resolution
M1 — memory-file hunk caused mergeable: dirty Dropped the .context/TASKS.md hunk entirely; the deferred replication warn-cadence follow-up (F2) is recorded on main post-merge per the standing contributor-PR pattern.
M2 — "next session retries" comments were false The throttle marker is stamped unconditionally after the pull, so a cut-off pull retries on the next day's first session, not the next session. Corrected both comment sites (sync.go, config/hub/hub.go). Also fixed Sync's doc comment — it returns only a status string, not a count.
N1 — keep eof() strict; inline the broadening Reverted eof() to strict == (protects the clean-end callers client.Sync/client.Listen from treating a future wrapped dirty-EOF as clean success). Broadened to errors.Is inline only at the replicate.go warn-suppression site — each polarity gets its matching strictness.
N2 — hung-hub test cleanup ordering Registered captureWarnings(t) before the black-hole listener, so LIFO cleanup closes the listener's connections before restoring the process-global warn sink — a stranded Sync goroutine on the failure path can no longer bleed a late warning into the next test.

Deliberate deviation from issue #100's proposed code (unchanged from #114)

Issue #100's sketch warns on every RecvMsg error — but io.EOF is the normal end of every sync stream, so that would emit a warning once per ReplicateInterval on perfectly healthy replication. The receive path stays silent for io.EOF and caller shutdown (ctx.Err() != nil) and warns on everything else. TestReplicateOnce_CleanReplicationDoesNotWarn pins this.

Tests

  • TestSync_WarnsOnLoadError, TestSync_WarnsOnDialError, TestSync_WarnsOnPullError — each error path asserts its warning.
  • TestReplicateOnce_WarnsOnDialError, TestReplicateOnce_WarnsOnTransportError — replication error paths assert their warnings.
  • TestSync_NoWarnOnEmptyResult — pins the un-conflation (genuine empty result stays silent).
  • TestSync_WarnsOnHungHub — black-hole TCP listener (accepts, never speaks gRPC) proves the deadline returns rather than hanging. Reverting the deadline to context.Background() makes it fail at its 5s guard.
  • TestReplicateOnce_CleanReplicationDoesNotWarn — zero warnings on a real round-trip + lastSeq == 2.
  • TestReplicateOnce_KeepsConsumingAfterAppendError — continue-on-append-failure.

make build, make lint (0 issues), and make test (full suite) all green locally.

…cation loop

Two hub code paths swallowed errors with no logging surface — the
session-start hubsync hook and the cluster replication loop. Operators
could not tell a healthy quiet sync from one that never reached the
network. Route every silent return through the warn sink with format
constants in config/warn, and un-conflate the sync-error check from the
empty-result check so only a real error warns; a genuine empty result
stays silent.

Bound the session-start pull with a 10s deadline (HubSyncTimeout) so a
hub that accepts the connection but never responds cannot stall session
start; the hook still returns "" on timeout like any other failure. The
replication recv loop stays silent on io.EOF (normal stream end) and
caller shutdown, and warns on anything else.

Both functions keep their signatures and never-block contracts; the
logging and the deadline are the only behavior changes.

Resubmission of the work from PR ActiveMemory#114, which was swept closed during the
DCO history-rewrite of main rather than rejected. Incorporates the final
review round:

- Drop the memory-file hunk from the PR; the deferred replication
  warn-cadence follow-up is recorded on main post-merge.
- Fix the "next session retries" comments — the throttle marker is
  stamped unconditionally after the pull, so a cut-off pull retries on
  the next day's first session, not the next session.
- Correct the Sync doc comment: it returns only a status string.
- Keep the eof helper strict (==) for the clean-end callers in
  client.go; broaden to errors.Is inline only at the replication
  warn-suppression site, so each polarity gets the matching strictness.
- Register the warn capture before the black-hole listener in the
  hung-hub test so LIFO cleanup cannot leak a late warning from a
  stranded goroutine into the next test.

Closes ActiveMemory#100.

Spec: specs/fix-hub-silent-error-suppression.md
Signed-off-by: Omer Kocaoglu <omergk28@gmail.com>
@omergk28 omergk28 requested a review from josealekhine as a code owner July 7, 2026 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hub: silent error suppression in hubsync hook + replication loop

1 participant