Skip to content

Manually deserialize debug traces without recursion limit#4615

Merged
MartinquaXD merged 4 commits into
mainfrom
deserialize-deep-base-tx
Jul 10, 2026
Merged

Manually deserialize debug traces without recursion limit#4615
MartinquaXD merged 4 commits into
mainfrom
deserialize-deep-base-tx

Conversation

@MartinquaXD

@MartinquaXD MartinquaXD commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Description

In order to find out which solver ultimately is responsible we need to get the debug traces of the settlement tx and analyze them.
This step fails for a base tx which apparently ran into endless recursion but was successful anyway due to hitting serde's default recursion limit of 128.

Changes

"manually" call debug_traceTransaction such that we can deserialize the response bytes by ourselves with a deserializer that has its recursion limit lifted.

Note that ethereum's transaction gas limit and the fact that each call only gets 63/64 of the remaining gas already limits the size of the JSON considerably so I don't think we have to worry about any crazy memory usage.
Also so far there was literally only 1 tx that needed the limit lifted in the first place.

How to test

tested it with a unit test verifying that the tx can now be deserialized. The test is not checked in because it requires an actual node which can be down.

#[cfg(test)]
mod tests {
    use {super::*, alloy::primitives::b256};

    #[tokio::test]
    async fn test_base_tx() {
        let provider = ethrpc::Web3::new_from_url("http://localhost:8545");
        let traces = fetch_debug_trace(
            &provider.provider,
            eth::TxId(b256!(
                "0x5200aab12fbe4e0aef019748bf0f79266155fbbea00557bf1071fa2859e7eb9b"
            )),
        )
        .await
        .unwrap();
        dbg!(traces);
    }
}

@MartinquaXD MartinquaXD requested a review from a team as a code owner July 10, 2026 08:48
@claude

claude Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Claude finished @MartinquaXD's task in 2m 34s —— View job


PR Review: Manually deserialize debug traces without recursion limit

Approach looks solid — manually issuing debug_traceTransaction and deserializing with disable_recursion_limit() is the right way to work around alloy/serde's default 128 recursion cap, and the tuple (hash, options) serializes to the same positional params alloy would have sent. The unbounded_depth feature flag and the map_err(Error::Alloy) on the receipt arm to align try_join! error types are both correct.

Two inline comments, neither blocking:

  • Misleading error mapping — deserialize failures reuse IncompleteTransactionData ("missing field {0}, node client bug?"); worth adding context.
  • Failure mode changed to a crash — disabling the recursion limit means an even deeper future trace can stack-overflow and abort the process rather than error gracefully. Reasonable trade-off given the trusted node + no configurable-limit option in serde_json; just flagging it, with notes on the recursive From<CallFrame> conversion and an optional large-stack-thread mitigation.
    deserialize-deep-base-tx

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates serde_json to use the unbounded_depth feature and introduces a fetch_debug_trace helper function to handle transactions with exceptionally deep call stacks by disabling the deserializer's recursion limit. The feedback suggests explicitly enabling the raw_value feature for serde_json in Cargo.toml to ensure compilation stability since to_raw_value is now used.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread Cargo.toml Outdated
Comment thread crates/autopilot/src/infra/blockchain/mod.rs Outdated
.await?;

let mut de = serde_json::Deserializer::from_str(raw.get());
de.disable_recursion_limit();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

disable_recursion_limit() removes serde's guard against stack overflow, so instead of failing cleanly at depth 128 an even deeper trace can now overflow the worker thread's stack and abort the whole autopilot process (a hard crash rather than a recoverable Error). Since serde_json only offers on/off (no configurable limit) and the response comes from our own trusted node, this is a reasonable trade-off — just flagging that the failure mode changed from "graceful error" to "process crash".

Two related notes:

  • The unit test only exercises fetch_debug_trace, but into_domain -> From<CallFrame> (domain/blockchain.rs:25) recurses to the same depth. In practice a .into() frame is far smaller than a serde frame, so if deserialize survives the conversion almost certainly does too, and the downstream traversal (find_settlement_trace_and_callers) is iterative (BFS), so that part is safe.
  • If you want defence-in-depth against a future even-deeper tx, running this deserialization on a dedicated thread with a large stack would prevent a crash — optional, likely overkill for this analysis path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this to the PR description:

Note that ethereum's transaction gas limit and the fact that each call only gets 63/64 of the remaining gas already limits the size of the JSON considerably so I don't think we have to worry about any crazy memory usage.
Also so far there was literally only 1 tx that needed the limit lifted in the first place.

@jmg-duarte jmg-duarte left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I'm only afraid that this can crash the process (I'm having an hard time validating this though, I'd expect the tokio thread to panic/crash, be removed and replaced)

Comment on lines +152 to +160
let params = serde_json::value::to_raw_value(&(
hash.0,
GethDebugTracingOptions::new_tracer(GethDebugBuiltInTracerType::CallTracer),
))
.expect("serialization of known-good types");

let raw = provider
.raw_request_dyn("debug_traceTransaction".into(), &params)
.await?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need this to bypass the alloy serde deserialization?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Otherwise alloy will create a Deserializer internally with the default recursion limit.

@MartinquaXD

Copy link
Copy Markdown
Contributor Author

LGTM, I'm only afraid that this can crash the process (I'm having an hard time validating this though, I'd expect the tokio thread to panic/crash, be removed and replaced)

Arguably the recursion is less of an issue than a transaction that just uses all the possible gas. Since we cant work around that either I think we can worry about this if it ever causes issues.

@MartinquaXD MartinquaXD added this pull request to the merge queue Jul 10, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 10, 2026
@MartinquaXD MartinquaXD added this pull request to the merge queue Jul 10, 2026
Merged via the queue into main with commit 840a1bb Jul 10, 2026
22 checks passed
@MartinquaXD MartinquaXD deleted the deserialize-deep-base-tx branch July 10, 2026 09:44
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 10, 2026
@squadgazzz

Copy link
Copy Markdown
Contributor

Asked Claude to double-check:

Stack overflow is not a panic tokio can absorb by replacing the thread, it's fatal runtime error: stack overflow, aborting, the whole process dies. On a 2 MiB tokio worker stack (release build) this parse aborts at ~610 nested call frames. The Base tx is 225 deep and used 1.4M gas, so roughly 4M gas of the same pattern crosses the line, and the observer would then crash-loop autopilot by re-picking the settlement every run.

Parsing CallFrame directly (into_domain bails on other variants anyway) wrapped in serde_stacker survived 10000 frames. serde_stacker alone does not help, untagged GethTrace re-walks a buffered copy of the tree outside its protection.

let mut de = serde_json::Deserializer::from_str(raw.get());
de.disable_recursion_limit();
let de = serde_stacker::Deserializer::new(&mut de);
let frame = CallFrame::deserialize(de).map_err(Error::DeserializationFailed)?;
Ok(GethTrace::CallTracer(frame))

Wdyt?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants