Manually deserialize debug traces without recursion limit#4615
Conversation
|
Claude finished @MartinquaXD's task in 2m 34s —— View job PR Review: Manually deserialize debug traces without recursion limitApproach looks solid — manually issuing Two inline comments, neither blocking:
|
There was a problem hiding this comment.
Code Review
This pull request updates serde_json to use the unbounded_depth feature and introduces a fetch_debug_trace helper function to handle transactions with exceptionally deep call stacks by disabling the deserializer's recursion limit. The feedback suggests explicitly enabling the raw_value feature for serde_json in Cargo.toml to ensure compilation stability since to_raw_value is now used.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| .await?; | ||
|
|
||
| let mut de = serde_json::Deserializer::from_str(raw.get()); | ||
| de.disable_recursion_limit(); |
There was a problem hiding this comment.
disable_recursion_limit() removes serde's guard against stack overflow, so instead of failing cleanly at depth 128 an even deeper trace can now overflow the worker thread's stack and abort the whole autopilot process (a hard crash rather than a recoverable Error). Since serde_json only offers on/off (no configurable limit) and the response comes from our own trusted node, this is a reasonable trade-off — just flagging that the failure mode changed from "graceful error" to "process crash".
Two related notes:
- The unit test only exercises
fetch_debug_trace, butinto_domain->From<CallFrame>(domain/blockchain.rs:25) recurses to the same depth. In practice a.into()frame is far smaller than a serde frame, so if deserialize survives the conversion almost certainly does too, and the downstream traversal (find_settlement_trace_and_callers) is iterative (BFS), so that part is safe. - If you want defence-in-depth against a future even-deeper tx, running this deserialization on a dedicated thread with a large stack would prevent a crash — optional, likely overkill for this analysis path.
There was a problem hiding this comment.
Added this to the PR description:
Note that ethereum's transaction gas limit and the fact that each call only gets 63/64 of the remaining gas already limits the size of the JSON considerably so I don't think we have to worry about any crazy memory usage.
Also so far there was literally only 1 tx that needed the limit lifted in the first place.
jmg-duarte
left a comment
There was a problem hiding this comment.
LGTM, I'm only afraid that this can crash the process (I'm having an hard time validating this though, I'd expect the tokio thread to panic/crash, be removed and replaced)
| let params = serde_json::value::to_raw_value(&( | ||
| hash.0, | ||
| GethDebugTracingOptions::new_tracer(GethDebugBuiltInTracerType::CallTracer), | ||
| )) | ||
| .expect("serialization of known-good types"); | ||
|
|
||
| let raw = provider | ||
| .raw_request_dyn("debug_traceTransaction".into(), ¶ms) | ||
| .await?; |
There was a problem hiding this comment.
we need this to bypass the alloy serde deserialization?
There was a problem hiding this comment.
Yes. Otherwise alloy will create a Deserializer internally with the default recursion limit.
Arguably the recursion is less of an issue than a transaction that just uses all the possible gas. Since we cant work around that either I think we can worry about this if it ever causes issues. |
|
Asked Claude to double-check: Stack overflow is not a panic tokio can absorb by replacing the thread, it's fatal runtime error: stack overflow, aborting, the whole process dies. On a 2 MiB tokio worker stack (release build) this parse aborts at ~610 nested call frames. The Base tx is 225 deep and used 1.4M gas, so roughly 4M gas of the same pattern crosses the line, and the observer would then crash-loop autopilot by re-picking the settlement every run. Parsing CallFrame directly (into_domain bails on other variants anyway) wrapped in serde_stacker survived 10000 frames. serde_stacker alone does not help, untagged GethTrace re-walks a buffered copy of the tree outside its protection. let mut de = serde_json::Deserializer::from_str(raw.get());
de.disable_recursion_limit();
let de = serde_stacker::Deserializer::new(&mut de);
let frame = CallFrame::deserialize(de).map_err(Error::DeserializationFailed)?;
Ok(GethTrace::CallTracer(frame))Wdyt? |
Description
In order to find out which solver ultimately is responsible we need to get the debug traces of the settlement tx and analyze them.
This step fails for a base tx which apparently ran into endless recursion but was successful anyway due to hitting serde's default recursion limit of 128.
Changes
"manually" call
debug_traceTransactionsuch that we can deserialize the response bytes by ourselves with a deserializer that has its recursion limit lifted.Note that ethereum's transaction gas limit and the fact that each call only gets 63/64 of the remaining gas already limits the size of the JSON considerably so I don't think we have to worry about any crazy memory usage.
Also so far there was literally only 1 tx that needed the limit lifted in the first place.
How to test
tested it with a unit test verifying that the tx can now be deserialized. The test is not checked in because it requires an actual node which can be down.