chore(antithesis): Rework rig containers, setup by blt · Pull Request #2017 · DataDog/saluki

blt · 2026-07-06T17:50:55Z

Summary

This commit modifies the rig so that we are always using Datadog Agent +
ADP as the SUT. In the general case this is ADP-on and in the
differential case this is between ADP-off and ADP-on. We make one small
modification to the Datadog Agent container in that we do not restart
ADP if it faults, giving antithesis a cleaner exit signal.

I have tried to make all our containers sync'ed properly with the rest
of the project, as they had drifted. If I understand the auto-update
proecess correctly this should all stay square from now on. From Vector
I have ported the launch script and, hopefully, this makes it a little
more clear how to launch these things. Of interest I have changed the
endpoint and turned OFF node faults for both scenarios as we cannot
ensure that, especially in the differential case, that faults happen at
the same instant and are fair.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

How did you test this PR?

References

blt · 2026-07-06T17:51:04Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

pr-commenter · 2026-07-06T17:57:23Z

Regression Detector (Agent Data Plane)

Run ID: 3a1b3508-1a46-44a3-af7b-074ecad43f9f
Baseline: 2715682b · Comparison: 2e097d4b · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.10	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.01	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.01	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.09	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.28	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 132 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 42.9 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 64.7 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 192 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 29.1 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

tobz

Hmmm, do you have any examples of what things looked like before this?

It would make sense to me if we had an out-of-date pinned version of ca-certificates, but I'm not seeing how the versionless dependency reference would end up giving us an out-of-date version... and given that, pinning everything feels a bit labored/fragile.

Copilot

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated no new comments.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 64519d0a5c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

This commit updates the harness driver to build dogstatsd to a byte limit, in the manner of datadog/lading. I have been unable to investigate SMPTNG-7611 well owing to the error log emission by Datadog Agent. I've added property tests to assert the payload limit is obeyed, accepting that this means making the dogstatsd generator pure with regard to Rng and may not use antithesis SDK's random_* directly.

Copilot

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated 2 comments.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 726b570162

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 51d49beaef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 693767bc4d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 1 comment.

This commit modifies the rig so that we are always using Datadog Agent + ADP as the SUT. In the general case this is ADP-on and in the differential case this is between ADP-off and ADP-on. We make one small modification to the Datadog Agent container in that we do not restart ADP if it faults, giving antithesis a cleaner exit signal. I have tried to make all our containers sync'ed properly with the rest of the project, as they had drifted. If I understand the auto-update proecess correctly this should all stay square from now on. From Vector I have ported the launch script and, hopefully, this makes it a little more clear how to launch these things. Of interest I have changed the endpoint and turned OFF node faults for both scenarios as we cannot ensure that, especially in the differential case, that faults happen at the same instant and are fair.

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

This comment has been minimized.

Sign in to view

blt mentioned this pull request Jul 6, 2026

chore(antithesis): Make rig intake lenient #2039

Open

4 tasks

blt force-pushed the blt/chore_antithesis_dogstatsd_line_and_payload_generators branch from f116dac to c2c153c Compare July 7, 2026 12:35

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from c545860 to a41b667 Compare July 7, 2026 12:36

blt force-pushed the blt/chore_antithesis_dogstatsd_line_and_payload_generators branch from c2c153c to eb51b1e Compare July 7, 2026 12:48

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from a41b667 to 60b0777 Compare July 7, 2026 12:49

tobz reviewed Jul 7, 2026

View reviewed changes

blt force-pushed the blt/chore_antithesis_dogstatsd_line_and_payload_generators branch from eb51b1e to 672554d Compare July 7, 2026 13:14

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from 60b0777 to 321d5b0 Compare July 7, 2026 13:14

blt force-pushed the blt/chore_antithesis_dogstatsd_line_and_payload_generators branch from 672554d to e9fb9a3 Compare July 7, 2026 22:36

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from 321d5b0 to 2dbcec5 Compare July 7, 2026 22:36

blt force-pushed the blt/chore_antithesis_dogstatsd_line_and_payload_generators branch from e9fb9a3 to c5cbc5a Compare July 7, 2026 22:51

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from 2dbcec5 to 83e3b6f Compare July 7, 2026 22:51

blt force-pushed the blt/chore_antithesis_dogstatsd_line_and_payload_generators branch from c5cbc5a to 732b36f Compare July 7, 2026 23:06

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from 83e3b6f to f820609 Compare July 7, 2026 23:06

blt force-pushed the blt/chore_antithesis_dogstatsd_line_and_payload_generators branch from 732b36f to 65c8858 Compare July 8, 2026 00:59

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch 2 times, most recently from 0a91d85 to 8bfacdd Compare July 8, 2026 01:28

blt force-pushed the blt/chore_antithesis_dogstatsd_line_and_payload_generators branch from 65c8858 to 896334b Compare July 8, 2026 01:28

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from 8bfacdd to 4b25d9b Compare July 8, 2026 02:09

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from b5b8e49 to 64519d0 Compare July 9, 2026 18:39

blt marked this pull request as ready for review July 9, 2026 20:19

blt requested review from a team as code owners July 9, 2026 20:19

Copilot AI review requested due to automatic review settings July 9, 2026 20:19

Copilot started reviewing on behalf of blt July 9, 2026 20:19 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread test/antithesis/scenarios/general/docker-compose.yaml

Comment thread test/antithesis/bin/launch.sh Outdated

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from 64519d0 to 726b570 Compare July 9, 2026 23:32

Copilot AI review requested due to automatic review settings July 9, 2026 23:32

Copilot started reviewing on behalf of blt July 9, 2026 23:33 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread test/antithesis/AGENTS.md Outdated

Comment thread test/antithesis/harness/src/bin/first_sample_config/main.rs

Copilot AI review requested due to automatic review settings July 9, 2026 23:45

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from 726b570 to 5203963 Compare July 9, 2026 23:45

Copilot started reviewing on behalf of blt July 9, 2026 23:46 View session

chatgpt-codex-connector Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/config.rs Outdated

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread test/antithesis/scenarios/differential/docker-compose.yaml

Copilot AI review requested due to automatic review settings July 9, 2026 23:56

blt force-pushed the blt/chore_antithesis_pin_all_rig_container_dependencies branch from 5203963 to 51d49be Compare July 9, 2026 23:56

Copilot started reviewing on behalf of blt July 9, 2026 23:57 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/config.rs Outdated

Comment thread test/antithesis/harness/src/config.rs Outdated

Copilot AI review requested due to automatic review settings July 10, 2026 00:16

Copilot AI reviewed Jul 10, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread test/antithesis/data-plane-finish Outdated

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/config.rs Outdated

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI reviewed Jul 10, 2026

Uh oh!

Conversation

blt commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type

How did you test this PR?

References

Uh oh!

blt commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

pr-commenter Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ✅ No significant changes detected

Uh oh!

tobz left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

blt commented Jul 6, 2026 •

edited

Loading

blt commented Jul 6, 2026 •

edited

Loading

pr-commenter Bot commented Jul 6, 2026 •

edited

Loading