Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
7a85e20
feat(ota_demo): OTA-over-SOVD demo - RB-Theron warehouse, bad lidar u…
bburda Jul 4, 2026
d543ace
fix(ota_demo): drop the recovery dance so the stall aborts cleanly
bburda Jul 4, 2026
61da2f3
feat(ota_demo): expose the health checks as a SOVD diagnostic function
bburda Jul 4, 2026
d9e81ed
fix(ota_demo): gate run-demo.sh on real drive-readiness, self-heal th…
bburda Jul 4, 2026
aef4653
fix(ota_demo): capture the diagnostic topics in the fault MCAP + fold…
bburda Jul 4, 2026
901d7e9
fix(ota_demo): mark the hotfix update automated so the UI enables Exe…
bburda Jul 4, 2026
9ae82b1
fix(ota_demo): make the fault MCAP self-contained + persist the fast-…
bburda Jul 4, 2026
bfcf830
feat(ota_demo): consolidate the 4 health checks into one run_health_c…
bburda Jul 4, 2026
71926e6
fix(ota_demo): stop latched_relay crashing on a re-declared use_sim_time
bburda Jul 6, 2026
4d9df53
fix(ota_demo): support automated apply + capture the robot footprint …
bburda Jul 6, 2026
8117b42
fix(ota_demo): localization health check judges AMCL covariance, not …
bburda Jul 6, 2026
c5d808b
perf(ota_demo): build only the packages the demo needs, not the whole…
bburda Jul 6, 2026
d48fb87
fix(tests/ota): wait for the boot auto-apply before the process check
bburda Jul 7, 2026
fbd755b
ci(ota): dump full logs and ROS state on failure instead of a log tail
bburda Jul 7, 2026
271a421
fix(ota_demo): send the nav goal via the action, not a one-shot topic…
bburda Jul 7, 2026
621ea21
fix(ota_demo): retry the nav goal until nav2 is ready to accept it
bburda Jul 7, 2026
8959a57
fix(ota_demo): version default, pgrep doc, no-new-privileges (review)
bburda Jul 7, 2026
1001546
fix(ota_demo): harden the OTA update plugin (validate, no shell, stag…
bburda Jul 7, 2026
195d62c
ci(ota_demo): build and run the plugin gtest + packer/server pytest i…
bburda Jul 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 136 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -163,3 +163,139 @@ jobs:
if: always()
working-directory: demos/multi_ecu_aggregation
run: docker compose --profile ci down

build-and-test-ota:
needs: lint
runs-on: ubuntu-24.04
steps:
- name: Show triggering source
if: github.event_name == 'repository_dispatch'
run: |
SHA="${{ github.event.client_payload.sha }}"
RUN_URL="${{ github.event.client_payload.run_url }}"
echo "## Triggered by ros2_medkit" >> "$GITHUB_STEP_SUMMARY"
echo "- Commit: \`${SHA:-unknown}\`" >> "$GITHUB_STEP_SUMMARY"
if [ -n "$RUN_URL" ]; then
echo "- Run: [View triggering run]($RUN_URL)" >> "$GITHUB_STEP_SUMMARY"
else
echo "- Run: (URL not provided)" >> "$GITHUB_STEP_SUMMARY"
fi

- name: Checkout repository
uses: actions/checkout@v4

- name: Build and start OTA demo
working-directory: demos/ota_nav2_sensor_fix
run: docker compose up -d --build

- name: Run smoke tests
run: ./tests/smoke_test_ota.sh

- name: Diagnostics on failure
if: failure()
run: ./tests/dump_ota_diagnostics.sh ota_diagnostics.log

- name: Upload diagnostics
if: failure()
uses: actions/upload-artifact@v4
with:
name: ota-diagnostics-smoke
path: ota_diagnostics.log

- name: Teardown
if: always()
working-directory: demos/ota_nav2_sensor_fix
run: docker compose down

# Actually runs the OTA demo's own test suites (they never run in the demo
# up/smoke jobs). The C++ gtest is built and executed inside the plugin test
# stage of Dockerfile.gateway (a failing test fails the docker build); the two
# pytest suites run natively.
plugin-tests:
needs: lint
runs-on: ubuntu-24.04
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Run ota_update_plugin C++ gtest (in image build)
working-directory: demos/ota_nav2_sensor_fix
run: docker build --target ota-plugin-test -f Dockerfile.gateway .

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install Python test dependencies
run: pip install pytest fastapi 'uvicorn[standard]' httpx

- name: Run ota_update_server pytest suite
working-directory: demos/ota_nav2_sensor_fix/ota_update_server
run: pytest tests

- name: Run pack_artifact pytest suite
working-directory: demos/ota_nav2_sensor_fix/scripts
run: pytest test_pack_artifact.py

# Separate job from build-and-test-ota: this one drives the full
# latch/publish/apply/clear narrative through the operator scripts. The
# entrypoint auto-applies broken_lidar_3_0_0 at boot; send-goal.sh then
# drives the robot into the phantom sector so Nav2 cannot make progress
# and navigate_to_pose aborts, which the log + action-status bridges
# surface as ACTION_NAVIGATE_TO_POSE_ABORTED on bt-navigator (latched,
# with a freeze-frame + MCAP capture). publish-fix.sh registers the
# forward hotfix fixed_lidar_3_0_1 (not in the boot catalog), then
# apply-fix.sh swaps scan_sensor_node to it, but the fault stays
# latched (no self-heal) until clear-fault.sh explicitly clears it, and
# only then does a fresh send-goal.sh resume clean. Catches regressions
# in that loop (phantom not stalling Nav2, the bridges not promoting the
# failure, fault_manager latching/capture, the fault clearing itself on
# apply). Slower than the API-only smoke job because it has
# to wait for nav2 lifecycle to settle and for /cmd_vel to actually
# fire, so it's split out and can fail in isolation without blocking
# the quick OTA-endpoint check.
ota-demo-narrative:
needs: lint
runs-on: ubuntu-24.04
steps:
- name: Show triggering source
if: github.event_name == 'repository_dispatch'
run: |
SHA="${{ github.event.client_payload.sha }}"
RUN_URL="${{ github.event.client_payload.run_url }}"
echo "## Triggered by ros2_medkit" >> "$GITHUB_STEP_SUMMARY"
echo "- Commit: \`${SHA:-unknown}\`" >> "$GITHUB_STEP_SUMMARY"
if [ -n "$RUN_URL" ]; then
echo "- Run: [View triggering run]($RUN_URL)" >> "$GITHUB_STEP_SUMMARY"
fi

- name: Checkout repository
uses: actions/checkout@v4

- name: Build and start OTA demo
working-directory: demos/ota_nav2_sensor_fix
# docker compose up --build runs the multi-stage build for
# ota_update_server which produces the catalog + tarballs
# internally - no separate "build artifacts on host" step
# needed (and the host wouldn't have ros2_medkit_msgs anyway).
run: docker compose up -d --build

- name: Run demo narrative smoke
run: ./tests/smoke_test_demo_narrative.sh

- name: Diagnostics on failure
if: failure()
run: ./tests/dump_ota_diagnostics.sh ota_diagnostics.log

- name: Upload diagnostics
if: failure()
uses: actions/upload-artifact@v4
with:
name: ota-diagnostics-narrative
path: ota_diagnostics.log

- name: Teardown
if: always()
working-directory: demos/ota_nav2_sensor_fix
run: docker compose down
36 changes: 35 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ All demos support:
| [TurtleBot3 Integration](demos/turtlebot3_integration/) | Full ros2_medkit integration with TurtleBot3 and Nav2 | SOVD-compliant API, manifest-based discovery, fault management | ✅ Ready |
| [MoveIt Pick-and-Place](demos/moveit_pick_place/) | Panda 7-DOF arm with MoveIt 2 manipulation and ros2_medkit | Planning fault detection, controller monitoring, joint limits | ✅ Ready |
| [Multi-ECU Aggregation](demos/multi_ecu_aggregation/) | Multi-ECU peer aggregation with 3 ECUs (perception, planning, actuation), mDNS discovery, cross-ECU functions | Peer aggregation, mDNS discovery, cross-ECU functions | ✅ Ready |
| [OTA over SOVD - nav2 sensor fix](demos/ota_nav2_sensor_fix/) | Dev-grade OTA plugin showing the SOVD `/updates` lifecycle - a bad lidar update breaks Nav2, publish + apply a forward hotfix over SOVD | SOVD-spec register + update, native binary swap, fork+exec process management, Foxglove panel + curl scripts | ✅ Ready |

### Quick Start

Expand Down Expand Up @@ -150,6 +151,37 @@ cd demos/multi_ecu_aggregation
- Unified SOVD-compliant REST API spanning all ECUs
- Web UI for browsing aggregated entity hierarchy

#### OTA over SOVD Demo (Dev-grade Update / Publish-a-Hotfix)

End-to-end demo of the SOVD `/updates` resource: a regressing sensor update
(`broken_lidar_3_0_0`) is auto-applied at boot and breaks perception. Nav2
cannot make progress and `navigate_to_pose` aborts; two generic bridges surface
that failure as SOVD faults on `bt-navigator` and `controller-server` (the
sensor node never reports itself). The operator downloads the captured MCAP,
sees the phantom, publishes and applies the forward hotfix
(`fixed_lidar_3_0_1`, not a rollback), and clears the latched fault - all
without SSH, all spec-compliant.

```bash
cd demos/ota_nav2_sensor_fix
./run-demo.sh # build artifacts + bring up gateway/plugin/update server
./check-demo.sh # show registered updates + per-id status + live process state
./send-goal.sh # drive into the phantom sector; Nav2 stalls, navigate_to_pose aborts
./publish-fix.sh # register fixed_lidar_3_0_1 (SOVD POST /updates) - not in the boot catalog
./apply-fix.sh # broken_lidar_3_0_0 -> fixed_lidar_3_0_1 (the headline: apply the published hotfix)
./clear-fault.sh # operator clear of the latched bt-navigator/controller-server faults
./stop-demo.sh
```

**Features:**

- Dev-grade `ota_update_plugin` C++ gateway plugin (UpdateProvider + GatewayPlugin)
- SOVD ISO 17978-3 compliant `/updates` resource: kind derived from
`updated_components` / `added_components` / `removed_components` metadata
- Native binary swap + `fork+exec` process management (no containers, no signing)
- Foxglove Studio panel mirrors the same SOVD client patterns as the web UI
- Pairs with the [`ros2_medkit_foxglove_extension`](https://github.com/selfpatch/ros2_medkit_foxglove_extension) Updates panel

## Getting Started

### Prerequisites
Expand Down Expand Up @@ -209,9 +241,11 @@ Each demo has automated smoke tests that verify the gateway starts and the REST
./tests/smoke_test.sh # Sensor diagnostics (full API coverage + fault injection + beacons)
./tests/smoke_test_turtlebot3.sh # TurtleBot3 (discovery, data, operations, scripts, triggers, logs)
./tests/smoke_test_moveit.sh # MoveIt pick-and-place (discovery, data, operations, scripts, triggers, logs)
./tests/smoke_test_multi_ecu.sh # Multi-ECU aggregation (per-ECU discovery + aggregated view)
./tests/smoke_test_ota.sh # OTA over SOVD (catalog, /updates spec shape, prepare/execute, process swap)
```

CI runs all 4 demos in parallel - each job builds the Docker image, starts the container, and runs the smoke tests against it. See [CI workflow](.github/workflows/ci.yml).
CI runs all demos in parallel - each job builds the Docker image, starts the container, and runs the smoke tests against it. See [CI workflow](.github/workflows/ci.yml).

## Related Projects

Expand Down
Loading
Loading