CI: auto-commit chatbot product map on docs-only drift

[nrichers]

Pull Request Description

What and why?

This PR improves how the chatbot product map gets committed, removing the need for authors to manually do the commit after regeneration.

The problem

When you change documentation (for example FAQ wording or headings), CI rebuilds the site/llm/chatbot-product-map.md file that the in-app assistant uses. This file maps product screens to docs and pulls section titles from our .qmd files.

If you updated the docs but did not regenerate and commit that map, CI failed, even when the only difference was your doc edits. Contributors had to know to run make generate-chatbot-product-map and commit the result, which was easy to miss (cf. my Slack convo with Beck).

Solution

CI still regenerates the map to check it is current. On pull requests from branches in this repo, if only chatbot-product-map.md is out of date, the workflow now commits and pushes the refreshed file for you.

You still need to fix things locally when:

• The PR comes from a fork (CI cannot push to the fork branch), or
• The frontend snapshot is stale (product routes or help links changed — run make refresh-chatbot-product-map with a local frontend checkout).

README and site/llm/README.md describe when to regenerate locally vs when to let CI handle it.

Shortcut: https://app.shortcut.com/validmind/story/16405

How to test

Tested on this PR (end-to-end):

1. Changed the FAQ callout heading in site/faq/faq-workflows.qmd (specific model → specific record (model)) without updating chatbot-product-map.md, then pushed.
2. Validate docs site run 26429112104 completed successfully. The Verify chatbot product map is up to date step auto-committed site/llm/chatbot-product-map.md (chore: refresh chatbot product map, commit 3747ed2b3).
3. Reverted the FAQ heading to specific model and pushed again.
4. Validate docs site run 26429755873 completed successfully and auto-committed the map again (a2792941d). FAQ and map text matched after the run.

Reviewer smoke test:

• Change a linked .qmd heading without refreshing chatbot-product-map.md on a same-repo PR — validate should auto-commit the map and pass (may take ~20 minutes for the full job).
• Confirm snapshot drift still fails with the refresh-chatbot-product-map message (not exercised on this PR).
• Locally: python3 site/scripts/generate_chatbot_product_map.py and python3 -m unittest discover -s site/scripts -p 'test_generate_chatbot_product_map.py' -v (both pass).

What needs special review?

Dependencies, breaking changes, and deployment notes

No breaking changes. CI-only; no deployment steps beyond the existing validate workflow.

Release notes

internal — no user-facing release notes.

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[github-actions]

Lighthouse check results

✓ INFO: No site pages to audit in this PR.

Commit SHA: eaa3cb7

[github-actions]

Lighthouse check results

✓ INFO: No site pages to audit in this PR.

Commit SHA: b63dd89

[nrichers]

@validbeck hello from a ghost of the past! Do you still want this PR or should I close it? It does solve an issue that otherwise requires manual maintenance, but it's not my decision to make.

(Just doing some housecleaning on open PRs ... And I hope you are well! 👋 )

[validbeck]

@nrichers Ack, sorry, this keeps getting bumped along due to other emergencies. If it's affecting your metrics, please feel free to close and I'll just use the branch to reopen if I need to.

[validbeck]

TY for setting this up for me!! I miss ur ☕ !

[nrichers]

Thank you, @validbeck! I miss our stand-ups and chats about life and stuff. 🩷 Haven't forgotten about inviting you over! Been elbow-deep in fixing our deck when I am not working but getting there and we haven't had anyone over all summer ... Picture proof:

Are you able to resolve the merge conflicts and merge? Just tried it and I can no longer push to branches in the main repo and would need to fork and re-open another PR to resolve them. A quick Cursor chat would probably do it on your end.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ github-actions[bot]
✅ validbeck
❌ Nik Richers

---

Nik Richers seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

PR Summary

This PR introduces several improvements to the CI workflow that validates and refreshes the in-app assistant’s chatbot product map. The main changes include:

1. Updates to the GitHub Actions workflow (.github/workflows/validate-docs-site.yaml):

   • The checkout action now explicitly specifies the ref, full fetch depth, and token to ensure that the branch is checked out correctly and comprehensively.
   • A more robust mechanism has been added for verifying the state of two files (chatbot-product-map.md and chatbot-product-map-frontend-snapshot.json). Instead of merely checking for differences with a single git diff command, the script now sets flags for each file to distinctly handle them.
   • If the snapshot file has changed, the CI will now exit with an error along with a clear message instructing developers to regenerate it locally using a specified command. In cases where it is a forked PR (i.e. the repository names differ), a custom error message is provided for regenerating the map locally.
   • When only chatbot-product-map.md needs updating, the CI auto-commits and pushes the refreshed file back to the branch. This automation is clearly communicated via logs.

2. Documentation Updates:

   • The README.md and site/llm/README.md files have been updated to reflect the auto-commit behavior on same-repo pull requests and the limitations for fork PRs or any snapshot drift. The instructions now clearly differentiate between the two scenarios and explain when local regeneration is required.

Overall, the changes aim to streamline the developer workflow by automating refreshes where possible while ensuring that outdated snapshots are caught and flagged appropriately.

Test Suggestions

• Verify that the workflow correctly identifies when no changes have been made to both the chatbot product map and the snapshot file, resulting in a successful completion.
• Test that modifying only one of the files (chatbot-product-map.md or chatbot-product-map-frontend-snapshot.json) triggers the appropriate error message and exit code.
• Simulate a scenario on a fork PR to ensure that the CI flags stale documentation appropriately without attempting an auto-commit.
• Run the unit tests for the chatbot product map generator to confirm that the changes in the CI script do not interfere with existing test cases.
• Manually trigger a same-repo PR to check that the auto-commit functionality works as intended and that the commit message is correct.

[validbeck]

@nrichers Yah, just pushed a change up; making sure the validate runs fine and then I'll smush your last validContribution to us in. #物の哀れ

I hope you're not a sleepwalker, that ledge looks steep...

[github-actions]

Validate docs site

✓ INFO: A live preview of the docs site is available — Open the preview

[github-actions]

Lighthouse check results

✓ INFO: No site pages to audit in this PR.

Commit SHA: 3eb36f2