diff --git a/README.md b/README.md index 3418d67..1cd239b 100644 --- a/README.md +++ b/README.md @@ -150,6 +150,8 @@ playwright-cli snapshot # capture page snapshot to obtain elemen playwright-cli snapshot --filename=f # save snapshot to specific file playwright-cli snapshot # snapshot a specific element playwright-cli snapshot --depth=N # limit snapshot depth for efficiency +playwright-cli find # search the snapshot for text, returns matching nodes +playwright-cli find --regex # search the snapshot with a regexp playwright-cli eval [ref] # evaluate javascript expression on page or element playwright-cli dialog-accept [prompt] # accept a dialog playwright-cli dialog-dismiss # dismiss a dialog @@ -264,6 +266,8 @@ playwright-cli highlight --hide # hide all page highlights ```bash playwright-cli open --browser=chrome # use specific browser +playwright-cli open --mobile # emulate a generic mobile device +playwright-cli open --device="iPhone 15" # emulate a specific device playwright-cli attach --extension=chrome # connect via Playwright Extension playwright-cli attach --cdp=chrome # attach to running Chrome/Edge by channel playwright-cli attach --cdp= # attach via CDP endpoint @@ -306,6 +310,13 @@ playwright-cli snapshot e34 # include each element's bounding box as [box=x,y,width,height] playwright-cli snapshot --boxes + +# search a large snapshot instead of capturing it all — returns matching nodes +# with 3 lines of context around each match (like grep -C) +playwright-cli find "Add to cart" +playwright-cli find --regex "\\$[0-9]+\\.[0-9]{2}" +# wrap the regexp in slashes to add flags, e.g. /i for case-insensitive +playwright-cli find --regex "/sign (in|up)/i" ``` ### Targeting elements @@ -346,13 +357,13 @@ playwright-cli kill-all # forcefully kill all browser processes ### Local installation -If global `playwright-cli` command is not available, try a local version via `npx playwright-cli`: +If global `playwright-cli` command is not available, try a local version via `npx playwright cli`: ```bash -npx --no-install playwright-cli --version +npx --no-install playwright --version ``` -When local version is available, use `npx playwright-cli` in all commands. Otherwise, install `playwright-cli` as a global command: +When local version is available, use `npx playwright cli` in all commands. Otherwise, install `playwright-cli` as a global command: ```bash npm install -g @playwright/cli@latest @@ -559,9 +570,8 @@ The installed skill includes detailed reference guides for common tasks: * **Request mocking** — intercept and mock network requests * **Running Playwright code** — execute arbitrary Playwright scripts * **Browser session management** — manage multiple browser sessions -* **Spec-driven testing (plan / generate / heal)** — drive tests from a written spec * **Storage state (cookies, localStorage)** — persist and restore browser state -* **Test generation** — generate Playwright tests from interactions +* **Test generation (plan / generate / heal)** — generate Playwright tests from a spec or interactions * **Tracing** — record and inspect execution traces * **Video recording** — capture browser session videos * **Inspecting element attributes** — get element id, class, or any attribute not visible in the snapshot diff --git a/package-lock.json b/package-lock.json index 12a5955..9413daf 100644 --- a/package-lock.json +++ b/package-lock.json @@ -9,14 +9,14 @@ "version": "0.1.15", "license": "Apache-2.0", "dependencies": { - "playwright": "1.62.0-alpha-2026-06-29", - "playwright-core": "1.62.0-alpha-2026-06-29" + "playwright": "1.62.0-alpha-2026-07-08", + "playwright-core": "1.62.0-alpha-2026-07-08" }, "bin": { "playwright-cli": "playwright-cli.js" }, "devDependencies": { - "@playwright/test": "1.62.0-alpha-2026-06-29", + "@playwright/test": "1.62.0-alpha-2026-07-08", "@types/node": "^25.2.1" }, "engines": { @@ -24,19 +24,19 @@ } }, "node_modules/@playwright/test": { - "version": "1.62.0-alpha-2026-06-29", - "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.62.0-alpha-2026-06-29.tgz", - "integrity": "sha512-g9foBBuWf7IoALxyck3GJjjl/dsUewmxXz8a3JNgKgqgHFOZou9DZA7isi0hLU9Lz+75LeBSxN92E+aI7KAbUQ==", + "version": "1.62.0-alpha-2026-07-08", + "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.62.0-alpha-2026-07-08.tgz", + "integrity": "sha512-D0qozGfVOuI8x9fD70vK0H5ntL+rNz5oOHRhT+SxMJIKP0OyjA6P95GfLFUeliaCxR1nR3jc4KRZcI2UhRYyXw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "playwright": "1.62.0-alpha-2026-06-29" + "playwright": "1.62.0-alpha-2026-07-08" }, "bin": { "playwright": "cli.js" }, "engines": { - "node": ">=18" + "node": ">=20" } }, "node_modules/@types/node": { @@ -64,33 +64,33 @@ } }, "node_modules/playwright": { - "version": "1.62.0-alpha-2026-06-29", - "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.62.0-alpha-2026-06-29.tgz", - "integrity": "sha512-ugaMuh6rAOqmbaxrM7+XQSs7M6yPXykq5gVugJPClHcJ4Cqxmky5rDHhrCt4wmaILUeMdI7kPBCx3zMx2Qu+bg==", + "version": "1.62.0-alpha-2026-07-08", + "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.62.0-alpha-2026-07-08.tgz", + "integrity": "sha512-5vivHdZSnbUW3iutwp2kbr9wjsdFR7OIEUcMvsR5g8Iy/PeYm0uZRwKwTsnZMPr43uFCdW4RQIj0ioszIL31MQ==", "license": "Apache-2.0", "dependencies": { - "playwright-core": "1.62.0-alpha-2026-06-29" + "playwright-core": "1.62.0-alpha-2026-07-08" }, "bin": { "playwright": "cli.js" }, "engines": { - "node": ">=18" + "node": ">=20" }, "optionalDependencies": { "fsevents": "2.3.2" } }, "node_modules/playwright-core": { - "version": "1.62.0-alpha-2026-06-29", - "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.62.0-alpha-2026-06-29.tgz", - "integrity": "sha512-/13EvW8l9Bmvt9nKHey+OcvZ8nob2FM8BCKBsUNwbXgmT4Hc0XX2b6mOErU0RBYNedOCa9q15USI7SY9K17v4w==", + "version": "1.62.0-alpha-2026-07-08", + "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.62.0-alpha-2026-07-08.tgz", + "integrity": "sha512-1GEx9eELpB6cdtb1MTSeh7K4LP5k9vuaT4HlIsldD/8nvVhf1k2DqBQZHHyAs3oSTfz6MhqVaMEM4VkyjoPsLw==", "license": "Apache-2.0", "bin": { "playwright-core": "cli.js" }, "engines": { - "node": ">=18" + "node": ">=20" } }, "node_modules/undici-types": { diff --git a/package.json b/package.json index 84e1f84..f87a6d2 100644 --- a/package.json +++ b/package.json @@ -18,12 +18,12 @@ "test": "playwright test" }, "devDependencies": { - "@playwright/test": "1.62.0-alpha-2026-06-29", + "@playwright/test": "1.62.0-alpha-2026-07-08", "@types/node": "^25.2.1" }, "dependencies": { - "playwright": "1.62.0-alpha-2026-06-29", - "playwright-core": "1.62.0-alpha-2026-06-29" + "playwright": "1.62.0-alpha-2026-07-08", + "playwright-core": "1.62.0-alpha-2026-07-08" }, "bin": { "playwright-cli": "playwright-cli.js" diff --git a/skills/playwright-cli/SKILL.md b/skills/playwright-cli/SKILL.md index 2fa5d0e..6f34a18 100644 --- a/skills/playwright-cli/SKILL.md +++ b/skills/playwright-cli/SKILL.md @@ -47,6 +47,11 @@ playwright-cli upload ./document.pdf playwright-cli check e12 playwright-cli uncheck e12 playwright-cli snapshot +# search the snapshot for text or a regexp, returns matching nodes with surrounding context +playwright-cli find "Sign in" +playwright-cli find --regex "Sign (in|up)" +# wrap the regexp in slashes to add flags, e.g. /i for case-insensitive +playwright-cli find --regex "/sign (in|up)/i" playwright-cli eval "document.title" playwright-cli eval "el => el.textContent" e5 # get element id, class, or any attribute not visible in the snapshot @@ -210,6 +215,12 @@ playwright-cli open --browser=firefox playwright-cli open --browser=webkit playwright-cli open --browser=msedge +# Emulate a generic mobile device (Pixel 10 for Chromium, iPhone 17 for WebKit). +# Prefer this when a mobile layout is acceptable: mobile pages are usually +# lighter, so snapshots are smaller and cheaper. +playwright-cli open --mobile +playwright-cli open --device="iPhone 15" + # Use persistent profile (by default profile is in-memory) playwright-cli open --persistent # Use persistent profile with custom directory @@ -279,6 +290,11 @@ playwright-cli snapshot e34 # include each element's bounding box as [box=x,y,width,height] playwright-cli snapshot --boxes + +# search a large snapshot instead of capturing it all — returns matching nodes +# with 3 lines of context around each match (like grep -C) +playwright-cli find "Add to cart" +playwright-cli find --regex "\\$[0-9]+\\.[0-9]{2}" ``` ## Targeting elements @@ -326,13 +342,13 @@ playwright-cli kill-all ## Installation -If global `playwright-cli` command is not available, try a local version via `npx playwright-cli`: +If global `playwright-cli` command is not available, try a local version via `npx playwright cli`: ```bash -npx --no-install playwright-cli --version +npx --no-install playwright --version ``` -When local version is available, use `npx playwright-cli` in all commands. Otherwise, install `playwright-cli` as a global command: +When local version is available, use `npx playwright cli` in all commands. Otherwise, install `playwright-cli` as a global command: ```bash npm install -g @playwright/cli@latest @@ -397,9 +413,8 @@ playwright-cli show --annotate * **Request mocking** [references/request-mocking.md](references/request-mocking.md) * **Running Playwright code** [references/running-code.md](references/running-code.md) * **Browser session management** [references/session-management.md](references/session-management.md) -* **Spec-driven testing (plan / generate / heal)** [references/spec-driven-testing.md](references/spec-driven-testing.md) * **Storage state (cookies, localStorage)** [references/storage-state.md](references/storage-state.md) -* **Test generation** [references/test-generation.md](references/test-generation.md) +* **Test generation (plan / generate / heal)** [references/test-generation.md](references/test-generation.md) * **Tracing** [references/tracing.md](references/tracing.md) * **Video recording** [references/video-recording.md](references/video-recording.md) * **Inspecting element attributes** [references/element-attributes.md](references/element-attributes.md) diff --git a/skills/playwright-cli/references/spec-driven-testing.md b/skills/playwright-cli/references/spec-driven-testing.md deleted file mode 100644 index 336dbfc..0000000 --- a/skills/playwright-cli/references/spec-driven-testing.md +++ /dev/null @@ -1,305 +0,0 @@ -# Spec-driven testing (plan → generate → heal) - -End-to-end workflow for authoring and maintaining Playwright tests using `playwright-cli`. The three sections below can be used independently: - -- **Planning** — explore the app, produce a spec file describing what to test. -- **Generate** — turn a spec into Playwright test files. Update the spec if it's vague or stale. -- **Heal** — diagnose failing tests, fix the code, reconcile the spec with reality. - -All three lean on the same mechanic: run `npx playwright test --debug=cli` in the background, then `playwright-cli attach tw-XXXX` to drive the paused page interactively. See [playwright-tests.md](playwright-tests.md) for the debug/attach mechanics and [test-generation.md](test-generation.md) for how every `playwright-cli` action emits Playwright TypeScript. - ---- - -## 1. Planning - -Goal: produce a spec file (e.g. `specs/.plan.md`) that enumerates the scenarios to test. **Always** write the spec to a file. - -### 1.1 Prerequisite: workspace - -Check the workspace has Playwright installed before anything else: - -```bash -# Either of these confirms a workspace: -test -f playwright.config.ts || test -f playwright.config.js -npx --no-install playwright --version -``` - -If there is no Playwright install, bootstrap one and let the user pick the defaults: - -```bash -npm init playwright@latest -``` - -### 1.2 Prerequisite: seed test - -A **seed test** is a minimal test that lands the page in the state every scenario starts from: navigation to the app, any required login, feature flags, etc. Scenarios assume a fresh start *after* the seed. `--debug=cli` pauses *inside* this test, so the seed is where every planning and generation session begins. - -Minimum viable seed: - -```ts -// tests/seed.spec.ts -import { test } from '@playwright/test'; - -test('seed', async ({ page }) => { - await page.goto('https://example.com/'); -}); -``` - -Preferred — push navigation into a fixture so scenario tests reuse it: - -```ts -// tests/fixtures.ts -import { test as baseTest } from '@playwright/test'; -export { expect } from '@playwright/test'; - -export const test = baseTest.extend({ - page: async ({ page }, use) => { - await page.goto('https://example.com/'); - await use(page); - }, -}); -``` - -```ts -// tests/seed.spec.ts -import { test } from './fixtures'; - -test('seed', async ({ page }) => { - // Fixture already navigates. This empty body tells agents where to start. -}); -``` - -If no seed exists, create one that at least navigates to the app. - -### 1.3 Explore the app - -Launch the app via the seed in the background and attach: - -```bash -PLAYWRIGHT_HTML_OPEN=never npx playwright test tests/seed.spec.ts --debug=cli -# wait for "Debugging Instructions" and the session name tw-XXXX -playwright-cli attach tw-XXXX -``` - -Resume so the seed runs, then probe the app: - -```bash -playwright-cli resume # resume so that seed test runs fully -playwright-cli snapshot # inventory of interactive elements -playwright-cli click e5 # follow a flow -playwright-cli eval "location.href" # read URL / state -playwright-cli show --annotate # ask the user to point at something -``` - -Map out: - -- Interactive surfaces (forms, buttons, lists, filters, modals). -- Primary user journeys end-to-end. -- Edge cases: empty states, validation errors, very long input, boundary values. -- Persistence: reload, local/session storage, URL fragments. -- Navigation: which controls change the URL, back/forward behaviour. - -**Important**: Do not just open the app url with playwright-cli, always go through the test to capture any custom setup done there. -**Important**: Stop the background test when done exploring. - -### 1.4 Write the spec file - -Save under `specs/.plan.md`. Use this structure: - -```markdown -# Test Plan - -## Application Overview - - - -## Test Scenarios - -### 1. - -**Seed:** `tests/seed.spec.ts` - -#### 1.1. - -**File:** `tests//.spec.ts` - -**Steps:** - 1. - - expect: - - expect: - 2. - - expect: - -#### 1.2. -... - -### 2. - -**Seed:** `tests/seed.spec.ts` -... -``` - -Guidelines: - -- Each scenario is independent and starts from the seed's fresh state — never chain scenarios. -- Scenario names are kebab-case and match the test file name (`should-add-single-todo` → `should-add-single-todo.spec.ts`). -- Cover happy path, edge cases, validation, negative flows, persistence. -- Write steps at the user level ("Type 'Buy milk' into the input"), not the API level ("call `fill`"). -- Put observable outcomes in `- expect:` bullets; each becomes an assertion during generation. - ---- - -## 2. Generate - -Goal: take a spec file and produce Playwright test files. Optionally update the spec if it has drifted. - -### 2.1 Inputs - -- **Spec file**, e.g. `specs/basic-operations.plan.md`. -- **Target**: either a single scenario (e.g. `1.2`), a whole group (`1`), or all. -- **Seed file**, read from the `**Seed:**` line of the scenario's group. - -### 2.2 Generate one scenario - -For each target scenario, in sequence (never in parallel — scenarios share the seed session): - -```bash -PLAYWRIGHT_HTML_OPEN=never npx playwright test --debug=cli # background -playwright-cli attach tw-XXXX -# resume -``` - -**Do not** just open the app url with playwright-cli, always go through the test to capture any custom setup done there. - -Walk the scenario's `Steps:` one by one with `playwright-cli`, treating the spec as the plan and the live app as the source of truth. If a step is vague ("click the button" — which button?), references an element that no longer exists, or contradicts the app's actual behaviour, use your judgement: update the spec to match what the app really does, then keep going. Editing the spec mid-generation is expected. - -Every action prints the equivalent Playwright TypeScript (see [test-generation.md](test-generation.md)): - -```bash -playwright-cli snapshot # find refs -playwright-cli fill e3 "John Doe" # -> page.getByRole('textbox', {...}).fill(...) -playwright-cli press Enter -playwright-cli click e7 -``` - -For each `- expect:` bullet, add an explicit assertion. See [test-generation.md](test-generation.md) for details. - -Collect the generated code and write the test file at the path given in the spec: - -```ts -// spec: specs/basic-operations.plan.md -// seed: tests/seed.spec.ts -import { test, expect } from './fixtures'; // or '@playwright/test' if no fixtures file - -test.describe('Signing in and out', () => { - test('should sign in', async ({ page }) => { - // 1. Navigate to the application - // (handled by the seed fixture) - - // 2. Type 'John Doe' into the username field - await page.getByRole('textbox', { name: 'username' }).fill('John Doe'); - - // 3. Type password - await page.getByRole('textbox', { name: 'password' }).fill('TestPassword'); - - // 4. Press Enter to submit - await page.getByRole('textbox', { name: 'password' }).press('Enter'); - - await expect(page.getByRole('heading')).toContainText('Welcome, John Doe!'); - }); -}); -``` - -Rules: - -- **One test per file.** File path, describe name, and test name come verbatim from the spec (minus the ordinal). -- Prefix each numbered step with a `// N. ` comment before its actions. -- Use the describe group name verbatim from the spec (no `1.` ordinal). -- Import from `./fixtures` if the project has one; otherwise `@playwright/test`. -- **Important**: close the CLI session and stop the background test before moving to the next scenario. - -### 2.3 Generate multiple scenarios - -Loop 2.2 over the targeted scenarios one at a time, restarting the seed between each so every test starts from a clean page. This is safe to parallelise due to unique generated session names - just make sure each test run is stopped. - -### 2.4 Run generated tests - -After generation, run the new tests once: - -```bash -PLAYWRIGHT_HTML_OPEN=never npx playwright test tests//.spec.ts -``` - -Any failure goes to Section 3. - ---- - -## 3. Heal - -Goal: fix failing tests, and update the spec if the app's intended behaviour changed. - -### 3.1 Find failing tests - -```bash -PLAYWRIGHT_HTML_OPEN=never npx playwright test -``` - -Record the list of failing `:` entries and process them one at a time. Do not attempt parallel fixes — shared state and the single CLI session make that fragile. - -### 3.2 Debug one failure - -Run the single failing test in debug mode in the background, then attach: - -```bash -PLAYWRIGHT_HTML_OPEN=never npx playwright test tests//.spec.ts: --debug=cli -# wait for "Debugging Instructions" and the tw-XXXX session name -playwright-cli attach tw-XXXX -``` - -The test is paused at the start. Step forward or run to until just before the failing action or assertion, then diagnose: - -```bash -playwright-cli snapshot # did the element change / move / rename? -playwright-cli console # app-side errors? -playwright-cli requests # failed request? wrong payload? -playwright-cli show --annotate # ask the user to point somewhere -``` - -Common causes: selector drift, new wrapper element, label/ARIA rename, timing (transition, async load), assertion text updated in the app, test data leaking between runs. - -Rehearse the corrected interaction with `playwright-cli` — the generated code in the output is what you paste back into the test. - -### 3.3 Apply the fix - -Edit the test file: update the locator, assertion, step order, or inputs to match the corrected behaviour. Stop the background debug run. Rerun the single test to confirm green. - -Never skip hooks or add sleeps as a fix. Never use `networkidle`. - -### 3.4 Reconcile with the spec - -Open the spec referenced by the `// spec:` header in the test file and locate the scenario that matches the test. - -- **Fix was purely technical** (locator drift, better assertion shape) and the spec's user-level behaviour still matches the app → leave the spec alone. -- **Fix changed user-visible steps, inputs, order, or expected outcomes** that the spec describes → update the spec to match reality. Keep the scenario id and file path stable; only the step / expect lines change. -- **Unclear whether the app change is intentional** (spec is stale) **or a regression** (test was right, app is wrong) → **stop and ask the user**. Provide: - - the scenario id (e.g. `2.3`), - - the spec lines that no longer match, - - the observed app behaviour (quote a snapshot excerpt or a concrete outcome). - -Only after the user answers, either update the spec (intentional change) or file/flag the test as covering a bug (regression). - -### 3.5 Iteration and giving up - -- Fix failures one at a time; rerun after each. -- If after thorough investigation you are confident the test is correct but the app is wrong *and* the user has confirmed it's a bug: mark the test `test.fixme(...)` with a comment pointing at the user's decision or issue link. Never silently skip. - ---- - -## Cross-references - -| For... | See | -|---|---| -| `--debug=cli` / attach mechanics | [playwright-tests.md](playwright-tests.md) | -| How `playwright-cli` actions become TS | [test-generation.md](test-generation.md) | -| Mocking requests during exploration/generation | [request-mocking.md](request-mocking.md) | -| Managing the CLI browser session | [session-management.md](session-management.md) | diff --git a/skills/playwright-cli/references/test-generation.md b/skills/playwright-cli/references/test-generation.md index a045c55..35a8d57 100644 --- a/skills/playwright-cli/references/test-generation.md +++ b/skills/playwright-cli/references/test-generation.md @@ -1,13 +1,19 @@ -# Test Generation +# Test generation (plan → generate → heal) -Generate Playwright test code automatically as you interact with the browser. +End-to-end workflow for authoring and maintaining Playwright tests with `playwright-cli`. Every `playwright-cli` action emits the equivalent Playwright TypeScript, and that generated code is the raw material for every test. The sections below can be used independently: -## How It Works +- **How generation works** — the core mechanic everything else relies on: actions become TypeScript, plus how to add assertions. +- **Plan** — explore the app, produce a spec file describing what to test. +- **Generate** — turn a spec into Playwright test files. Update the spec if it's vague or stale. +- **Heal** — diagnose failing tests, fix the code, reconcile the spec with reality. -Every action you perform with `playwright-cli` generates corresponding Playwright TypeScript code. -This code appears in the output and can be copied directly into your test files. +Plan / generate / heal lean on the same mechanic: run `npx playwright test --debug=cli` in the background, then `playwright-cli attach tw-XXXX` to drive the paused page interactively. See [playwright-tests.md](playwright-tests.md) for the debug/attach mechanics. -## Example Workflow +--- + +## 0. How generation works + +Every action you perform with `playwright-cli` generates corresponding Playwright TypeScript code. This code appears in the output and can be copied directly into your test files. ```bash # Start a session @@ -31,7 +37,7 @@ playwright-cli click e3 # await page.getByRole('button', { name: 'Sign In' }).click(); ``` -## Building a Test File +### Building a test file Collect the generated code into a Playwright test: @@ -50,9 +56,7 @@ test('login flow', async ({ page }) => { }); ``` -## Best Practices - -### 1. Use Semantic Locators +### Use semantic locators The generated code uses role-based locators when possible, which are more resilient: @@ -64,7 +68,7 @@ await page.getByRole('button', { name: 'Submit' }).click(); await page.locator('#submit-btn').click(); ``` -### 2. Explore Before Recording +### Explore before recording Take snapshots to understand the page structure before recording actions: @@ -75,7 +79,7 @@ playwright-cli snapshot playwright-cli click e5 ``` -### 3. Add Assertions Manually +### Add assertions manually Generated code captures actions but not assertions. Add expectations in your test using one of the recommended matchers: @@ -132,3 +136,298 @@ await expect(page.getByRole('navigation')).toMatchAriaSnapshot(` - link "Profile" `); ``` + +--- + +## 1. Planning + +Goal: produce a spec file (e.g. `specs/.plan.md`) that enumerates the scenarios to test. **Always** write the spec to a file. + +### 1.1 Prerequisite: workspace + +Check the workspace has Playwright installed before anything else: + +```bash +# Either of these confirms a workspace: +test -f playwright.config.ts || test -f playwright.config.js +npx --no-install playwright --version +``` + +If there is no Playwright install, bootstrap one and let the user pick the defaults: + +```bash +npm init playwright@latest +``` + +### 1.2 Prerequisite: seed test + +A **seed test** is a minimal test that lands the page in the state every scenario starts from: navigation to the app, any required login, feature flags, etc. Scenarios assume a fresh start *after* the seed. `--debug=cli` pauses *inside* this test, so the seed is where every planning and generation session begins. + +Minimum viable seed: + +```ts +// tests/seed.spec.ts +import { test } from '@playwright/test'; + +test('seed', async ({ page }) => { + await page.goto('https://example.com/'); +}); +``` + +Preferred — push navigation into a fixture so scenario tests reuse it: + +```ts +// tests/fixtures.ts +import { test as baseTest } from '@playwright/test'; +export { expect } from '@playwright/test'; + +export const test = baseTest.extend({ + page: async ({ page }, use) => { + await page.goto('https://example.com/'); + await use(page); + }, +}); +``` + +```ts +// tests/seed.spec.ts +import { test } from './fixtures'; + +test('seed', async ({ page }) => { + // Fixture already navigates. This empty body tells agents where to start. +}); +``` + +If no seed exists, create one that at least navigates to the app. + +### 1.3 Explore the app + +Launch the app via the seed in the background and attach: + +```bash +PLAYWRIGHT_HTML_OPEN=never npx playwright test tests/seed.spec.ts --debug=cli +# wait for "Debugging Instructions" and the session name tw-XXXX +playwright-cli attach tw-XXXX +``` + +Resume so the seed runs, then probe the app: + +```bash +playwright-cli resume # resume so that seed test runs fully +playwright-cli snapshot # inventory of interactive elements +playwright-cli click e5 # follow a flow +playwright-cli eval "location.href" # read URL / state +playwright-cli show --annotate # ask the user to point at something +``` + +Map out: + +- Interactive surfaces (forms, buttons, lists, filters, modals). +- Primary user journeys end-to-end. +- Edge cases: empty states, validation errors, very long input, boundary values. +- Persistence: reload, local/session storage, URL fragments. +- Navigation: which controls change the URL, back/forward behaviour. + +**Important**: Do not just open the app url with playwright-cli, always go through the test to capture any custom setup done there. +**Important**: Stop the background test when done exploring. + +### 1.4 Write the spec file + +Save under `specs/.plan.md`. Use this structure: + +```markdown +# Test Plan + +## Application Overview + + + +## Test Scenarios + +### 1. + +**Seed:** `tests/seed.spec.ts` + +#### 1.1. + +**File:** `tests//.spec.ts` + +**Steps:** + 1. + - expect: + - expect: + 2. + - expect: + +#### 1.2. +... + +### 2. + +**Seed:** `tests/seed.spec.ts` +... +``` + +Guidelines: + +- Each scenario is independent and starts from the seed's fresh state — never chain scenarios. +- Scenario names are kebab-case and match the test file name (`should-add-single-todo` → `should-add-single-todo.spec.ts`). +- Cover happy path, edge cases, validation, negative flows, persistence. +- Write steps at the user level ("Type 'Buy milk' into the input"), not the API level ("call `fill`"). +- Put observable outcomes in `- expect:` bullets; each becomes an assertion during generation. + +--- + +## 2. Generate + +Goal: take a spec file and produce Playwright test files. Optionally update the spec if it has drifted. + +### 2.1 Inputs + +- **Spec file**, e.g. `specs/basic-operations.plan.md`. +- **Target**: either a single scenario (e.g. `1.2`), a whole group (`1`), or all. +- **Seed file**, read from the `**Seed:**` line of the scenario's group. + +### 2.2 Generate one scenario + +For each target scenario, in sequence (never in parallel — scenarios share the seed session): + +```bash +PLAYWRIGHT_HTML_OPEN=never npx playwright test --debug=cli # background +playwright-cli attach tw-XXXX +# resume +``` + +**Do not** just open the app url with playwright-cli, always go through the test to capture any custom setup done there. + +Walk the scenario's `Steps:` one by one with `playwright-cli`, treating the spec as the plan and the live app as the source of truth. If a step is vague ("click the button" — which button?), references an element that no longer exists, or contradicts the app's actual behaviour, use your judgement: update the spec to match what the app really does, then keep going. Editing the spec mid-generation is expected. + +Every action prints the equivalent Playwright TypeScript (see [How generation works](#0-how-generation-works)): + +```bash +playwright-cli snapshot # find refs +playwright-cli fill e3 "John Doe" # -> page.getByRole('textbox', {...}).fill(...) +playwright-cli press Enter +playwright-cli click e7 +``` + +For each `- expect:` bullet, add an explicit assertion. See [How generation works](#0-how-generation-works) for details. + +Collect the generated code and write the test file at the path given in the spec: + +```ts +// spec: specs/basic-operations.plan.md +// seed: tests/seed.spec.ts +import { test, expect } from './fixtures'; // or '@playwright/test' if no fixtures file + +test.describe('Signing in and out', () => { + test('should sign in', async ({ page }) => { + // 1. Navigate to the application + // (handled by the seed fixture) + + // 2. Type 'John Doe' into the username field + await page.getByRole('textbox', { name: 'username' }).fill('John Doe'); + + // 3. Type password + await page.getByRole('textbox', { name: 'password' }).fill('TestPassword'); + + // 4. Press Enter to submit + await page.getByRole('textbox', { name: 'password' }).press('Enter'); + + await expect(page.getByRole('heading')).toContainText('Welcome, John Doe!'); + }); +}); +``` + +Rules: + +- **One test per file.** File path, describe name, and test name come verbatim from the spec (minus the ordinal). +- Prefix each numbered step with a `// N. ` comment before its actions. +- Use the describe group name verbatim from the spec (no `1.` ordinal). +- Import from `./fixtures` if the project has one; otherwise `@playwright/test`. +- **Important**: close the CLI session and stop the background test before moving to the next scenario. + +### 2.3 Generate multiple scenarios + +Loop 2.2 over the targeted scenarios one at a time, restarting the seed between each so every test starts from a clean page. This is safe to parallelise due to unique generated session names - just make sure each test run is stopped. + +### 2.4 Run generated tests + +After generation, run the new tests once: + +```bash +PLAYWRIGHT_HTML_OPEN=never npx playwright test tests//.spec.ts +``` + +Any failure goes to Section 3. + +--- + +## 3. Heal + +Goal: fix failing tests, and update the spec if the app's intended behaviour changed. + +### 3.1 Find failing tests + +```bash +PLAYWRIGHT_HTML_OPEN=never npx playwright test +``` + +Record the list of failing `:` entries and process them one at a time. Do not attempt parallel fixes — shared state and the single CLI session make that fragile. + +### 3.2 Debug one failure + +Run the single failing test in debug mode in the background, then attach: + +```bash +PLAYWRIGHT_HTML_OPEN=never npx playwright test tests//.spec.ts: --debug=cli +# wait for "Debugging Instructions" and the tw-XXXX session name +playwright-cli attach tw-XXXX +``` + +The test is paused at the start. Step forward or run to until just before the failing action or assertion, then diagnose: + +```bash +playwright-cli snapshot # did the element change / move / rename? +playwright-cli console # app-side errors? +playwright-cli requests # failed request? wrong payload? +playwright-cli show --annotate # ask the user to point somewhere +``` + +Common causes: selector drift, new wrapper element, label/ARIA rename, timing (transition, async load), assertion text updated in the app, test data leaking between runs. + +Rehearse the corrected interaction with `playwright-cli` — the generated code in the output is what you paste back into the test. + +### 3.3 Apply the fix + +Edit the test file: update the locator, assertion, step order, or inputs to match the corrected behaviour. Stop the background debug run. Rerun the single test to confirm green. + +Never skip hooks or add sleeps as a fix. Never use `networkidle`. + +### 3.4 Reconcile with the spec + +Open the spec referenced by the `// spec:` header in the test file and locate the scenario that matches the test. + +- **Fix was purely technical** (locator drift, better assertion shape) and the spec's user-level behaviour still matches the app → leave the spec alone. +- **Fix changed user-visible steps, inputs, order, or expected outcomes** that the spec describes → update the spec to match reality. Keep the scenario id and file path stable; only the step / expect lines change. +- **Unclear whether the app change is intentional** (spec is stale) **or a regression** (test was right, app is wrong) → **stop and ask the user**. Provide: + - the scenario id (e.g. `2.3`), + - the spec lines that no longer match, + - the observed app behaviour (quote a snapshot excerpt or a concrete outcome). + +Only after the user answers, either update the spec (intentional change) or file/flag the test as covering a bug (regression). + +### 3.5 Iteration and giving up + +- Fix failures one at a time; rerun after each. +- If after thorough investigation you are confident the test is correct but the app is wrong *and* the user has confirmed it's a bug: mark the test `test.fixme(...)` with a comment pointing at the user's decision or issue link. Never silently skip. + +--- + +## Cross-references + +| For... | See | +|---|---| +| `--debug=cli` / attach mechanics | [playwright-tests.md](playwright-tests.md) | +| Mocking requests during exploration/generation | [request-mocking.md](request-mocking.md) | +| Managing the CLI browser session | [session-management.md](session-management.md) |