Skip to content

FEAT: Garak sysprompt extraction scenario#2142

Open
rlundeen2 wants to merge 1 commit into
microsoft:mainfrom
rlundeen2:rlundeen2-garak-sysprompt-scenario
Open

FEAT: Garak sysprompt extraction scenario#2142
rlundeen2 wants to merge 1 commit into
microsoft:mainfrom
rlundeen2:rlundeen2-garak-sysprompt-scenario

Conversation

@rlundeen2

Copy link
Copy Markdown
Contributor

Ports garak's sysprompt_extraction.SystemPromptExtraction probe as a PyRIT scenario, operationally similar to the garak probe but built with scenario constructs.

What it does

  • Installs a real system prompt (sampled from the garak_drh_system_prompts / garak_tm_system_prompts datasets) on the target, then sends an extraction request asking the model to reveal it.
  • Extraction templates live in a new local garak_system_prompt_extraction dataset; each template carries a per-seed technique tag mapping to one of the 9 garak categories, which become SystemPromptExtractionStrategy members.
  • Each selected category becomes one AtomicAttack whose seed groups are (system prompt x template) combinations. Across all selected categories the total combinations are randomly sampled down to prompt_cap (garak's soft_probe_prompt_cap) so a default run stays bounded.
  • Responses are scored deterministically by SystemPromptExtractionScorer (character n-gram containment overlap, reusing ApproximateTextMatching), wrapped by FloatScaleThresholdScorer.

Notes

  • Follows the web_injection scenario pattern for dataset loading/resolution.
  • Docs (doc/scanner/garak.py / .ipynb) updated with a minimal executable run.
  • Added derczynski2024garak to the bibliography.
  • Tests: 23 passing; ruff + ty clean.

Ports garak's sysprompt_extraction.SystemPromptExtraction probe as a PyRIT
scenario. Installs a real system prompt on the target, then sends extraction
templates (9 garak categories) and scores responses with a deterministic
n-gram containment scorer.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant