[BREAKING] FEAT: Standardize Jailbreak scenario defaults

[varunj-msft]

Description

Part of the Standardizing Scenarios effort. Redesigns the Jailbreak scenario around three orthogonal selection axes, matching the model Rich outlined and the pattern the rapid-response scenario already uses:

• Objectives — the harmful behaviors to test, via --dataset-names. The default moves off the small airt_harms sample to HarmBench (max_dataset_size=4 by default).
• Techniques — how each jailbroken objective is delivered. The default is to "just send" it (a scenario-local prompt_sending technique); the shared core techniques (role_play, many_shot, tap, crescendo, ...) are selectable via --strategies, plus single_turn / multi_turn / all aggregates. This replaces the old hand-written JailbreakStrategy enum and its match block.
• Jailbreaks — which jailbreak templates to apply. A random sample by default (num_templates, default 10) or a specific set via jailbreak_names. Both are exposed as runtime parameters (--num-templates, --jailbreak-names). Each selected template is applied to every (technique x objective) via a TextJailbreakConverter, and results group by jailbreak template.

The selected template names are still persisted to ScenarioResult.metadata and replayed on --resume, and the chosen templates remain visible in the "Scenario Inputs" scenario output.

Jailbreak application uses the existing TextJailbreakConverter (which renders the objective into each template). The general-technique / role-based-delivery mechanism from the earlier review sketch (jailbreaks as is_general_technique seeds via with_technique, system/user delivery roles) is intentionally deferred: that path needs net-new infrastructure that doesn't exist yet, and the exact mechanism will change once the upcoming params refactor lands. This PR delivers the standardized 3-axis param structure now; the internal jailbreak-application mechanism can migrate later without changing the user-facing axes.

This is BREAKING because a default scan now produces different atomic attacks (new dataset, new default technique). VERSION is bumped to 3, so stored results from the old default raise cleanly on --resume instead of silently mixing. Constructor signatures and the public API are otherwise unchanged.

Naming note: the jailbreaks axis is exposed as jailbreak_names + num_templates, mirroring the dataset axis's dataset_names + max_dataset_size levers. Happy to consolidate to a single jailbreaks selector when the params refactor lands.

Tests and Documentation

Rewrote tests/unit/scenario/airt/test_jailbreak.py around the 3-axis shape: class-level properties (VERSION 3, harmbench default, param names), the dynamic strategy class (prompt_sending default, core techniques available), init validation (mutually-exclusive num_templates/jailbreak_names, unknown-name error), atomic-attack construction (default "just send" crosses templates x objectives, display_group = template, num_attempts multiplies with unique names, explicit strategy adds the technique axis, baseline prepended centrally), and resume (persisted names replayed). All pass; full scenario suite green; ruff, ruff-format, and ty clean.
Updated the AIRT scanner doc (doc/scanner/airt.py + .ipynb) to the 3-axis usage and fast path; jupytext .py/.ipynb pair in sync.

[rlundeen2]

I think this is easier to think about what we want it to do; and I think the goal is to see how effective various jailbreaks are (or find jailbreaks that work)

To do this, let's look at end parameters what a user actually sets on the CLI since I think that's the clearest way to see whether the design hangs together. Roughly the run I'd want to work:

pyrit_scan airt.jailbreak \
  --initializers target load_default_datasets \
  --target openai_chat \
  --dataset-names harmbench \           # WHAT: harmful objectives
  --jailbreak-names dan aim \           # WHICH jailbreaks (or --num-templates N)
  --strategies system user variation \  # HOW: delivery methods
  --num-attempts 1

Getting the below to work are over-simplifications, and I could see us needing to think about each one, and some may need a PR pre-req. But I think they should be close to working.

---

1. --dataset-names — the harmful objectives (the what)

Standard DatasetConfiguration. I'd switch the default off the small airt_harms sample to something effectiveness-oriented like HarmBench. Each entry is a SeedObjective; nothing exotic here.

2. --jailbreak-names / --num-templates — which jailbreaks

A selector with a default set that can be overridden. The jailbreaks themselves should be SeedAttackTechniqueGroups (is_general_technique=True) so they merge into every objective via with_technique() without touching the SeedObjective.

On how to build the selector: we don't need a heavy new class. DatasetConfiguration's generic core is already objective-agnostic — get_seed_groups() returns plain SeedGroups, and the select-by-name-or-explicit + max_dataset_size sampling + _load_seed_groups_for_dataset hook is exactly the "default set, or override" behavior we want. Only the terminal *_attack_groups() wrappers assume one objective. So: keep that selection core and add a technique-flavored terminal (get_technique_groups() → SeedAttackTechniqueGroups) plus a load hook that pulls the templates. The knobs then map straight across: names → jailbreak names, explicit seed_groups → override, max_dataset_size → --num-templates.

(Alternative if we'd rather not subclass: register the templates into memory as is_general_technique=True seed datasets and load them by name. Either works — just pick one.)

Note --jailbreak-names would be a new runtime parameter — today only --num-templates and --num-attempts are exposed in supported_parameters().

3. --strategies — the delivery methods (the how)

Registered as defaults, more selectable if registered:

• baseline (objective only — already auto-prepended)
• jailbreak on the system prompt — technique seed role="system"
• jailbreak as the first user prompt — technique seed role="user"
• variation of the jailbreak, on system + user
• translation (random language), on system + user

system-vs-user is just the seed's role — no special converter. Each method is an AttackTechniqueFactory / AttackTechnique carrying the jailbreak seed_technique at the right role + converter config, instead of the current match strategy block with TextJailbreakConverter. That gets the strategy enum back to pure techniques.

For variation/translation: attach the converter to the attack (don't pre-convert offline) and scope it with PrependedConversationConfig(apply_converters_to_roles=[...]). request_converters also hit the live objective turn, so the variation/translation lands on both the jailbreak and the objective — which I think is what we want, and it avoids a build-time LLM call so resume stays cheap.

4. --num-attempts — repeats per (jailbreak × method × objective)

Keep this knob, like it. Free from the base class.

---

Cross-cutting: output grouping

Group by jailbreak template — that's the axis people compare across. Heads-up: the default _build_display_group hook only gets technique_name + seed_group_name; the template is a third axis, so since we're keeping a thin _get_atomic_attacks_async, just set display_group=<template_name> directly when constructing each AtomicAttack.

Follow-up param: skip objectives the baseline already cracked

I like this (don't spend jailbreak attempts on objectives that fail open with no jailbreak), but it's more than a one-liner: today baseline is just another prepended AtomicAttack and we build + run all atomic attacks together, so there's no feed-forward. It needs a two-phase run (baseline → collect objectives that scored true → build/filter the rest). I'd land it as an opt-in flag (default off) in a follow-up rather than blocking this PR.

---

Net: keep a thin _get_atomic_attacks_async (the template axis is a genuine third dimension), but every delivery method becomes a factory carrying a role-tagged seed_technique + converter config. Strategy enum = pure techniques, jailbreak axis = a selector, objectives = --dataset-names.

[rlundeen2]

I think this should help: #2071

[varunj-msft]

I think this is easier to think about what we want it to do; and I think the goal is to see how effective various jailbreaks are (or find jailbreaks that work)

To do this, let's look at end parameters what a user actually sets on the CLI since I think that's the clearest way to see whether the design hangs together. Roughly the run I'd want to work:

pyrit_scan airt.jailbreak \
  --initializers target load_default_datasets \
  --target openai_chat \
  --dataset-names harmbench \           # WHAT: harmful objectives
  --jailbreak-names dan aim \           # WHICH jailbreaks (or --num-templates N)
  --strategies system user variation \  # HOW: delivery methods
  --num-attempts 1

Getting the below to work are over-simplifications, and I could see us needing to think about each one, and some may need a PR pre-req. But I think they should be close to working.

1. --dataset-names — the harmful objectives (the what)

Standard DatasetConfiguration. I'd switch the default off the small airt_harms sample to something effectiveness-oriented like HarmBench. Each entry is a SeedObjective; nothing exotic here.

2. --jailbreak-names / --num-templates — which jailbreaks

A selector with a default set that can be overridden. The jailbreaks themselves should be SeedAttackTechniqueGroups (is_general_technique=True) so they merge into every objective via with_technique() without touching the SeedObjective.

On how to build the selector: we don't need a heavy new class. DatasetConfiguration's generic core is already objective-agnostic — get_seed_groups() returns plain SeedGroups, and the select-by-name-or-explicit + max_dataset_size sampling + _load_seed_groups_for_dataset hook is exactly the "default set, or override" behavior we want. Only the terminal *_attack_groups() wrappers assume one objective. So: keep that selection core and add a technique-flavored terminal (get_technique_groups() → SeedAttackTechniqueGroups) plus a load hook that pulls the templates. The knobs then map straight across: names → jailbreak names, explicit seed_groups → override, max_dataset_size → --num-templates.

(Alternative if we'd rather not subclass: register the templates into memory as is_general_technique=True seed datasets and load them by name. Either works — just pick one.)

Note --jailbreak-names would be a new runtime parameter — today only --num-templates and --num-attempts are exposed in supported_parameters().

3. --strategies — the delivery methods (the how)

Registered as defaults, more selectable if registered:

• baseline (objective only — already auto-prepended)
• jailbreak on the system prompt — technique seed role="system"
• jailbreak as the first user prompt — technique seed role="user"
• variation of the jailbreak, on system + user
• translation (random language), on system + user

system-vs-user is just the seed's role — no special converter. Each method is an AttackTechniqueFactory / AttackTechnique carrying the jailbreak seed_technique at the right role + converter config, instead of the current match strategy block with TextJailbreakConverter. That gets the strategy enum back to pure techniques.

For variation/translation: attach the converter to the attack (don't pre-convert offline) and scope it with PrependedConversationConfig(apply_converters_to_roles=[...]). request_converters also hit the live objective turn, so the variation/translation lands on both the jailbreak and the objective — which I think is what we want, and it avoids a build-time LLM call so resume stays cheap.

4. --num-attempts — repeats per (jailbreak × method × objective)

Keep this knob, like it. Free from the base class.

Cross-cutting: output grouping

Group by jailbreak template — that's the axis people compare across. Heads-up: the default _build_display_group hook only gets technique_name + seed_group_name; the template is a third axis, so since we're keeping a thin _get_atomic_attacks_async, just set display_group=<template_name> directly when constructing each AtomicAttack.

Follow-up param: skip objectives the baseline already cracked

I like this (don't spend jailbreak attempts on objectives that fail open with no jailbreak), but it's more than a one-liner: today baseline is just another prepended AtomicAttack and we build + run all atomic attacks together, so there's no feed-forward. It needs a two-phase run (baseline → collect objectives that scored true → build/filter the rest). I'd land it as an opt-in flag (default off) in a follow-up rather than blocking this PR.

Net: keep a thin _get_atomic_attacks_async (the template axis is a genuine third dimension), but every delivery method becomes a factory carrying a role-tagged seed_technique + converter config. Strategy enum = pure techniques, jailbreak axis = a selector, objectives = --dataset-names.

Rebuilt this around the 3 axes: harmbench objectives, techniques (default just-send + the core set), and jailbreak selection via num_templates/jailbreak_names on TextJailbreakConverter. I left the delivery-method/general-technique axis for a follow-up since the shape might shift if a refactor is in play. Mentioned in description also.

[rlundeen2]

Similar to the previous ones, this branch is really far behind main and has a bunch of conflicts. The contract has changed (and should be simpler)

You likely will want to start from a branch off of main