Skip to content

[BREAKING] MAINT: Standardize garak.encoding defaults and fix atomic-attack name collisions#2058

Open
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/8380-Standardizing-Scenarios-Garak-Encoding-Defaults
Open

[BREAKING] MAINT: Standardize garak.encoding defaults and fix atomic-attack name collisions#2058
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/8380-Standardizing-Scenarios-Garak-Encoding-Defaults

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Description

Part of the Standardizing Scenarios effort. This standardizes the garak.encoding
scenario so its default run is fast and representative instead of exhaustive, and
fixes a latent atomic-attack naming bug.

Two changes:

  1. Default strategy is now a curated DEFAULT aggregate (Base16, ROT13, MorseCode —
    one base-N, one substitution cipher, one symbolic alphabet) instead of ALL. This
    drops a default run from 96 to 16 atomic attacks. ALL is still available for an
    exhaustive run. The fast path is --strategies rot13 --max-dataset-size 1.

  2. Atomic-attack names were not unique: every converter variant of an encoding shared
    the encoding name (e.g. all four base64 variants × five prompt configs were named
    "base64"). Since results and the display map are keyed by atomic_attack_name, those
    collapsed to a single key, corrupting result tracking and --resume. Names are now
    unique per variant (e.g. base64_urlsafe_decode0), and display_group keeps the
    per-encoding grouping in reports.

The encoding axis here is the scheme, not an attack technique, so SINGLE_TURN/MULTI_TURN
aggregates don't apply and are intentionally not added.

Breaking: same constructor call now produces different default atomic attacks, so the
scenario VERSION is bumped 1 -> 2. On --resume against an old result this raises a clear
ValueError instead of silently merging incompatible runs. Public API and constructor
signatures are unchanged.

Also adds a backend regression test pinning the real EncodingDatasetConfiguration
round-trip, since the backend silently degrades a lost config subclass to a base config.

Tests and Documentation

Tests (tests/unit/scenario/garak/test_encoding.py, tests/unit/backend/test_scenario_run_service.py):

  • VERSION bump and DEFAULT-as-default-strategy
  • DEFAULT membership = {Base16, ROT13, MorseCode}, and DEFAULT ⊂ ALL
  • atomic counts: DEFAULT=16, ALL=96, ROT13 fast path=6
  • name uniqueness under ALL (96 unique) and across multi-variant encodings
  • display_group still groups variants by encoding
  • baseline-disabled and custom-template count edge cases
  • real EncodingDatasetConfiguration round-trips through the backend service

Full scenario unit suite passes (708). ruff, ruff format, and ty are clean.

Documentation (doc/scanner/garak.py + .ipynb):

  • documented DEFAULT (curated) vs ALL (exhaustive) and the rot13 fast path
  • ran jupytext to keep the .py and .ipynb in sync; notebook outputs preserved

"""

# Aggregate member
# Aggregate members

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make this run more attacks? What's the run time, and how does it compare to other scanners?

I think target of 10-20 minutes is good and this may finish too qquickly

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Measured after curation: DEFAULT ~0.7 min, ALL ~3.3 min (seed-capped). Getting into the 10–20 min range means raising the per-dataset seed caps. Do you see that as a tuning/follow up or a blocker for this PR?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the biggest change in this PR is changing the defaults, so we might want to change the dataset size a bit and adding more defaults

Comment thread pyrit/scenario/scenarios/garak/encoding.py Outdated
@rlundeen2 rlundeen2 self-assigned this Jun 20, 2026
@varunj-msft varunj-msft force-pushed the varunj-msft/8380-Standardizing-Scenarios-Garak-Encoding-Defaults branch 2 times, most recently from 8fe1194 to 2b54413 Compare July 1, 2026 02:55
@varunj-msft varunj-msft force-pushed the varunj-msft/8380-Standardizing-Scenarios-Garak-Encoding-Defaults branch from 2b54413 to 411f918 Compare July 6, 2026 21:18

@rlundeen2 rlundeen2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to psychosocial this has a lot of conflicts with main. But the fixes here are all applicable.

Rebase onto main, then re-apply the three genuinely valuable changes inside the new structure:

  • EncodingStrategy: add DEFAULT = ("default", {"default"}), tag Base16/ROT13/MorseCode, and the get_aggregate_tags() override. This part is contract-agnostic and ports verbatim. Flip default_strategy=EncodingStrategy.DEFAULT and bump VERSION → 2.
  • Name-collision fix: add variant_slug to the converter table and set atomic_attack_name=f"{variant_slug}_{config_suffix}" + display_group=encoding_name — but inside main's _get_prompt_attacks(..., seed_groups=...) signature (seeds from context, not self._resolved_seed_groups).
  • Drop the manual baseline block entirely — baseline is now central. Several of the PR's tests assert on manual baseline behavior and will need reworking against the central path.
  • Tests: EncodingDatasetConfiguration is now a DatasetAttackConfiguration; the backend round-trip test and _get_prompt_attacks call sites need the new signatures.

And I recommend tweaking the defaults/dataset sizes further

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants