[BREAKING] MAINT: Standardize garak.encoding defaults and fix atomic-attack name collisions#2058
Conversation
| """ | ||
|
|
||
| # Aggregate member | ||
| # Aggregate members |
There was a problem hiding this comment.
Should we make this run more attacks? What's the run time, and how does it compare to other scanners?
I think target of 10-20 minutes is good and this may finish too qquickly
There was a problem hiding this comment.
Measured after curation: DEFAULT ~0.7 min, ALL ~3.3 min (seed-capped). Getting into the 10–20 min range means raising the per-dataset seed caps. Do you see that as a tuning/follow up or a blocker for this PR?
There was a problem hiding this comment.
I think the biggest change in this PR is changing the defaults, so we might want to change the dataset size a bit and adding more defaults
8fe1194 to
2b54413
Compare
2b54413 to
411f918
Compare
rlundeen2
left a comment
There was a problem hiding this comment.
Similar to psychosocial this has a lot of conflicts with main. But the fixes here are all applicable.
Rebase onto main, then re-apply the three genuinely valuable changes inside the new structure:
- EncodingStrategy: add DEFAULT = ("default", {"default"}), tag Base16/ROT13/MorseCode, and the get_aggregate_tags() override. This part is contract-agnostic and ports verbatim. Flip default_strategy=EncodingStrategy.DEFAULT and bump VERSION → 2.
- Name-collision fix: add variant_slug to the converter table and set atomic_attack_name=f"{variant_slug}_{config_suffix}" + display_group=encoding_name — but inside main's _get_prompt_attacks(..., seed_groups=...) signature (seeds from context, not self._resolved_seed_groups).
- Drop the manual baseline block entirely — baseline is now central. Several of the PR's tests assert on manual baseline behavior and will need reworking against the central path.
- Tests: EncodingDatasetConfiguration is now a DatasetAttackConfiguration; the backend round-trip test and _get_prompt_attacks call sites need the new signatures.
And I recommend tweaking the defaults/dataset sizes further
Description
Part of the Standardizing Scenarios effort. This standardizes the garak.encoding
scenario so its default run is fast and representative instead of exhaustive, and
fixes a latent atomic-attack naming bug.
Two changes:
Default strategy is now a curated DEFAULT aggregate (Base16, ROT13, MorseCode —
one base-N, one substitution cipher, one symbolic alphabet) instead of ALL. This
drops a default run from 96 to 16 atomic attacks. ALL is still available for an
exhaustive run. The fast path is
--strategies rot13 --max-dataset-size 1.Atomic-attack names were not unique: every converter variant of an encoding shared
the encoding name (e.g. all four base64 variants × five prompt configs were named
"base64"). Since results and the display map are keyed by atomic_attack_name, those
collapsed to a single key, corrupting result tracking and --resume. Names are now
unique per variant (e.g. base64_urlsafe_decode0), and display_group keeps the
per-encoding grouping in reports.
The encoding axis here is the scheme, not an attack technique, so SINGLE_TURN/MULTI_TURN
aggregates don't apply and are intentionally not added.
Breaking: same constructor call now produces different default atomic attacks, so the
scenario VERSION is bumped 1 -> 2. On --resume against an old result this raises a clear
ValueError instead of silently merging incompatible runs. Public API and constructor
signatures are unchanged.
Also adds a backend regression test pinning the real EncodingDatasetConfiguration
round-trip, since the backend silently degrades a lost config subclass to a base config.
Tests and Documentation
Tests (tests/unit/scenario/garak/test_encoding.py, tests/unit/backend/test_scenario_run_service.py):
Full scenario unit suite passes (708). ruff, ruff format, and ty are clean.
Documentation (doc/scanner/garak.py + .ipynb):