One command to benchmark AI guardrails and coding agents across safety, security, jailbreak, prompt-injection, and secure-code tasks.
BYOA Agent harness that ensures alignment end-to-end – with built-in eval system.
Local Codex MCP harness: contracts, persistent RAG memory, raw traces, verification records, governance policy and PASS/FLAG/BLOCK audits, observability reports, harness profiles, eval runs, Meta-Harness-lite promotion records, natural-language harness specs, MCP resources/prompts, multi-client installer, and completion gates.
Evaluate AI agents with Unix-style pipeline commands. Schema-driven adapters for any CLI agent, trajectory capture, pass@k metrics, and multi-run comparison.
Deterministic synthetic two-party conversation corpus generator for testing AI scoring systems.
LLM-powered clinical extraction + structured evals. Prompt strategies, hallucination detection, and per-field F1 scoring.
A from-scratch hybrid rule + LLM-as-judge eval harness: pass-rate, judge-vs-rule Cohen's kappa, latency/cost, HTML report, CI gate. Runs offline.
Production-minded LLM eval harness for safety, reliability, cost, and latency analysis.
Prototype adapter for CLI agents to play Crusader Kings III through a constrained, auditable CK3 mod bridge.
Production-oriented template for building AI agent skills as verifiable software components — offline eval harness, source-grounding validators, structured logs/traces/metrics, replay artifacts, and a CI quality gate. Runs fully offline with a deterministic mock model.
Document processing pipeline engine — adapters, contracts, domain packs, eval harnesses
Open Agentsia Labs benchmark harness for model runners, multi-turn evals, reproducible rubric scoring, proof bundles, and RunRecord output.
Clinical AI platform: LangGraph RAG over PubMed, ICD-10/CPT/HCPCS billing intelligence, 3 role-based portals. FastAPI · PostgreSQL · Pinecone · React · AWS EC2 · Faithfulness 0.93
Codex-native OZM skill pack for AI coding agent governance, agentic coding loops, claim ceilings, and AGENTS.md-aware workflows.
Verification-native local coding agent runtime with eval gates, memory, subagents, and model profiles.
She who catches hubris in agents — a Python evaluation harness for agentic failure modes.
Read-only BigQuery MCP server with mandatory dry-run cost caps, agent-friendly structured errors, and a Spider/BIRD-style NL-to-SQL eval harness.
Tolerant apply_patch for LLM-generated diffs, plus an eval harness for code agents.
YAML-driven evaluation harness for WhatsApp RAG bots
Production-style LLM evaluation harness for structured clinical extraction — compares prompt strategies across accuracy, cost, and hallucination.
Add a description, image, and links to the eval-harness topic page so that developers can more easily learn about it.
To associate your repository with the eval-harness topic, visit your repo's landing page and select "manage topics."