feat: Add AI online evaluations (Judge, Evaluator, IRunner)

[mattrmc1]

Summary

Adds AI online evaluations to LaunchDarkly.ServerSdk.Ai. A caller that supplies a runnerFactory to LdAiClient gets automatic judge evaluation wired into every CompletionConfig and AgentConfig — each returned config carries an Evaluator that can score model output against the judges declared in the flag's judgeConfiguration. When no runnerFactory is provided (or no judges are configured), configs receive a noop Evaluator so callers never need null checks.

Implements the AIEVALS and AIRUNNER specs (sections 1.1–1.4). createJudge (AIEVALS 1.2) is intentionally omitted per .NET/Java convention — judges are created internally by the SDK, not by user code.

New public types

// Provider-facing runner interface (AIRUNNER 1.2)
public interface IRunner
{
    Task<RunnerResult> RunAsync(string input,
        IReadOnlyDictionary<string, object> outputType = null);
}

// Runner return type (AIRUNNER 1.3)
public sealed record RunnerResult(
    string Content,
    AiMetrics Metrics,
    object Raw = null,
    IReadOnlyDictionary<string, object> Parsed = null);

// Evaluation orchestrator (AIEVALS 1.4)
public sealed class Evaluator
{
    public static Evaluator Noop();
    public Task<IReadOnlyList<JudgeResult>> EvaluateAsync(string input, string output);
}

// Single-judge executor (AIEVALS 1.1)
public sealed class Judge
{
    public LdAiJudgeConfig Config { get; }
    public IRunner Runner { get; }
    public Task<JudgeResult> EvaluateAsync(string input, string output, double? samplingRate = null);
    public Task<JudgeResult> EvaluateMessagesAsync(
        IReadOnlyList<LdAiConfigTypes.Message> messages,
        RunnerResult runnerResult, double? samplingRate = null);
}

LdAiClient changes

// New optional parameter
public LdAiClient(ILaunchDarklyClient client,
    Func<LdAiJudgeConfig, IRunner> runnerFactory = null);

When runnerFactory is non-null, ConfigFactory.BuildEvaluator iterates the flag's judgeConfiguration, evaluates each judge key as a flag variation, creates a Judge + IRunner pair per enabled judge, and attaches the resulting Evaluator to the config. Disabled judges, null runners, and initialization exceptions are logged and skipped — no single judge failure prevents the others from being built.

LdAiConfig base class

All config types (LdAiCompletionConfig, LdAiAgentConfig, LdAiJudgeConfig) now carry an Evaluator property via the base class. LdAiJudgeConfig always receives Evaluator.Noop() (judges don't evaluate themselves).

JudgeResult changes

Updated to match AIEVALS 1.3.1 defaults:

Field	Before	After
MetricKey	required	optional (default null)
Score	required	optional (default 0.0)
Sampled	default true	default false
Success	default true	default false
ErrorMessage	—	new field
Reasoning	—	new field

Null safety (ref: java-core#175 discussion)

BuildEvaluator handles every failure mode without throwing:

• runnerFactory == null or empty judgeConfiguration → Evaluator.Noop()
• Runner factory returns null → warn + skip judge
• Judge config disabled → warn + skip judge
• Any exception during judge init → warn + skip judge
• Missing evaluationMetricKey → JudgeResult(success: false, errorMessage: ...)
• Score out of [0, 1] range → JudgeResult(success: false, errorMessage: ...)
• Sampling rate NaN/Infinity/negative/> 1.0 → normalized to safe bounds

Judge constructor still uses ArgumentNullException guards, but these are never hit in practice because BuildEvaluator validates inputs before construction.

Migration

None required. The LdAiClient constructor gains an optional runnerFactory parameter (default null) — existing callers are unaffected. JudgeResult default changes are source-compatible (all parameters are now optional with safe defaults). No members removed or renamed.

Test plan

• dotnet test pkgs/sdk/server-ai/test/LaunchDarkly.ServerSdk.Ai.Tests.csproj --framework net8.0 passes
• JudgeTest (435 lines) covers: successful evaluation with score/reasoning extraction, sampling skip path, samplingRate edge cases (NaN, negative, > 1.0), runner exception handling, missing evaluationMetricKey validation, out-of-range score rejection with errorMessage, evaluateMessages formatting, null/empty message handling
• EvaluatorTest (210 lines) covers: noop returns empty list, multi-judge execution with per-judge sampling, missing judge key logs warning and skips, noop does not log warnings
• LdAiCompletionConfigTest (189 lines added) covers: Evaluator attached when runnerFactory provided, noop Evaluator when no runner factory, noop when judgeConfiguration is empty, disabled judge skipped, null runner skipped
• LdAiJudgeConfigTest covers: JudgeResult ErrorMessage and Reasoning fields, judge config always receives noop Evaluator
• LdAiConfigTrackerTest covers: TrackJudgeResult with new optional fields

---

Note

Medium Risk
New async judge execution and extra flag evaluations on config build can affect latency and cost; JudgeResult default changes are source-compatible but may alter behavior for callers relying on old implicit defaults.

Overview
Adds AI online evaluations to the server AI SDK: callers can pass an optional runnerFactory to LdAiClient so completion and agent configs get a non-noop Evaluator when the flag includes judgeConfiguration.

ConfigFactory now builds evaluators by resolving each judge key as a flag variation, creating Judge + IRunner pairs (with failures logged and skipped), and attaching the result on normal and default fallback paths—including agent graphs via graphKey for judge tracking. Judge prompt variables from the parent config call are forwarded into nested judge config builds.

New public surface: IRunner, RunnerResult, Judge, and Evaluator (with Noop()). LdAiConfig exposes a always-non-null Evaluator; judge configs use Evaluator.Noop().

JudgeResult gains ErrorMessage and Reasoning, looser optional constructor defaults (Sampled/Success now default false), and JudgeConfiguration.Judge.SamplingRate is double? (omitted on wire when null; parse only reads numeric samplingRate).

^{Reviewed by Cursor Bugbot for commit 6f7d88d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6f7d88d. Configure here.}

[cursor]

Judge sampling ignores interpolate flag

Medium Severity

BuildEvaluator always calls BuildJudgeConfig with the default interpolate: true, even when the parent config was built with interpolate: false (template APIs). Nested judge configs then run Mustache interpolation (including ldctx) while completion/agent instructions and messages stay as raw templates, so template configs and runnerFactory output can disagree with the documented template behavior.

Additional Locations (2)

• pkgs/sdk/server-ai/src/Config/ConfigFactory.cs#L267-L308
• pkgs/sdk/server-ai/src/LdAiClient.cs#L156-L165

 

^{Reviewed by Cursor Bugbot for commit 6f7d88d. Configure here.}

[cursor]

Same random draw per judges

Medium Severity

Each EvaluateAsync call uses new Random().NextDouble() for sampling. When an Evaluator runs multiple judges in one request, those instances are often created in the same tick and share the same seed, so they draw the same value and always skip or always run together instead of independent per-judge sampling.

 

^{Reviewed by Cursor Bugbot for commit 6f7d88d. Configure here.}

[cursor]

Duplicate judge keys run twice

Low Severity

If judgeConfiguration.judges lists the same key more than once, BuildEvaluator stores one Judge in a dictionary but builds filteredConfig from the full list without deduplicating. EvaluateAsync then invokes the same judge once per duplicate entry and returns multiple results for one logical judge.

Additional Locations (1)

• pkgs/sdk/server-ai/src/Config/ConfigFactory.cs#L305-L307

 

^{Reviewed by Cursor Bugbot for commit 6f7d88d. Configure here.}