Search results

[FT] Introduce a static 'lint' command to validate custom tasks before execution

Status: Open.

#1266 In huggingface/lighteval;

· Kesav2k04 opened on Jun 18, 2026

MMLU-Pro incorrect prompt format

bug

Status: Open.

#1265 In huggingface/lighteval;

· RasmusHoier opened on Jun 17, 2026

Lighteval's LCB grader fails correct solutions that read sys.stdin.buffer

bug

Status: Open.

#1255 In huggingface/lighteval;

· RasmusHoier opened on Jun 12, 2026

Guidance on adding REFUTE benchmark task (scientific critique & calibration)

Status: Open.

#1253 In huggingface/lighteval;

· connerlambden opened on Jun 5, 2026

[FT] IRT-based scoring as alternative ranking method for fairer model comparison

Status: Open.

#1240 In huggingface/lighteval;

· testofschool opened on May 20, 2026

[BUG] [vLLM] Batch truncation bug: context length check uses first prompt instead of longest

bug

Status: Open.

#1204 In huggingface/lighteval;

· paulovsantanas opened on Apr 6, 2026

Custom task: RAIL Score responsible AI evaluation (8-dimension scoring)

Status: Open.

#1202 In huggingface/lighteval;

· SumitVermakgp opened on Apr 2, 2026

[META] Is this repo still maintained? Community PRs going unreviewed for months

Status: Open.

#1200 In huggingface/lighteval;

· kryvokhyzha opened on Mar 29, 2026

[BUG]

bug

Status: Open.

#1198 In huggingface/lighteval;

· w-ahmad1a10 opened on Mar 23, 2026

squad_v2.py : unanswerable questions are filtered out, making the task equivalent to SQuAD 1.1

Status: Open.

#1184 In huggingface/lighteval;

· Matteovanypersele opened on Mar 9, 2026

[BUG] LogProbTokenNorm crashes with IndexError in CFFormulation for belebele

bug

Status: Open.

#1170 In huggingface/lighteval;

· Rijgersberg opened on Feb 18, 2026

Add jfinqa: Japanese Financial Numerical Reasoning QA

Status: Open.

#1168 In huggingface/lighteval;

· ajtgjmdjp opened on Feb 17, 2026