AgenticVBench-Omni

AgenticVBench-Omni / Meetings and procedures

Europarl

europarl-2026-single-market-speaker-turn-ledger

You are given:

Family
Meetings and procedures
Inputs
videoaudiodocuments
Media
1 video, 2 h 8 min, with sound
Reference answer
86 speaker turns
Deliverable
/workspace/output/solution.json
Scoring
F1 / precision–recall

Task package · verifier code

Instruction

Full instruction given to the agent

European Parliament Speaker-Turn Ledger

You are given:

  • /workspace/materials/debate.mp4: one continuous European Parliament debate with original floor audio;
  • /workspace/materials/roster.json: the anonymous speaker IDs available for your answer;
  • /workspace/materials/roster/: one official reference portrait per anonymous speaker ID;
  • /workspace/materials/excerpts.json: shuffled anonymous English inference cards, each stating a source-entailed counterfactual, implication, policy test, or premise/outcome tension from one scored intervention.

Reconstruct the complete chronological ledger of substantive speaker turns.

A substantive turn:

  • belongs to the person delivering the speech, not the presiding chair;
  • lasts at least 30 seconds;
  • starts at the floor-audio handover into that speaker's intervention, which can precede the first spoken word by a brief pause;
  • ends at the floor-audio handover away from that speaker.

The scored official interval may include a few seconds of overlapping chair speech or room audio at either handover. Report the official intervention window, not a voice-activity-only crop.

Exclude short chair or presiding-officer interjections. Preserve separate appearances by the same speaker as separate turns. Include borderline interventions that span approximately 30 seconds rather than rounding them down. Use the portraits and visible speaker to identify speaker_id; use the floor audio to locate the intervention boundaries. Also identify the language spoken on the original floor track. Use one of these ISO 639-1 codes:

bg, cs, da, de, el, en, es, fi, fr, hr, hu, it, lt, lv, nl, pl, pt, ro, sk, sl, sv.

The offline command detect-language <audio.wav> [...] is available. It accepts one or more 16 kHz mono WAV excerpts and returns JSON with the top language codes. The offline command transcribe-audio [--task transcribe|translate] <audio.wav> [...] is also available and returns detected language plus transcript text. Translation mode produces English text for cross-lingual card matching. Each input excerpt must be at most 180 seconds; split long audio before invoking the command so one broad transcription cannot consume the task runtime. Run only one transcribe-audio process at a time; pass multiple clips to that one command to batch them safely within the memory limit. Treat language detection as a candidate ranking rather than an oracle and cross-check uncertain or accented speech against the transcript.

Match each intervention's spoken content to exactly one excerpt_id from excerpts.json. The cards express inferential consequences rather than quotations or summaries; they are shuffled and do not reveal speaker identity, order, or timestamps.

Boundary estimates should be within 3.5 seconds.

Write /workspace/output/solution.json in exactly this shape:

{
  "turns": [
    {
      "turn_index": 1,
      "speaker_id": "speaker_001",
      "language_code": "en",
      "excerpt_id": "excerpt_001",
      "start_time_s": 42.0,
      "end_time_s": 105.0
    }
  ]
}

Requirements:

  • turn_index: consecutive one-based chronological index.
  • speaker_id: one ID present in roster.json.
  • language_code: one code from the supplied closed vocabulary.
  • excerpt_id: one ID present in excerpts.json.
  • start_time_s, end_time_s: seconds from the beginning of debate.mp4.
  • Turns must be ordered chronologically and each end must be after its start.

Use only the supplied materials. Do not use internet lookup, public schedules, external speaker lists, prior knowledge of the sitting, or memory of public figures.

Media

Media files are distributed separately from the repository. Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.337
Claude Opus 5.50.256
Claude Opus 50.058
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent gets one continuous European Parliament debate with its original floor audio, in which members speak their own languages; a roster of 742 anonymous speaker IDs, each with an official portrait; and 86 shuffled cards in English, each stating something that follows from one speech. It lists every substantive speaker turn in order: speaker ID, language, the card that belongs to the speech, start and end. The reference holds 86 turns by 79 speakers.

What a good answer has to get right, and how the verifier checks it

Every turn is found, in order, and bounded
Entries are aligned with the reference in order. A pair can align only when start and end are each within 3.5 s.
The speaker is identified
The speaker ID must be exact, out of 742 portraits.
The language is identified
The language code must be exact, one of 21.
The speech is understood
The card attached to the turn must be the one written from that speech. The cards are in English and state implications instead of quoting, so matching one requires following what was said in the speaker’s own language.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall), with recall over the 86 turns and precision over every turn in the file, including turns the verifier rejects. A turn counts only when speaker, language, card and both times are right.

What scores zero

No solution file, one over 2 MB, one that is not an object with a “turns” list, or more than 1000 turns. Otherwise, a list in which no turn is right on every field.

Outside the score: speech that the task does not count as a substantive turn.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
boundary_tolerance_s3.5value the verifier reports using

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.34
Wall time
25.7 min
Steps shown
103

Loading the run…