AgenticVBench-Omni

AgenticVBench-Omni / Meetings and procedures

Council

council-meeting-motion-ledger

You are given the public recordings of fourteen consecutive meetings of a small city commission (January to August), their agendas, and a roster with seating charts.

Family
Meetings and procedures
Inputs
videoaudiodocuments
Media
14 videos, 52 h 33 min, no sound track; 14 audio files, 52 h 40 min
Reference answer
103 motions
Deliverable
/workspace/output
Scoring
F1 / precision–recall

Task package · verifier code

Instruction

Full instruction given to the agent

Council Motion Ledger

You are given the public recordings of fourteen consecutive meetings of a small city commission (January to August), their agendas, and a roster with seating charts.

Files in the working directory:

  • m01_video.mp4 … m14_video.mp4 — the meeting videos, 480p, no sound. m04 holds two back-to-back meetings: a short organizational meeting (the outgoing commission adjourns, a newly elected commissioner is sworn in, officers are chosen) and then that day's regular meeting. Some recordings also contain other bodies' meetings or workshops held in the same room before the commission meeting.
  • m01_audio.m4a … m14_audio.m4a — each meeting's sound, in a separate file. The audio has been edited: many pauses were lengthened or shortened, each by a different amount, so the audio does not line up with the video and the offset between them changes through the recording. Nothing in the files tells you how they line up; work it out from their content.
  • agendas/ — the published agenda of every session: m01.txt … m14.txt, plus m04_org.txt for the organizational meeting.
  • roster.json — the six people who held commission seats in this period (ids belhumeur, cooley, cunningham, santore, sherman, spradley), the non-voting mayor, and, for every session, which roster id sits in each dais seat, numbered from left to right as seen in the wide camera shot of the dais.

Sessions

session is one of m01, m02, m03, m04_org, m04, m05, … m14 (m04_org is the organizational meeting inside m04_video.mp4; m04 is the regular meeting that follows it).

Task

Reconstruct the ledger of every motion of the City Commission in these sessions: every motion that was made, seconded and put to a vote. Do not report: nominations; motions to adjourn; motions that die for lack of a second; items the commission only discussed, took no action on, or settled by consensus; items removed from the agenda; and anything in another body's meeting or a workshop held in the same recording.

For each motion report:

  • session, and agenda_item — the item the motion was taken on, written as in that session's agenda (3a, 6b, 8c). A motion approving the consent agenda as a whole uses the consent section number (5, 6).
  • seq — 1 for the first ledger motion on that agenda item in that session, 2 for the second, and so on.
  • moved_at — hh:mm:ss (decimals allowed) in that session's video timeline, at the first word of the sentence in which the mover makes the motion. Tolerance: 2 seconds.
  • result_at — hh:mm:ss in the video timeline, at the first word of the sentence in which the chair or clerk announces the result of the vote (for example "Motion passes unanimously"). Tolerance: 2 seconds. Use null if the result is never announced aloud.
  • mover, seconder — roster ids.
  • outcome — carried or failed.
  • votes — an object with all six roster ids as keys, each yes, no, or not_voting (not_voting = not seated as a commissioner on the dais for that vote, or abstained). On a voice vote where nobody votes no, every seated commissioner is yes.
  • view_at_motion and view_at_result — what the video shows at moved_at and at result_at. One of:
  • dais — the wide shot of the commissioners' dais;
  • podium — the public lectern and/or the audience seating, at any zoom;
  • staff — close shot of the staff table (city attorney and city manager);
  • clerk — close shot of the clerk's table;
  • slides — a full-screen presentation, map or document;
  • card — a full-screen title card (for example "recess" or "starting soon").
  • prior — the earlier ledger motion this motion follows up, as [session, agenda_item, seq], or null. It is set only when: the motion is on an ordinance that already had a ledger motion at an earlier session (use its most recent earlier ledger motion); the item was tabled by an earlier ledger motion (use that motion); or the motion reconsiders or amends an action already voted earlier in the same session (use the motion it reconsiders or amends, or the most recent ledger motion in that chain). Resolution numbers alone do not create a link.

Each entry is judged as a whole record.

Output

Write output/motions.json:

{
  "motions": [
    {"session": "m03", "agenda_item": "9z", "seq": 1, "moved_at": "00:00:00.0", "result_at": "00:00:00.0",
     "mover": "<roster id>",
     "seconder": "<roster id>", "outcome": "carried",
     "votes": {"belhumeur": "yes", "cooley": "no", "cunningham": "yes", "santore": "not_voting",
               "sherman": "yes", "spradley": "yes"},
     "view_at_motion": "dais", "view_at_result": "staff", "prior": ["m02", "8c", 1]}
  ]
}

One entry per motion, any order. The session can end at any time and only what is in output/motions.json counts, so write the file early and rewrite it as you add motions.

Rules

  • Stay inside this working directory. There is no internet access. Do not rely on memory or outside knowledge of this city, its commissioners or its meetings; every entry must come from the files here.
  • Installed tools: ffmpeg/ffprobe, sox, Python 3 with numpy, scipy, pandas, pillow, opencv-python-headless, soundfile, matplotlib, and faster_whisper with the small speech-to-text model already cached (WhisperModel("small", device="cpu", compute_type="int8", local_files_only=True)). pip install will not work.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 44 files (environment/media/SHA256SUMS.txt). Licences named for this task: public records.

Score

AgentScore (0 to 1)
GPT-6-Astra0.078
Claude Opus 5.50.359
Claude Opus 50.029
Gemini 3.8 Flash0.010
DeepSeek 4.1 Flash0.000

One run per agent.

The agent gets the recordings of fourteen meetings of a small city commission, with picture and sound in separate files, plus the agendas and a roster with seating charts. It lists every motion: when it was moved and when its result was announced, who moved and seconded it, the outcome, how each of the six commissioners voted, which camera view was live at those two moments, and the earlier motion it follows from, if any. The reference holds 103 motions.

What a good answer has to get right, and how the verifier checks it

Every motion is found under the right item
A motion is identified by its meeting, agenda item and order within that item. An entry whose key is not in the reference, or repeats one already used, is a false positive.
The motion and its result are timed
The moment it was moved and the moment the result was announced must each be within 2 s.
Mover and seconder are named
Both must be exact. They are checked only where the recording itself makes the name certain: the name is spoken with the role, or the voice matches clearly.
The outcome and every vote are right
The outcome must be exact, and so must the vote recorded for each of the six commissioners.
The picture is used as well as the sound
The camera view that was live when the motion was made, and when its result was announced, must both be exact.
Linked motions are linked
Where a motion amends or follows an earlier one, the link to that motion must be right; where it does not, none may be given.

How the checks become one score

Score = F1 = 2 × matched / (103 + entries counted). Every motion counts the same, and a motion counts only when every checked field is right: one wrong vote makes it a miss and a false positive.

What scores zero

No answer or one that cannot be read, a “motions” field that is not a list, an empty list, or a list in which no motion is right on every checked field.

Scores from this verifier: the reference answer with one thing changed

SubmissionScoreMatchedListed
Careful work
The reference answer1.000103103
Every time 1 s offEach time is reported 1 s late or early, alternately.1.000103103
One kind of mistake
Every motion recorded as unanimousEach “no” vote is recorded as “yes” (15 motions have one); absences are kept. Everything else is correct.0.85488103
The first 7 meetings onlyMotions of the first 7 of 14 meetings are correct; the other meetings are left out.0.7045656
Camera view always the daisThe live camera view is given as the dais at both moments of every motion. Everything else is correct.0.53455103
Mover and seconder swappedEverything else is correct. Motions where neither name is checked still count.0.20421103
Every time 5 s lateEverything else is correct.0.0000103
No real work
Items and outcomes, nothing elseEvery motion is listed under the right item with its outcome, with all times at zero and no names, votes or views.0.0000103
No file0.00000
An agent’s answer
A sampled runClaude Opus 5.5 · Claude Code 2.1.280. The run, step by step0.35937103

Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: 13 further motions that the reference marks as don’t-care, for which entries are dropped, and any field the reference leaves empty because it could not be confirmed from the recordings. The verifier also reports accuracy per field, as diagnostics.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5.5 · Claude Code 2.1.280
Score
0.36
Wall time
156.2 min
Steps shown
2962

Loading the run…