AgenticVBench-Omni

AgenticVBench-Omni / Music and synchronization

Piano Duo

piano-duo-score-deviation-ledger

Your working directory contains:

Family
Music and synchronization
Inputs
videoaudiodocuments
Media
1 video, 25 min, with sound
Reference answer
186 departures
Deliverable
/workspace/output
Scoring
F1 / precision–recallone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

Two-Piano Recital: Departures from the Score

Your working directory contains:

  • recital.mp4: a film of a recital (about 25 minutes, 1280x720, 30 fps, one mono audio track). Two identical concert grand pianos, Piano I and Piano II, play four pieces once each, in the order A, B, C, D, with a pause of several seconds between pieces. The pianos have no players; their keys and dampers move on their own. Both pianos use the same sampled sound and are mixed into the single mono channel, so they sound identical.
  • scores/A.pdf, scores/B.pdf, scores/C.pdf, scores/D.pdf: the printed score of each piece, as scanned page images. In every system the upper pair of staves (bracketed "Piano I") belongs to Piano I and the lower pair ("Piano II") to Piano II.

Telling the pianos apart on screen: each piano has a nameboard reading PIANO I or PIANO II just behind its keys, and a card reading I or II on its music desk. The camera cuts between hand-held close-ups that show part of one keyboard, wide shots of both pianos, and cutaways that show no keyboard. Every departure happens while the keyboard of the piano that made it is on screen.

Task

The performance departs from the printed score in a number of places. Find every departure and report it.

These are never departures: tempo, rubato, held fermatas, dynamics, articulation, sustain pedal, the spread of rolled (arpeggiated) chords, and the conventional realization of printed ornaments: trills, mordents and turns are played as fast alternations with the neighbouring note, tremolos as rapid repeated notes or alternations, grace notes just before the beat.

A departure is exactly one of these types:

type meaning pitch
wrong_note one printed note is played at a different pitch, not an octave away the pitch played
octave_error one printed note is played exactly one octave higher or lower the pitch played
omitted_note one printed note is not played at all the printed pitch that was left out
extra_note one note that is not in the score is played together with printed notes of that piano, replacing nothing the pitch played

Each departure concerns a single note of a single piano, and is judged against that piano's own part only: a note missing from, added to, or changed in Piano I's part is a departure of Piano I even if Piano II plays that same pitch at the same moment, and vice versa.

For each departure report:

  • piece: "A", "B", "C" or "D".
  • bar: the bar number in that piece's printed score. Bar numbers are printed at the start of each system; count bars from there. An incomplete bar before the first full bar of a piece is bar 0.
  • beat: where in the bar the affected note is printed, in quarter notes counted from 1 at the barline: beat = 1 + (quarter notes from the start of the bar to the note). In 4/4 the third quarter is 3, the eighth note after the downbeat is 1.5; in 6/8 the fourth eighth note is 2.5; the second note of a quarter-note triplet starting on the downbeat is 1.6667. For extra_note, use the printed position of the notes it was played together with. Accepted within ±0.125.
  • piano: "I" or "II", the piano that departed from its part.
  • pitch: sounding pitch in scientific pitch notation with middle C = C4, written as a letter, an optional # or b, and an octave number (for example "F#3", "Bb5", "C4"). Enharmonic spellings are equivalent. Printed 8va/8vb lines change the sounding octave.

Output

Write output/deviations.json:

{
  "deviations": [
    {"piece": "B", "bar": 12, "beat": 2.5, "piano": "II", "type": "wrong_note", "pitch": "G#3"},
    {"piece": "D", "bar": 40, "beat": 1, "piano": "I", "type": "extra_note", "pitch": "E5"}
  ]
}

One entry per departure, in any order. Entries that do not match a real departure count against you. Write the file early and overwrite it as you confirm more departures; whatever the file contains when the session ends is what counts.

Rules

  • Work only inside this working directory. There is no internet access.
  • Do not rely on memory of any piece or edition. The printed scores provided here are the reference, even where they differ from other editions.
  • Use only the tools already installed (for example ffmpeg/ffprobe, sox, pdftoppm, and Python 3 with numpy, scipy, pandas, pillow, opencv, soundfile, matplotlib). pip install will not work.

Media

Frame from recital.mp4
recital.mp4 · 381 s
Frame from recital.mp4
recital.mp4 · 761 s
Frame from recital.mp4
recital.mp4 · 1142 s

Frames above are from the task’s input videos. Media files are distributed separately from the repository; this task’s hash manifest lists 5 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY 3.0, CC BY 4.0, CC0 1.0, MIT, Public Domain.

Score

AgentScore (0 to 1)
GPT-6-Astra0.011
Claude Opus 5.50.010
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a 25-minute film of two player-less pianos performing four pieces, reads the printed scores, and lists every note where the performance departs from the score: piece, bar, beat, piano, type of departure and pitch. The reference answer holds 186 departures.

What a good answer has to get right, and how the verifier checks it

Every departure is found, and none is invented
The score is an F1 over departures with one-to-one matching. A missed departure lowers recall. Every entry in the file counts as a claim, including one the verifier cannot read, so an entry that matches nothing lowers precision.
The note is located in the printed score
Piece and bar must be exact, and the beat must be within 0.125 of a quarter note.
The right piano is named
The piano, I or II, must be the one that departed from its own part. Both pianos are mixed into one mono track with the same sound, so this has to be read from the picture.
The kind of departure is right
The type must be exactly one of four: wrong note, octave error, omitted note or extra note.
The pitch is right
Pitches are compared as MIDI note numbers, so C#4 and Db4 are the same answer. For an omitted note it is the printed pitch; otherwise the pitch that was played.

How the checks become one score

Score = F1 = 2 × matched / (entries listed + 186). Every departure counts the same, and an entry counts only when all six fields are right.

What scores zero

No deviations.json, a file that is not valid JSON or whose “deviations” is not a list, an empty list, or a list in which no entry matches a departure on all six fields.

Scores from this verifier: the reference answer with one thing changed

SubmissionScorePrecisionRecall
Careful work
The reference answer1.0001.0001.000
Every beat 0.1 offEach beat is reported 0.1 of a quarter note late or early, alternately.1.0001.0001.000
One kind of mistake
Omitted notes never noticedThe 14 departures of type omitted_note are left out. Everything else is correct.0.9611.0000.925
Octave errors called wrong notesThe 33 octave errors are reported as wrong notes. Everything else is correct.0.8230.8230.823
Only the first two piecesDepartures in pieces A and B only.0.6021.0000.430
Bars counted from 1 instead of 0Every bar number is one too high, as when an incomplete first bar is counted as bar 1.0.0270.0270.027
Right notes, wrong pianoEvery departure is attributed to the other piano. Everything else is correct.0.0000.0000.000
No real work
Empty list0.0000.0000.000
No file0.0000.0000.000
An agent’s answer
A sampled runGPT-6-Astra · Codex 0.154.0. The run, step by step0.0110.3330.005

Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: tempo, dynamics, articulation, pedalling and the usual way of playing printed ornaments, which the task says are never departures. The verifier also reports how many entries would match if the piano, the pitch, the type or the beat were ignored; those figures are diagnostics and do not enter the score.

The score line and constants in the verifier code

Tolerances

NameValueWhere it is set
BEAT_TOL0.125constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.01
Wall time
46.5 min
Steps shown
154

Loading the run…