AgenticVBench-Omni

AgenticVBench-Omni / Music and synchronization

Symphony

symphony-wrong-note-ledger

Your working directory contains:

Family
Music and synchronization
Inputs
audiodocuments
Media
9 audio files, 2 h 2 min
Reference answer
399 departures
Deliverable
/workspace/output
Scoring
F1 / precision–recallone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

Orchestral Performance: Departures from the Full Score

Your working directory contains:

  • audio/movement_1.m4a … audio/movement_9.m4a: a recording of an orchestra playing nine symphonic movements, one file per movement (about two hours in total, stereo, 48 kHz). Instruments sit in fixed seats across the stereo image for the whole recording, and the hall adds reverberation.
  • scores/movement_1.pdf … scores/movement_9.pdf: the printed full score of each movement. Every system names its staves at the left (for example Flute 1, A Clarinet 2, F Horn 3, Violoncello), and every bar is numbered above the top staff.

Task

The players sometimes depart from the printed score. Find every departure and report it.

These are never departures: tempo (the conductor's tempo, rubato, ritardandi, pauses and held fermatas), small timing differences between players (well under a quarter note), dynamics and balance (a note or passage played louder or softer than marked), articulation (notes played shorter or longer, staccato or legato, than marked), and the realisation of printed ornaments: trills starting on the upper or main note at any speed, tremolos, and grace notes played just before or on the beat.

A departure is exactly one of these types. A passage is a staff's run of notes that starts after at least two quarter notes of rest in that staff and continues until the staff's next rest of at least one quarter note.

type meaning where (bar, beat) pitch
wrong_pitch one printed note is played one or two semitones away from its printed pitch the printed note the pitch played
wrong_octave one printed note is played exactly one octave higher or lower the printed note the pitch played
missed_entrance a whole passage is not played at all its first printed note the printed pitch of that note
early_entrance a whole passage is played with every note displaced earlier by the same amount, at least one quarter note its first printed note the printed pitch of that note
late_entrance a whole passage is played with every note displaced later by the same amount, at least one quarter note its first printed note the printed pitch of that note
extra_note a note that is not in the score is played by an instrument while its staff is resting where the note starts the pitch played
wrong_transposition a clarinet, horn, trumpet or English horn plays a whole passage of at least four notes with every note shifted by the same one or two semitones, as if reading the part for an instrument in another key its first printed note the pitch played for that first note

A displaced passage is one early_entrance or late_entrance, never a missed entrance plus extra notes, and a wrong_transposition is one entry for the whole passage; for it also give end_bar, the bar of the passage's last note. Every departure concerns one staff and is judged against that staff's own printed part, even where another instrument plays the same music.

For each departure report:

  • movement: 1 to 9, the number in the file names.
  • bar: the printed bar number.
  • beat: position in the bar in quarter notes counted from 1 at the barline, beat = 1 + (quarter notes from the start of the bar). In 2/2 the second half note is 3; in 6/8 the fourth eighth note is 2.5; in 3/8 the third eighth note is 2. For printed notes use the printed position (accepted within ±0.25); for an extra_note use where it starts, on the same scale (accepted within ±0.5).
  • instrument: the staff name exactly as printed at the left of the staff.
  • type: one of the seven types above.
  • pitch: sounding pitch in scientific pitch notation, middle C = C4: a letter, an optional # or b, and an octave number ("F#3", "Bb5"). Enharmonic spellings are equivalent. Printed 8va/8vb lines change the octave. Transposing staves are printed at written pitch and sound as follows: A Clarinet a minor third lower; Bb Clarinet a major second lower; A Bass Clarinet a major tenth lower; F Horn and English Horn a perfect fifth lower; E Horn a minor sixth lower; F Trumpet a perfect fourth higher; Contrabass an octave lower. All other staves sound as written.
  • end_bar: only for wrong_transposition.

Output

Write output/departures.json:

{
  "departures": [
    {"movement": 2, "bar": 57, "beat": 3, "instrument": "Oboe 1", "type": "wrong_pitch", "pitch": "D#5"},
    {"movement": 8, "bar": 311, "beat": 1, "instrument": "F Horn 3", "type": "wrong_transposition", "pitch": "G3", "end_bar": 318}
  ]
}

One entry per departure, in any order. An entry counts only if every field matches a real departure; entries that match nothing count against you. Write the file early and overwrite it as you confirm more departures; whatever the file contains when the session ends is what counts.

Rules

  • Work only inside this working directory. There is no internet access.
  • Do not rely on memory of any piece, edition or recording. The printed scores provided here are the reference, even where they differ from other editions.
  • Use only the tools already installed (for example ffmpeg/ffprobe, sox, pdftoppm, and Python 3 with numpy, scipy, pandas, pillow, opencv, soundfile, matplotlib). pip install will not work.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 18 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY-SA, CC0 1.0, MIT.

Score

AgentScore (0 to 1)
GPT-6-Astra0.076
Claude Opus 5.50.000
Claude Opus 50.000
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent listens to a two-hour recording of an orchestra playing nine symphonic movements, reads the printed full scores, and lists every departure from the score: movement, bar, beat, instrument, type of departure and pitch. The reference answer holds 399 departures of seven types.

What a good answer has to get right, and how the verifier checks it

Every departure is found, and none is invented
The score is an F1 over departures with one-to-one matching. A missed departure lowers recall; an entry that matches nothing, or that the verifier cannot read, lowers precision.
The note is located in the printed score
Movement and bar must be exact. The beat must be within 0.25 of a quarter note, or 0.5 for an extra note, whose position has to be read from the recording because no printed note marks it.
The instrument is named
The instrument must be the staff name printed in the score, compared without regard to capitals or spacing. Second flute and first flute are different answers.
The kind of departure is right
The type must be exactly one of seven: wrong pitch, wrong octave, early, late or missed entrance, extra note, or wrong transposition. A wrong transposition must also give the bar where the passage ends.
The pitch is right
Pitches are compared as MIDI note numbers, so enharmonic spellings are the same answer.
What cannot be confirmed from the recording is not held against the answer
A further 160 departures were performed but could not be recovered when the recording was audited without the answer key. They are excused: each one cancels at most one otherwise unmatched entry of the same movement and type within one bar. Reporting one neither helps nor costs.

How the checks become one score

Score = F1 = 2 × matched / (entries counted + 399), where entries counted is every entry in the file less those cancelled by an excused departure. Every departure counts the same, and an entry counts only when every field is right.

What scores zero

No departures.json, a file that is not valid JSON or whose “departures” is not a list, an empty list, or a list in which no entry matches a departure on every field.

Outside the score: the 160 excused departures; tempo, small timing differences between players, dynamics, articulation and the playing of printed ornaments, which the task says are never departures. The verifier also reports how many entries are right on location alone, or on location and one other field; those figures are diagnostics.

The score line and constants in the verifier code

Tolerances

NameValueWhere it is set
BEAT_TOL0.25constant in the verifier code
EXTRA_BEAT_TOL0.5constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.08
Wall time
41.4 min
Steps shown
180

Loading the run…