AgenticVBench-Omni

AgenticVBench-Omni / Asset reconstruction

Ardour

ardour-mixing-session-reimplementation

session.mp4 (1920x1080, with narration) is one continuous screen recording of an audio engineer mixing a short track in Ardour, a digital audio workstation. It starts from a brand-new, empty session and ends with the finished mix; the first part of the recording shows the raw stems being imported and named. stems/ contains those same raw audio files (same neutral filenames the recording imports from) so you can build alongside it. Start your own session the normal way (a new Ardour session in your working directory) and import them the same way.

Family
Asset reconstruction
Inputs
videoaudio
Media
1 video, 15 min, with sound; 21 audio files, 6 min
Reference answer
134 scored items
Deliverable
/workspace/output
Scoring
per-componentchecklist / accuracy

Task package · verifier code

Instruction

Full instruction given to the agent

Rebuild a mix from a screen recording

session.mp4 (1920x1080, with narration) is one continuous screen recording of an audio engineer mixing a short track in Ardour, a digital audio workstation. It starts from a brand-new, empty session and ends with the finished mix; the first part of the recording shows the raw stems being imported and named. stems/ contains those same raw audio files (same neutral filenames the recording imports from) so you can build alongside it. Start your own session the normal way (a new Ardour session in your working directory) and import them the same way.

Rebuild that mix: produce the Ardour session it had at the end of the recording.

Deliverable

output/session.ardour, a valid Ardour XML session file (any Ardour version's native format is fine; only this one file is read, not a whole session folder, and it is never opened in the application — parsed directly as XML). Give each track and bus the name it is shown or called by in the recording. Only the file at exactly this path is scored.

What is not scored

  • Track/bus count and existence by itself, and which of the closed set of instrument plugins (if any) generates a track's sound before your effects — these are all obvious from watching a few seconds of the recording and are not worth points on their own. Points are for the mixing decisions layered on top: sends, busses, EQ, dynamics, time-based effects, gain and pan, and arrangement edits.
  • Anything discussed but not actually done, or done and then undone before the recording ends — only the final state matters.
  • A track's fader/pan left exactly where an untouched track starts (unity gain, centered pan) is never scored as a "correct" answer on its own — only tracks whose level was visibly moved are checked, and getting those specific tracks right is the point.

What must match

Judged only in final, end-of-recording state; several settings are visibly set once and then revisited/changed again later — only where they end up counts, but you need to keep track of which track/parameter is being changed the second time.

  1. Fader levels. For the handful of tracks whose gain is visibly moved off unity, the final gain in dB, within about 0.6 dB.

  2. Processor chain. For each track/bus that gets any plugin at all, the plugins present and their order, front to back, by exact plugin name as shown in the plugin browser/insert list (e.g. "LSP Parametric Equalizer x16 Stereo", "ACE Compressor (stereo)", "Dragonfly Hall Reverb", "LSP Multiband Compressor Stereo x8", "x42-Autotune", "LSP Compressor Stereo"). Only plugins actually available to insert in your own Ardour are ever asked for.

  3. EQ bands. For every band of a parametric EQ that is visibly moved off flat (shown as a node dragged on the graphical curve, not a flat line sitting at its default frequency/shape), that band's filter type (closed vocabulary: whatever the EQ's own type selector offers, e.g. bell/shelf/highpass/lowpass), frequency (within about 1/8 octave), gain (within about 0.9 dB), and Q where the recording shows it changed from the default (within 30% relative). A band never shown moving off its default is not scored, whether or not you touch it.

  4. Compressors (single-band): ratio (within 2x for the very high, drum-style ratios; treat "ratio" as sound-to-eye from the graphical curve/knob, not a printed number), attack and release time (order-of-magnitude accuracy — within 50% relative for attack, 20 ms for release), threshold and makeup gain in dB (within 1.5-3 dB, matching how precisely the recording shows the knob/curve position).

  5. Multiband compressor. Which of its available bands is actually engaged (shown as that band's curve moving), each engaged band's ratio (within 50% relative) and, where the recording shows attack visibly set away from the plugin's own default, attack time (within 60% relative); the overall output/makeup trim in dB (within 1.5 dB); and, where the whole unit's knee is visibly hardened away from default on every band, that (within 30% relative).

  6. Reverb. Size, decay time, dry level, wet ("late") level, width and pre-delay, each read from the reverb's own knobs, within roughly 15-25% relative (decay within 0.5 s, pre-delay within 3 ms) — these are graphical knob positions, not numbers.

  7. Pitch-correction. Whether correction is actually engaged (as opposed to the plugin just sitting in the chain doing nothing) on each track that has it; where engaged, its mode and which note(s), if any, it is locked to (closed vocabulary: the 12 chromatic note names).

  8. Routing. Any signal path shown being built that is not simply "track straight to master" — a send from one track feeding into another track or bus's input (including a sidechain input of a plugin on that other track), and any track's output being routed to feed a different track's input rather than the master bus. Identify the source and destination correctly.

  9. Arrangement (region edits). For every track built from more than one placed audio clip (a repeated hit, a chopped-up performance, or comped alternate takes stacked on the same track), the position of each clip within about 0.2 s, and its length (within 15-25% relative, floor 0.15-0.3 s) — including short clips under two seconds and clips that overlap because an earlier take was kept underneath a later one. Get the number of clips/takes right, not just their rough shape: a session that places one long clip where the recording shows eight short ones, or that skips a stacked-but-unheard earlier take, is missing real information the recording contains.

Environment

  • Work only inside your current working directory. There is no internet access. Do not use any memory you may have of this song or its source — build only what the recording and stems show.
  • Ardour is installed and runs headless-capable in this sandbox (a virtual display is available if you want the GUI; ardour --help and the Lua scripting console are both usable, as is the built-in "Dummy" audio backend if you don't have real audio hardware here). Every plugin named above is installed and available to insert.
  • ffmpeg/ffprobe and sox are available for inspecting/extracting audio from session.mp4 and the stems; faster-whisper (model small.en) is available for a transcript of the narration. Python has numpy, scipy, soundfile, opencv-python-headless, pillow.
  • The narration only ever states what is being changed and why (which track, what kind of move) — it never states a dB value, a Hz value, a ratio, a time or a knob's numeric setting. Every exact amount has to come from watching the picture (fader/knob/curve position on screen), not the words.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 23 files (environment/media/SHA256SUMS.txt). Licences named for this task: see the provenance notes in the task package.

Score

AgentScore (0 to 1)
GPT-6-Astra0.709
Claude Opus 5.50.754
Claude Opus 50.410
Gemini 3.8 Flash0.112
DeepSeek 4.1 Flash0.396

One run per agent.

The agent watches a narrated screen recording of an engineer mixing a short track in Ardour, has the raw stems the recording imports, and delivers an Ardour session file that reproduces the mix. The verifier reads the session file as XML, without opening Ardour, and checks 134 items against the expert’s session.

What a good answer has to get right, and how the verifier checks it

Tracks and plugins are the ones in the recording
Tracks and plugins are found by the names shown on screen. An item that depends on a track or plugin missing from the session counts as wrong.
Every clip is placed
95 items cover the audio regions: each clip’s position must be within 0.2 s and its length within that clip’s own tolerance. A submitted clip can satisfy only one reference clip, so one clip cannot stand in for several chops that lie close together.
Where the recording shows the clips
The clips are put down one at a time. The Vox takes are placed from 9:13 with the timeline in view. The clips of the other tracks are placed from 11:01 and from 13:54, while the main track area is scrolled to a later, empty part of the timeline. Those clips appear only in the overview strip along the bottom of the window, where a 3.3 s clip is 20 to 27 pixels wide and one pixel is 0.12 to 0.16 s. The position tolerance is 0.2 s.
The faders are set
5 items: the final gain of each track that the recording moves off its default must be within 0.6 dB.
Equalizer and compressors are dialled in
30 items: equalizer bands, single-band compressors and the multiband compressor. Each parameter has its own tolerance, absolute, relative or in octaves depending on the parameter.
Plugin order and routing are right
4 items: the order of plugins on two tracks, and two routing items.
What the recording does not show is not scored
52 further items were taken out of the scored set. During recording, the plugins on four tracks dropped out of the on-screen mixer partway through, so their settings cannot be read from the video.

How the checks become one score

Score = items right / 134, counted over all items together and not averaged per group. The 95 clip items therefore carry about 71% of the score, and the 39 mixer items about 29%.

What scores zero

No output/session.ardour, or a file that cannot be parsed as XML. A session that parses but is incomplete is scored item by item.

Scores from this verifier: the reference answer with one thing changed

SubmissionScoreClips, of 95Mixer, of 39
Careful work
The reference sessionA session file written from the ground-truth values.1.0009539
Every clip 0.05 s lateEach clip’s position on the timeline is 0.05 s late.1.0009539
One kind of mistake
Every clip 0.15 s lateEach clip’s position is 0.15 s late, inside the 0.2 s tolerance. Chops that lie closer together than that can then pair with the wrong neighbour.0.9709139
Every clip 0.5 s lateEach clip’s position is 0.5 s late. The mixer is correct.0.6424739
Clips placed, no pluginsAll equalizers, compressors and other effect plugins are left off. Clips, faders and sends are correct.0.761957
Faders left at their defaultEvery track’s gain is at 0 dB. Everything else is correct.0.9639534
Mixer set up, no clipsThe tracks, plugins, faders and sends are correct; no clip is on the timeline.0.291039
No real work
An empty sessionA session file with no tracks.0.00000
No file0.00000
An agent’s answer
A sampled runClaude Opus 5.5 · Claude Code 2.1.280. The run, step by step0.7547427

Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: the 52 items named above.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5.5 · Claude Code 2.1.280
Score
0.75
Wall time
104 min
Steps shown
315

Loading the run…