session.mp4 (1920x1080, with narration) is one continuous screen recording of an audio engineer mixing a short track in Ardour, a digital audio workstation. It starts from a brand-new, empty session and ends with the finished mix; the first part of the recording shows the raw stems being imported and named. stems/ contains those same raw audio files (same neutral filenames the recording imports from) so you can build alongside it. Start your own session the normal way (a new Ardour session in your working directory) and import them the same way.
Full instruction given to the agent
Rebuild a mix from a screen recording
session.mp4 (1920x1080, with narration) is one continuous screen recording of an audio engineer
mixing a short track in Ardour, a digital audio workstation. It starts from a brand-new, empty
session and ends with the finished mix; the first part of the recording shows the raw stems being
imported and named. stems/ contains those same raw audio files (same neutral filenames the
recording imports from) so you can build alongside it. Start your own session the normal way (a
new Ardour session in your working directory) and import them the same way.
Rebuild that mix: produce the Ardour session it had at the end of the recording.
Deliverable
output/session.ardour, a valid Ardour XML session file (any Ardour version's native format is
fine; only this one file is read, not a whole session folder, and it is never opened in the
application — parsed directly as XML). Give each track and bus the name it is shown or called by
in the recording. Only the file at exactly this path is scored.
What is not scored
- Track/bus count and existence by itself, and which of the closed set of instrument plugins
(if any) generates a track's sound before your effects — these are all obvious from watching a
few seconds of the recording and are not worth points on their own. Points are for the mixing
decisions layered on top: sends, busses, EQ, dynamics, time-based effects, gain and pan, and
arrangement edits.
- Anything discussed but not actually done, or done and then undone before the recording ends —
only the final state matters.
- A track's fader/pan left exactly where an untouched track starts (unity gain, centered pan) is
never scored as a "correct" answer on its own — only tracks whose level was visibly moved are
checked, and getting those specific tracks right is the point.
What must match
Judged only in final, end-of-recording state; several settings are visibly set once and then
revisited/changed again later — only where they end up counts, but you need to keep track of which
track/parameter is being changed the second time.
-
Fader levels. For the handful of tracks whose gain is visibly moved off unity, the final
gain in dB, within about 0.6 dB.
-
Processor chain. For each track/bus that gets any plugin at all, the plugins present and
their order, front to back, by exact plugin name as shown in the plugin browser/insert
list (e.g. "LSP Parametric Equalizer x16 Stereo", "ACE Compressor (stereo)", "Dragonfly Hall
Reverb", "LSP Multiband Compressor Stereo x8", "x42-Autotune", "LSP Compressor Stereo"). Only
plugins actually available to insert in your own Ardour are ever asked for.
-
EQ bands. For every band of a parametric EQ that is visibly moved off flat (shown as a
node dragged on the graphical curve, not a flat line sitting at its default frequency/shape),
that band's filter type (closed vocabulary: whatever the EQ's own type selector offers, e.g.
bell/shelf/highpass/lowpass), frequency (within about 1/8 octave), gain (within about 0.9 dB),
and Q where the recording shows it changed from the default (within 30% relative). A band never
shown moving off its default is not scored, whether or not you touch it.
-
Compressors (single-band): ratio (within 2x for the very high, drum-style ratios; treat
"ratio" as sound-to-eye from the graphical curve/knob, not a printed number), attack and
release time (order-of-magnitude accuracy — within 50% relative for attack, 20 ms for
release), threshold and makeup gain in dB (within 1.5-3 dB, matching how precisely the
recording shows the knob/curve position).
-
Multiband compressor. Which of its available bands is actually engaged (shown as that
band's curve moving), each engaged band's ratio (within 50% relative) and, where the recording
shows attack visibly set away from the plugin's own default, attack time (within 60%
relative); the overall output/makeup trim in dB (within 1.5 dB); and, where the whole unit's
knee is visibly hardened away from default on every band, that (within 30% relative).
-
Reverb. Size, decay time, dry level, wet ("late") level, width and pre-delay, each read
from the reverb's own knobs, within roughly 15-25% relative (decay within 0.5 s, pre-delay
within 3 ms) — these are graphical knob positions, not numbers.
-
Pitch-correction. Whether correction is actually engaged (as opposed to the plugin just
sitting in the chain doing nothing) on each track that has it; where engaged, its mode and
which note(s), if any, it is locked to (closed vocabulary: the 12 chromatic note names).
-
Routing. Any signal path shown being built that is not simply "track straight to master" —
a send from one track feeding into another track or bus's input (including a sidechain input
of a plugin on that other track), and any track's output being routed to feed a different
track's input rather than the master bus. Identify the source and destination correctly.
-
Arrangement (region edits). For every track built from more than one placed audio clip (a
repeated hit, a chopped-up performance, or comped alternate takes stacked on the same track),
the position of each clip within about 0.2 s, and its length (within 15-25% relative,
floor 0.15-0.3 s) — including short clips under two seconds and clips that overlap because an
earlier take was kept underneath a later one. Get the number of clips/takes right, not just
their rough shape: a session that places one long clip where the recording shows eight short
ones, or that skips a stacked-but-unheard earlier take, is missing real information the
recording contains.
Environment
- Work only inside your current working directory. There is no internet access. Do not use any
memory you may have of this song or its source — build only what the recording and stems show.
- Ardour is installed and runs headless-capable in this sandbox (a virtual display is available if
you want the GUI;
ardour --help and the Lua scripting console are both usable, as is the
built-in "Dummy" audio backend if you don't have real audio hardware here). Every plugin named
above is installed and available to insert.
ffmpeg/ffprobe and sox are available for inspecting/extracting audio from session.mp4
and the stems; faster-whisper (model small.en) is available for a transcript of the narration.
Python has numpy, scipy, soundfile, opencv-python-headless, pillow.
- The narration only ever states what is being changed and why (which track, what kind of
move) — it never states a dB value, a Hz value, a ratio, a time or a knob's numeric setting.
Every exact amount has to come from watching the picture (fader/knob/curve position on screen),
not the words.
Media files are distributed separately from the repository; this task’s hash manifest lists 23 files (environment/media/SHA256SUMS.txt). Licences named for this task: see the provenance notes in the task package.
One run per agent.
The agent watches a narrated screen recording of an engineer mixing a short track in Ardour, has the raw stems the recording imports, and delivers an Ardour session file that reproduces the mix. The verifier reads the session file as XML, without opening Ardour, and checks 134 items against the expert’s session.
What a good answer has to get right, and how the verifier checks it
- Tracks and plugins are the ones in the recording
- Tracks and plugins are found by the names shown on screen. An item that depends on a track or plugin missing from the session counts as wrong.
- Every clip is placed
- 95 items cover the audio regions: each clip’s position must be within 0.2 s and its length within that clip’s own tolerance. A submitted clip can satisfy only one reference clip, so one clip cannot stand in for several chops that lie close together.
- Where the recording shows the clips
- The clips are put down one at a time. The Vox takes are placed from 9:13 with the timeline in view. The clips of the other tracks are placed from 11:01 and from 13:54, while the main track area is scrolled to a later, empty part of the timeline. Those clips appear only in the overview strip along the bottom of the window, where a 3.3 s clip is 20 to 27 pixels wide and one pixel is 0.12 to 0.16 s. The position tolerance is 0.2 s.
- The faders are set
- 5 items: the final gain of each track that the recording moves off its default must be within 0.6 dB.
- Equalizer and compressors are dialled in
- 30 items: equalizer bands, single-band compressors and the multiband compressor. Each parameter has its own tolerance, absolute, relative or in octaves depending on the parameter.
- Plugin order and routing are right
- 4 items: the order of plugins on two tracks, and two routing items.
- What the recording does not show is not scored
- 52 further items were taken out of the scored set. During recording, the plugins on four tracks dropped out of the on-screen mixer partway through, so their settings cannot be read from the video.
How the checks become one score
Score = items right / 134, counted over all items together and not averaged per group. The 95 clip items therefore carry about 71% of the score, and the 39 mixer items about 29%.
What scores zero
No output/session.ardour, or a file that cannot be parsed as XML. A session that parses but is incomplete is scored item by item.
Scores from this verifier: the reference answer with one thing changed
Every row is a file scored by this task’s verifier in the same way as an agent’s output. Outside the score: the 52 items named above.
Verifier code · script that builds and scores these files
Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.