AgenticVBench-Omni

AgenticVBench-Omni / Asset reconstruction

Synthesizer

audio-production-session-reimplementation

The working directory holds one file, session_recording.mp4: a screen recording (1920×1080, 30 fps, about 153 minutes, with its original sound) of a musician producing a short electronic track in the Ardour DAW. Every instrument in the track is a separate instance of the Surge synthesizer (version 1.8), and the musician designs each sound from the initial patch while talking about it. The sound in the recording is the computer's audio output plus the musician's voice. The small white boxes that Surge pops up with a control's exact numeric value while it is being adjusted have been painted over; the controls themselves, their positions and everything else on screen are unchanged. No project files are provided; everything has to come from the recording.

Family
Asset reconstruction
Inputs
videoaudio
Media
1 video, 2 h 34 min, with sound
Reference answer
9 synth patches
Deliverable
/workspace/output
Scoring
per-componentmean of componentschecklist / accuracygeometric (IoU / distance)

Task package · verifier code

Instruction

Full instruction given to the agent

Rebuild the Synth Patches From a Recorded Production Session

The working directory holds one file, session_recording.mp4: a screen recording (1920×1080, 30 fps, about 153 minutes, with its original sound) of a musician producing a short electronic track in the Ardour DAW. Every instrument in the track is a separate instance of the Surge synthesizer (version 1.8), and the musician designs each sound from the initial patch while talking about it. The sound in the recording is the computer's audio output plus the musician's voice. The small white boxes that Surge pops up with a control's exact numeric value while it is being adjusted have been painted over; the controls themselves, their positions and everything else on screen are unchanged. No project files are provided; everything has to come from the recording.

Task: rebuild the Surge patch of each of the nine instrument tracks below, as it stands at the end of the recording, so that it sounds the same as the musician's.

Ardour track name file to write
Meow output/patches/meow.fxp
Kick output/patches/kick.fxp
Bass output/patches/bass.fxp
Snare output/patches/snare.fxp
Supersauce output/patches/supersauce.fxp
Growl Bass output/patches/growl_bass.fxp
Clav output/patches/clav.fxp
Chromatic Mess output/patches/chromatic_mess.fxp
Clav 2 output/patches/clav_2.fxp

Only the Surge patch itself counts: oscillators, mixer, filters, waveshaper, amp, envelopes, LFOs (including step-sequencer and MSEG shapes), modulation routings, macros, scene mode and Surge's own effect slots. Ardour's mixer, its other plugins, the MIDI notes and the automation lanes are not part of the answer.

Surge in this environment

Surge 1.8.0 is installed as the headless Python module surgepy (use help(surgepy)). It is the same engine your patches are judged with. Useful calls:

  • s = surgepy.createSurge(48000) starts from the initial patch (the same state a new Surge instance had in the recording); s.loadPatch(path), s.savePatch(path) read and write .fxp.
  • s.getParamTable() lists every parameter as (id, storage_name, full_name, valtype, ctrltype, min, max, value); s.getControlGroup(...), s.getParamVal/Min/Max/Display, s.setParamVal, s.getModSource, s.setModulation, s.getAllModRoutings edit values and routings.
  • s.playNote(channel, note, velocity, 0), s.releaseNote(channel, note, velocity), s.setMacro(i, value01) (macros 0–7), s.createMultiBlock(n) and s.processMultiBlock(buffer, start, n) render audio (32 samples per block, stereo). s.seedRNG(n) makes random phases and noise repeat.
  • A .fxp file is a 60-byte header, a 32-byte block starting with sub3 whose next 4 bytes (little-endian) are the XML length, the patch XML, then wavetable data. Things the API does not reach (for example MSEG and step-sequencer shapes) can be edited in that XML; keep the length field correct.
  • Surge's factory data (wavetables, factory patches) is in /usr/share/surge.

How the patches are compared

Each submitted patch is compared with the musician's patch for the same track, both by how it sounds and by how it is built; a track only scores well if both match.

Sound. Both patches are loaded into the same Surge 1.8.0 build, reset to the same random seed, and rendered at 48 kHz: (a) playing the MIDI notes that track plays in the finished arrangement (140 BPM), with the arrangement's automation of the patch macros applied exactly as recorded; and (b) single notes MIDI 36, 48, 60 and 72 at velocities 50 and 110, each held 0.5 s, plus the chord 48‑55‑60‑64 at velocity 90 held 1 s. Each pair of renders is compared with a multi-resolution log-mel spectral distance (both channels, smoothed over 50 ms, 60 dB below the loudest cell as floor); level, tone, pitch, timing, stereo image and effects all change it. Differences at the size that random oscillator phases and noise cause on their own, or that reading every control's position off the recording (to about 1.5 pixels) would cause, are not held against you.

Parameters. The patch is split into modules: global settings (volume, scene mode, active scene or split point, FX bypass, character); for each scene in use each of oscillators 1–3, the mixer (levels, mutes, solos, routing, noise colour), voice settings (octave, pitch, portamento, play mode, drift, keytrack root), FM, scene output (volume, pan, width, sends), amp (level, velocity sensitivity, low cut), filter configuration (routing, feedback, balance), waveshaper, filter 1, filter 2, amp EG, filter EG, and each of the 12 LFOs with its shape data; each of the 8 effect slots (type and parameters, plus the return level for the two send slots); each macro; and each modulation routing (target, source) with its depth. A module counts as correct only if every value in it matches: continuous values within 2.5% of the parameter's range (modulation depth within 2.5% of the target's range), everything discrete (types, subtypes, modes, octaves, switches, temposync and other flags, step and MSEG node counts) exactly, and step values and MSEG node positions within 0.02. Modules that cannot be heard are ignored: an oscillator whose mixer channel is muted or at zero and that feeds no FM or ring modulation, a filter or waveshaper set to Off, a filter EG that modulates nothing, an LFO that modulates nothing, a scene that does not play, an effect slot that is empty or bypassed, a macro that modulates nothing. Modules left at the initial patch's values in both patches are also ignored. Extra routings or modules that differ from the musician's count against you.

A missing file, or a file that is not a Surge patch, scores zero for that track.

Rules

  • Work only inside this working directory; write results under output/. There is no internet access. Do not rely on memory of any recording, musician or preset; everything must come from session_recording.mp4.
  • Tools available are what is installed: ffmpeg/ffprobe, sox, and Python 3 with surgepy, numpy, scipy, pandas, pillow, opencv-python-headless, soundfile and matplotlib. pip install will not work.
  • Write each patch file early and update it as you go; whatever is in output/patches/ when the session ends is what counts.

Media

Media files are distributed separately from the repository; this task’s hash manifest lists 1 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY, CC0 1.0, GPL-3.0-or-later.

Score

AgentScore (0 to 1)
GPT-6-Astra0.175
Claude Opus 5.50.745
Claude Opus 50.007
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a 153-minute screen recording, with sound, of a musician producing a short electronic track in which every instrument is a Surge synthesizer patch designed from the initial patch. The pop-up boxes that show exact values have been painted over. The agent delivers the nine synth patches, one per track.

What a good answer has to get right, and how the verifier checks it

Each patch is built the same way
Structure score: of the modules that make sound in either patch (oscillators, filters, envelopes, LFOs, effects, modulation routings), the share whose every value matches the expert’s within tolerance. A module left untouched at its initial setting in both patches is not counted.
Each patch sounds the same
Sound score: both patches are rendered on the same notes and compared by spectral distance in decibels. The distance a careful person would leave after reading every control off the recording is allowed for free, per track; beyond it, credit falls in a straight line and reaches 0 at 3 dB more.
Both are required
A track’s score is the structure score multiplied by the sound score. A patch that imitates the sound with a different construction, or copies the construction and sounds different, does not score well.
Every track is delivered
A missing or invalid patch scores 0 for its track and still counts in the mean.

How the checks become one score

Each track’s score = structure × sound. Score = the mean over the nine tracks, each weighing the same.

What scores zero

No patch files, or patches none of which can be loaded. A track also scores 0 when no module of its patch is right, or when it sounds 3 dB or more beyond its allowance on every test note.

Outside the score: settings of the synth’s editor that do not change the sound, and the arrangement of the track in Ardour.

The score line and constants in the verifier code

Reward formula in the verifier code

Tolerances

NameValueWhere it is set
FLOAT_TOL0.025constant in the verifier code
SEQ_TOL0.02constant in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
Claude Opus 5.5 · Claude Code 2.1.280
Score
0.74
Wall time
57.7 min
Steps shown
778

Loading the run…