AgenticVBench-Omni

AgenticVBench-Omni / Music and synchronization

A/V Sync

av-sync-drift-ledger

reel.mp4 in your working directory is about 261 minutes of boxing gym training sessions, 1280x720 at 29.97 frames per second, with one mono soundtrack. It is made of a few long continuous recordings joined end to end. Within a recording there are jump cuts, where picture and sound skip ahead together, and brief audio glitches, where under half a second of sound is replaced by nearby sound. The soundtrack also carries timing jitter of a few milliseconds throughout, and its level changes abruptly by a few decibels every so often. Jump cuts, glitches, jitter and level changes are not faults.

Family
Music and synchronization
Inputs
videoaudiodocuments
Media
1 video, 4 h 21 min, with sound
Reference answer
35 faulty stretches
Deliverable
/workspace/output
Scoring
F1 / precision–recallone-to-one matching

Task package · verifier code

Instruction

Full instruction given to the agent

Sync fault ledger

reel.mp4 in your working directory is about 261 minutes of boxing gym training sessions, 1280x720 at 29.97 frames per second, with one mono soundtrack. It is made of a few long continuous recordings joined end to end. Within a recording there are jump cuts, where picture and sound skip ahead together, and brief audio glitches, where under half a second of sound is replaced by nearby sound. The soundtrack also carries timing jitter of a few milliseconds throughout, and its level changes abruptly by a few decibels every so often. Jump cuts, glitches, jitter and level changes are not faults.

In most of the reel, sound and picture are correctly synchronized: a punch landing on a pad or bag is heard at the moment it is seen. Some stretches have a sync fault. Find every faulty stretch and describe it.

Fault kinds

  • offset: the stretch plays its own sound, displaced in time. The offset at a moment is how many milliseconds late the sound is relative to the picture (positive = sound after picture, negative = sound before picture). Within one stretch the offset is either constant or changes at a steady rate (a straight line over time, at most 5 ms per second); its sign never changes and the line's magnitude stays between 100 and 400 ms. The offset does not jump at the ends: it moves steadily from 0 onto the line, and from the line back to 0, at about 100 ms per second, and start_s and end_s are the midpoints of those two transitions.
  • foreign: the sound belongs to a different moment of the reel, at least 10 minutes away (it may be in a different recording). It stays in step with that other moment (no offset or drift of its own relative to it). A foreign stretch is 6 to 16 seconds long.

Structure you can rely on:

  • Faulty stretches start and end at arbitrary moments, not at cuts. No faulty stretch contains the join between two recordings, and no foreign stretch contains a jump cut.
  • Between two faulty stretches there are always at least 4 seconds of correct sync.
  • Sound used in a foreign stretch is never also heard at its own place in the reel.
  • "Correctly synchronized" means aligned the way the unaffected parts of the reel are aligned. Measure offsets against that.

What to report

For each faulty stretch:

  • start_s, end_s: reel timestamps in seconds where the fault begins and ends.
  • kind: offset or foreign.
  • For offset: offset_start_ms and offset_end_ms, the value of the stretch's straight-line offset at start_s and at end_s (equal if the offset is constant).
  • For foreign: source_s, the reel timestamp in seconds whose picture belongs with the sound heard at start_s, that is, the place in the reel where that sound should have been. If that picture does not appear anywhere in the reel, source_s is null.

A stretch counts as correct only if every field is right: same kind, start_s and end_s each within 1 s, both offsets within 40 ms, and source_s within 1 s (or both null). Reporting a stretch that is not faulty, or reporting one faulty stretch as several pieces, counts against you.

Output

Write output/sync_ledger.json:

{
  "stretches": [
    {"start_s": 812.40, "end_s": 841.73, "kind": "offset", "offset_start_ms": 160, "offset_end_ms": 160},
    {"start_s": 1502.10, "end_s": 1561.87, "kind": "offset", "offset_start_ms": -130, "offset_end_ms": -310},
    {"start_s": 2003.34, "end_s": 2019.95, "kind": "foreign", "source_s": 3561.2},
    {"start_s": 2400.03, "end_s": 2411.51, "kind": "foreign", "source_s": null}
  ]
}

Entries may be in any order. Write the file as soon as you have a first finding and keep overwriting it as you go; whatever is in the file when the session ends is what counts.

Rules

  • Work only inside this working directory. There is no internet access. Do not rely on memory or outside knowledge of these recordings; every entry must come from what is in reel.mp4.
  • Nothing is installed beyond what is already there: ffmpeg/ffprobe, sox, and Python 3 with numpy, scipy, pandas, opencv-python-headless, soundfile, pillow and matplotlib. pip install will not work.
  • There is no way to play the media; analyze it with the tools above.

Media

Frame from reel.mp4
reel.mp4 · 400 s
Frame from reel.mp4
reel.mp4 · 7550 s
Frame from reel.mp4
reel.mp4 · 11740 s

Frames above are from the task’s input videos. Media files are distributed separately from the repository; this task’s hash manifest lists 1 files (environment/media/SHA256SUMS.txt). Licences named for this task: CC BY.

Score

AgentScore (0 to 1)
GPT-6-Astra0.407
Claude Opus 5.50.592
Claude Opus 50.044
Gemini 3.8 Flash0.000
DeepSeek 4.1 Flash0.000

One run per agent.

The agent watches a 261-minute reel of boxing training and lists every stretch where sound and picture are out of sync: its start and end, its kind, and either the offset at both ends or the moment the sound was taken from. The reference answer holds 35 stretches, 31 with an offset and 4 with foreign sound.

What a good answer has to get right, and how the verifier checks it

Every faulty stretch is found, and nothing in sync is listed
The score is an F1 over stretches with one-to-one matching. A missed stretch lowers recall. A listed stretch that matches no reference stretch lowers precision, and so does an entry the verifier cannot read.
The kind of fault is right
A stretch counts only when its kind, offset or foreign, is the same as the reference stretch’s.
The stretch is placed in time
Its start and end must each be within 1.0 s of the reference. The verifier’s default is 0.5 s; the ground-truth file sets 1.0 s for this reel, because the audit that re-measured every stretch from the media alone was off by up to 0.78 s.
The offset is measured at both ends
For an offset stretch, the offset at the start and the offset at the end must each be within 40 ms. A drifting offset therefore has to be reported as drifting, not as one average value.
The source of foreign sound is identified
For a foreign stretch, the moment of the reel the sound was taken from must be within 1.0 s of the true one.

How the checks become one score

Score = F1 = 2 × precision × recall / (precision + recall). Precision is matched entries over listed entries; recall is matched stretches over 35. Every stretch counts the same. A stretch that is right on some fields and wrong on one counts as not matched: there is no credit for a stretch found at the right place with the wrong offset.

What scores zero

No sync_ledger.json, a file that is not valid JSON or whose “stretches” is not a list, an empty list, or a list in which no entry matches a reference stretch on every field.

Outside the score: the jump cuts, audio glitches, jitter and level changes that the task says are not faults, and 5 stretches of a decoy class that the ground truth marks as unscored. An entry that matches one of those 5 on every field is left out of the count; any other entry placed on one counts as wrong.

The score line and constants in the verifier code

Reward formula in the verifier code

Two sampled runs

Two of the five runs on this task in the paper’s diagnostic sample, which has one run per system: the highest-scoring run, and the run in the middle of the other systems by score. Only runs that ended normally and whose record could be exported are eligible. Each is shown step by step: what the agent said, which tools it called, and what it got back (long outputs and images are clipped). The verifier's output for the run comes first, then the steps.

Agent
GPT-6-Astra · Codex 0.154.0
Score
0.41
Wall time
35.4 min
Steps shown
112

Loading the run…