01Dataset design
Scores are not enough
Scores failed in pilots. Trainers apply different thresholds to the same scale.
So we annotate coaching actions: identify · diagnose · prescribe · compare · localize.
Two subsets for two needs
One song, Amazing Grace, for everyone. Takes can be ranked against each other.
Different songs from nine public datasets. Varied styles and registers, closer to real coaching.
One protocol per recording
Per trainer: listen to the full take → review in issue-tagged blocks → rank the Top-3 issues.
Same-song adds a ranked triplet, because takes of one song are directly comparable. Diverse-song adds problem regions marked on the waveform, because with different songs the question is where the problem is, not who sang better. Two trainers per recording.
From reviews to evaluable targets
Reviews → atomic claims: diagnosis, correction (linked to its issue), strength.
Structured targets are scored deterministically. Free-form feedback is scored claim by claim, since trainers prioritize differently.
Counts by subset
| Annotation | Same-song | Diverse-song | Used for |
|---|---|---|---|
| Annotators | 6 | 12 | Two independent reviews per recording |
| Coaching review → atomic claims | 1,397 blocks · 3,972 claims | 3,046 blocks · 8,079 claims | Open-ended diagnosis and correction (judge-based) |
| Top-3 issue labels | ✓ | ✓ | Top-3 issue prediction |
| Triplet ranking of 3 performances | 450 rankings · 189 eval triplets | – | Pairwise ranking accuracy, Kendall's τ |
| Segment-grounded feedback | – | 2,526 segments · 262 consensus events | Segment issue classification |
Atomic claims: 6,087 diagnosis · 4,527 correction · 1,437 strength.
02Expert annotations
Four diverse-song recordings, two trainers each. Corrections are colour-linked to the issue they target.
03Model feedback vs. expert claims
The model listens to the same recording and writes feedback. A judge then checks it claim by claim against the trainers.
The model writes feedback: five diagnoses and five corrections, or free text that is split into claims afterwards.
Each expert diagnosis is checked against the model's diagnoses.
Each model correction is graded on its own.
04Leaderboard
13 aligned models: 9 open-weight and 4 closed API models. Updated 26 October 2026 from verified evaluation artifacts. Click a header to sort.
Top-3 uses seven-class macro-F1, not the older F1@3 overlap metric. Scores below are percentages; N/A means unavailable, not zero.
Details
- Denominator. All 515 recordings (same-song 207, diverse-song 308),
strict_all_attempts: malformed or missing outputs count as errors. - Top-3. Seven-class macro-F1 over 1,064 expert references, with each recording given equal total weight. PHONATION and the legacy internal label VOCALIZATION denote the same class.
- Judge.
openai/gpt-oss-120b: diagnosis = strict hit / coarse hit / miss / contradict; correction = valid / weak / invalid. Simple-protocol reviews are first decomposed into claims by the same model. - Protocols. Main = structured JSON prompts; Simple = short prompts with a distinct extraction protocol. Open/Closed identifies model access, not a separate evaluation protocol. The majority baseline sees no audio and has only a Top-3 metric.
- Ranking. Score-derived and Direct pairwise accuracy are separate tasks: 189 same-song triplets, 567 pair decisions, 1,134 expert-reference attempts. Audio Flamingo 3 and Music Flamingo Direct use concatenated audio; Covo Direct is N/A.
- Other denominators. Segment: 262. Diagnosis: 6,162 expert claims. Prescription validity: 2,575 fixed slots (515 × 5), including missing slots as failures; not validity conditional on emitted prescriptions.
- Claims policy. Corrected515 format-reported: empty, unparseable or backend-error outputs retain zero claims and remain in the fixed denominator. See per-model notes for generation and parsing exceptions.