Teaching a machine to read piano, by ear and eye
A near state-of-the-art piano transcription model that fuses what it hears with what it sees.
In service of Musicians and researchers who want an honest transcript, not a confident-but-wrong one.
Result Beat audio-only baselines by ~8%, near state of the art — a first-class dissertation (Distinction).
My first-class dissertation: a novel piano music transcription system that fuses deep audio and visual models, beating standard audio-only models by roughly 8%.
Turning a piano performance back into notes is hard from audio alone, because sound is ambiguous in ways a video of the hands is not. So I fused the two: a deep audio model and a visual model watching the keys, each covering for the other’s blind spots.
The result beats a standard audio-only baseline by about 8%, near the state of the art. It earned a first, with 77% on the writing and 73% on the presentation — I cared as much about explaining it clearly as building it.
This is the project that taught me how much careful evaluation matters: knowing exactly where a model wins, and where it quietly fails.