svsdval/video2midi

What it solves

Synthesia piano tutorial videos show falling notes on a virtual keyboard, but they are not in a format you can edit or share as music. This tool automatically extracts the keyboard activity from such videos and creates a MIDI file, so you can replay, rearrange, or use the music for learning and sharing.

How it works

The program scans the video frame by frame. You load a downloaded Synthesia video (720p recommended), then in the graphical interface you adjust the on-screen keyboard positions and key colors to match the video. Pressing Q starts the automatic recognition: it tracks which keys are pressed at which moment, records note on/off events, and writes them to a MIDI file. You can also set start and end frames, force splitting into two channels based on a key position, and handle overlapping notes. After processing, a built-in note editor lets you fine-tune individual notes (move, resize, delete, split) before saving.

Who it’s for

It is aimed at musicians, piano students, or anyone who wants to turn Synthesia style videos into usable MIDI files. You need some technical comfort because you run a Python script and must install dependencies (OpenCV, Pygame, MIDIUtil). It is a hobby project but well documented for basic use.

Highlights

• Frame-by-frame scanning of the virtual keyboard to detect key presses
• Graphical interface to adjust key positions, colors, and recognition options
• Multi-channel MIDI output based on key colors (e.g., separate channels for different hands or instruments)
• Built-in note editor: move, resize, delete, split notes, and add notes manually
• All settings are stored in INI files for easy repetition
• Available for Linux, Windows (with Anaconda), and also as a web version and Android app
• Community support via GitHub discussions and linked video tutorials