GPT-2 piano
I trained a GPT-2-based LLM to create music, even though it sometimes sounds wobbly — it’s fascinating.
I got the idea from a project a friend of mine was doing at MIT. The plan was live piano music for PhantomHunt that reacts to what’s happening in the game. That never worked out, because the hardware it would have needed was ridiculous and completely out of scope. So it became a project of its own. To train the model, I asked another friend if I could use his M1 Max with 64 GB of memory.
You give it a short piano passage and it tries to continue it. There is no text prompt: the music itself is the prompt. It doesn’t make any sound either. It writes MIDI (which notes to play, when, for how long and how loudly), and a separate piano instrument turns that into audio.
The model is a GPT-2-style transformer with 12 layers and a context of 2,048 tokens, trained from scratch rather than fine-tuned from the English GPT-2. Instead of words, it learns musical events. I turn the MIDI into tokens with MidiTok’s REMI format, which marks bars and positions within a bar so the model can follow the beat. It’s inspired by Google Magenta’s research on symbolic music, especially Music Transformer, but it isn’t a copy: I used a plain GPT-2 instead of Music Transformer’s special attention, because it’s simpler and well supported.
For training I used MAESTRO, a collection of about 1,300 performances from an international piano competition, almost 200 hours of music. Despite the repository’s name, it was not trained on 12,000 pieces; the “12k” is a leftover working name. I split the files randomly into training, validation and test sets. Only the training pieces were transposed, up or down by up to three semitones and only when every note stayed within the range of a piano, so the model learns patterns in several keys. One weakness: the split is random by file, not MAESTRO’s official one, so closely related pieces can end up on both sides, which can flatter the validation results.
I trained on Apple Silicon with PyTorch’s MPS backend. Only two sequences fit at a time, so the gradients are collected over eight steps before each update, which works like a batch of 16. Before GPT-2 I looked at RWKV, which would suit live generation better, but the code I found relies on CUDA and Triton and doesn’t run on a Mac.
Training longer didn’t automatically make it better. The checkpoint after two epochs had a lower validation loss than the one after four (1.43 against 1.53), so epoch 2 stayed my best one. Loss doesn’t tell the whole story, though, so I also generated continuations from different checkpoints to hear how they compare. The results sometimes sound wobbly, but hearing a model continue a piano passage is fascinating. The sample on the homepage was chosen from a wider search of continuations, and its long pauses are shortened.
The game idea left one smaller experiment behind: you give it a short melody and an intensity from 0 to 1, and it plays back a softer and sparser or a louder and denser version. It only works on short phrases and is still an experiment.