Making Music and Art Through Machine Learning

Eck says Magenta aims to build tools that help people make new kinds of art, rather than to make its engineers into machine learning artists. He sees potential in the quirks and failures of new media, such as sounds created by moving between learned representations.

Magenta’s NSynth project generates sounds by compressing audio into a space that users can explore, though listening to those sounds was not yet real time. Eck says the team wanted stronger music models that account for expressive timing and dynamics, and feedback from musicians showed that command-line tools were not fluid enough for their workflows. Artists also used SketchRNN, trained on 20-second Quick, Draw! sketches, to sample drawings and find unusual examples.

Eck describes AI Duet as a call-and-response game: musicians listen to a simple model’s output and adapt their playing, instead of expecting it to continue a melody like “Für Elise.” He says musicians brought their own improvisational skill to these experiments. He also wants Magenta to support creative coding, where people can retrain models or alter data to shape results.

Eck explains that generative models can play it safe, while adversarial training and reinforcement learning can push them toward less predictable results. In one experiment, deep Q-learning rewarded an LSTM for following simple 19th-century counterpoint rules, producing music he found catchier. He suggests users could write measurable rewards for qualities they want, such as avoiding straight lines in sketches or making music sound shimmery.

Eck hopes future models can create longer pieces with structure, such as chord changes, while handling local details like expressive timing and musical texture. He says creative coding needs more accessible tools, perhaps something as approachable as GarageBand, and that Magenta’s next steps include making its API more usable and expressive. He invites people to explore the project site, open issues, and discussion list.