i've spent the last couple years experimenting with LLMs that write procedural, ambient music: instead of generating an audio file, we're using systems that write actual source code (supercollider synths) so the music runs live and can be embedded in and controlled in real time by software like games. a surprising amount of the work is taste -- prompt engineering and curating an aesthetic so the generated code reliably compiles and sounds good, with a compiler and automatic repair in the agent loop when it doesn't build. today, it produces single-synth musical systems, but I'm researching how to orchestrate multiple synths into fuller, evolving compositions with deep musical form and structure. LLMs make algorithmic music far more accessible to artists, and I'm eager to hear the new kinds of music that come out of it. hit me up if you're an audiophile interested in collaborating.
a little suite of word games I've been building as the testbed for Underscore. every puzzle is generative: crosswords with LLM-written clues, a word ladder, a guess-as-it-draws game, and a deduction game scored on how few clues you need. the crossword grids come out of a Rust constraint-satisfaction solver — backtracking fill over a bitmask-indexed word list, enforcing 180-degree rotational symmetry and slot intersections. each generated puzzle gets its own generated, continuously-evolving soundtrack with no audio files involved. right now, the music is generated per puzzle, but it needs some work. it's currently too drone-y, and needs to be more magical/whimsical. where I want to take it is having it respond to the gameplay itself — which is really the whole point of Underscore
I’m working on Underscore: a web SDK and browser sandbox for making procedural music systems with LLMs.
The basic idea is "music with source code." Instead of prompting for finished audio files, you use an LLM to help write and revise a SuperCollider-based system that runs in the browser via WebAssembly [1]. The result is executable music: inspectable, editable, versionable, and controllable at runtime.
I’m especially interested in adaptive sound for software: games, creative tools, meditation apps, AI agents, interactive art. Places where a static audio file feels too dead, but hiring a composer/sound designer for every variation is unrealistic.
It’s early, but the thesis is that LLMs make algorithmic music much more approachable because code becomes a conversational medium. I wrote a longer piece about the idea here: https://x.com/osetinsky/status/2053674503801028944?s=20
- Sam Aaron for building SuperSonic, allowing for SuperCollider in the browser as an AudioWorklet: https://sonic-pi.net/supersonic/demo.html. Earlier, pre-LLM versions of Underscore relied on low-latency WebRTC implementations for streaming SC synths running on servers to browsers in real-time
Very half baked, even took that demo site featured in the video down. Intended to run hundreds of "channels" of music, each of which could support thousands of independent sessions (think cloud gaming but for audio/music).
Uses Pion WebRTC, Gstreamer, JACK, SuperCollider and a simple Golang API for converting http requests to OSC toggles to manipulate audio running on a cloud server in realtime over the internet.
Questions – is this crazy? Would game makers ever be interested in adopting a third party SDK for their music needs? What if that music had to be streamed over the wire (not baked into their console/mobile client builds)? Is the prospect of game player input influencing the sounds/music they're listening to compelling, or not really?
The people behind Dots are an eclectic mix of illustrators, musicians, animators, designers and engineers that believe everyone can enjoy a great game.
We're a happy team with great people in a beautiful office located in downtown Manhattan, with lunch provided four days of the week (breakfast on the other day):
Taleb talks about this (though he doesn't use the term Sortition) in Antifragile: "instead of having the rulers randomize the jobs of citizens, we should have citizens randomize the jobs of rulers, naming them by raffles and removing them at random as well. That is similar to simulated annealing—and it happens to be no less effective. It turned out that the ancients—again, those ancients!—were aware of it: the members of the Athenian assemblies were chosen by lot, a method meant to protect the system from degeneracy. Luckily, this effect has been investigated with modern political systems. In a computer simulation, Alessandro Pluchino and his colleagues showed how adding a certain number of randomly selected politicians to the process can improve the functioning of the parliamentary system."
I ask myself this question as it pertains to the future of music. Digital Audio Workstations such as Ableton Live and Logic Pro dominate the scene when it comes to making music with a computer (at least in pop music), and companies such as Splice are trying to create collaborative networks around them.
Text based interfaces and audio programming languages such as SuperCollider open up a whole new world of musical creativity, and would lend themselves to collaboration (IMO) better than GUI based ones. The obvious hurdle for people is learning how to create things/music/art with text.
I'm in the same position with similar circumstances. You did what you had to do - instead of going back to school and learning the theory, you decided to focus on the problem and build practical solutions in the form of prototypes/mvps. There's no shame in that.
My approach will be to stay humble and eager to learn. I want to hire people smarter than me so that I can grow by working with them.