How AirStems Uses LALAL.AI API to Let You Remix Any Song with Your Hands

The Musicathon run by Musixmatch in June has come to an end, so we wanted to chat with the winners and share a bit more about the projects than the participants did in their submissions. 

AirStems is among the tools that have taken the LALAL.AI Special Award during the event; it allows you to remix songs with your… bare hands. AirStems turns stem separation, synced lyrics, and AI tagging, that typically live behind sliders and a timeline, into a live instrument. As a result, you can remix tracks in real time with your hands without touching a single button in your DAW. Sounds futuristic? 

We sat down with Lluís, the creator of AirStems and talked to him about the idea behind AirStems, how he built the tool, his plans on developing the project, and how LALAL.AI API helped him bring this creative idea to life. 

“Musicathon partner APIs reshaped my existing project” 

AirStems is built on my existing open-source project, Aetheric Geometry, which began from a straightforward idea: I wanted to play and shape sound using only my hands and a webcam, without relying on dedicated controller hardware. It’s a gesture-based instrument in which MediaPipe hand-tracking drives an additive synthesizer tuned in just intonation. I had been developing it for roughly a month before the Musicathon, and by then, it already included the hand-tracking, the DSP effects, and the rendering that AirStems would later reuse.

The key realization was that hackathon’s partner APIs allowed me to change its source: instead of my hands controlling oscillators, they could control the actual vocals, drums and bass of a real song. In that sense, the theme and the APIs reshaped an existing project, even though these services don’t belong in the real-time loop at all. LALAL.AI, Musixmatch and Cyanite all run in advance, offline, and write their results to disk indexed by the song's name. The live engine then only accesses local files: the stems, an LRC lyrics file, and a small JSON file of analysis data.

"I had expected the stems to degrade under aggressive real-time soloing, which is a more demanding test than simply sitting within a mix, but they held up well."

Separating the slower, network-based preparation from the real-time performance is what keeps the instrument stable, since the audio pipeline never has to wait for a network request during playback.

Besides, I wrote a custom sounddevice callback in order to have full control and the lowest possible latency, mixing four stereo stems together with a filter and reverb on each audio block. The most difficult aspect was achieving click-free transitions: muting or unmuting a stem instantly creates a discontinuity, so every gain change is applied as a short ramp rather than an abrupt jump, and the callback itself must remain lightweight and free of memory allocation in order to avoid audio dropouts. This real-time, lock-free approach is something I carried over from developing JUCE plugins.

"Conduct a song with your bare hands" is the main idea behind AirStems

I built AirStems entirely on my own. It was demanding, but manageable because I wasn’t starting from scratch. The hand-tracking, the real-time DSP and the rendering had already been developed and tested in Aetheric Geometry, which allowed me to focus the hackathon on the genuinely new components: the multi-stem playback engine, the beat-synchronization, and the three API integrations. Reusing my own existing framework is what made a solo project feasible within the timeframe.

The idea "conduct a song with your bare hands" comes from music production. When mixing a track, you’re already deciding where a vocal should drop out or where the drums should return.

"I wanted to turn that decision-making into a physical performance rather than an operation carried out with a mouse." 

I deliberately use the word "conduct" rather than "control" because the goal is to perform the song with intention and timing, not simply to operate it. Since the control is entirely gesture-based and contactless, it also lends itself to more practical uses: you can mute a specific part in order to perform it yourself; for example, muting the guitar to play along on a real guitar, or muting the vocals to sing over the track as in karaoke, all without touching a mouse or any hardware.

A mouse controls a single parameter at a time. Hands, by contrast, are continuous and can act in parallel: in a single movement, I can mute several stems with one hand while adjusting a filter with the other and controlling reverb through the openness of my fingers. This allows several expressive dimensions to be controlled simultaneously and in time with the music. It stopped being a novelty for me at the point where muting a stem on the beat felt like playing an instrument rather than operating software.

The most memorable moment was the first time I connected the stems to the gesture engine. At that stage, there was no gating or smoothing in place, so every hand movement triggered everything simultaneously: all four stems played at once and the filter and reverb responded erratically, producing a very chaotic result.

"Despite that, it was clear that the underlying concept worked, as my hand movements were genuinely driving the song. That first disorganized test is what convinced me the idea was worth developing properly."

How AirStems works for an end user 

The user turns on the webcam, raises their hands, and immediately sees them represented on screen. The core mapping, four fingers for four stems, is intuitive enough that most people can mute and reintroduce parts within a minute.

Hand height for the filter and hand openness for reverb take a few minutes more to become comfortable with. Beat-synchronization shortens the learning curve considerably, because changes are aligned to the beat and therefore remain musical even while the user is still learning.

Each mapping is designed so that the gesture reflects its function.

🟡 Fingers control the stems because four fingers correspond naturally to four parts, and the action is discrete: a part is either present or not.

🟡 A closed fist or open palm triggers a full drop or full mix, as it’s the most immediate and expressive gesture available. 

🟡 The height of the left hand controls the low-pass filter, since raising the hand is an intuitive metaphor for opening up the sound.

🟡 Opening and closing the hand controls reverb, as spreading the hand corresponds naturally to opening up space. The aim was for the mappings to be ergonomic and intuitive, so that they do not need to be memorized.

🟡 Low-pass filter and Schroeder reverb map cleanly onto the two remaining gestures: a low-pass filter for brightness, controlled by hand height, and reverb for a sense of space, controlled by hand openness.

Most of the tuning involved smoothing: a filter cutoff that follows the natural jitter of the hand directly produces audible stepping, so the cutoff is smoothed enough to feel responsive without becoming sluggish. For the Schroeder reverb, the work consisted of selecting comb and all-pass settings that provide a musical tail without blurring transients or placing excessive load on the audio callback.

When a song is loaded, librosa analyzes the drums stem and produces a map of where each beat occurs, essentially a grid of timestamps. Without beat-synchronization, a gesture takes effect immediately. With it enabled, the gesture becomes a request that is held and then applied at the next beat on that grid. In other words, the hand determines what change is made, and the song's own rhythm determines when it takes effect.

If a gesture comes in slightly early or late relative to the beat grid, it always resolves to the next beat, it's queued and applied at the following beat on the grid, so a movement made slightly early simply waits a few milliseconds, and one made slightly late is applied on the next beat. It’s a quantization to the nearest upcoming beat rather than a predictive look-ahead; I preferred reliable alignment to the grid over attempting to anticipate the user's intention.

For users who want immediate, unquantized control, beat-synchronization can be switched off.

"Hand-tracking latency is low enough that gestures feel instrument-like as the key principle is that the audio thread never waits for the camera."

Hand-tracking runs independently and simply writes the latest gesture state to shared variables that the audio callback reads, so any delay in the vision processing never interrupts the sound. I also keep the video frames light for MediaPipe. Beat-synchronization helps further, because changes are aligned to the next beat and any small tracking latency is absorbed within that alignment.

“Integrating the LALAL.AI API was one of the more straightforward parts of the project”

Separation quality was the most important factor. In a studio context, a small amount of bleed between stems isn’t noticeable, because the stems are heard together; in a live instrument, however, as soon as a stem is soloed or fully muted, any bleed becomes immediately apparent. Low cross-talk was therefore essential, along with full-bandwidth stems of consistent loudness, delivered as lossless WAV files so that they can be played back and mixed sample-for-sample without compression artefacts. 

Once the stems are returned, I ensure that all four are the same length and aligned from the same starting sample, and then load them into memory so that the audio callback never has to read from disk. The drums stem is analyzed with librosa to extract the beat grid used for quantization. From there, the process is a straightforward per-sample mix using the smoothed gains and ramps; because everything is pre-aligned, synchronizing to the beat is simply a matter of determining which sample index corresponds to the next beat.

I used the LALAL.AI API for the separation because AirStems requires the stems as files that it can load, buffer, and mix itself.

"Two factors were decisive: the stems should be provided as clean, separate files that I can load directly into memory, and the separation quality should be high enough that muting and soloing them in real time sounds convincing. That combination is what makes LALAL.AI API viable."

Integrating the LALAL.AI API was one of the more straightforward parts of the project: uploading a track, waiting while it’s separated, and downloading the resulting stems. The client itself was quick to implement, and most of the effort went into handling the asynchronous polling and the waiting period cleanly, rather than into the API itself. Compared with building the real-time engine, the integration was relatively simple.

I had expected the stems to degrade under aggressive real-time soloing, which is a more demanding test than simply sitting within a mix, but they held up well.

Being able to reduce a song to its vocal alone and then reintroduce the full arrangement in real time and without noticeable artefacts was the point at which I became confident that a studio-grade separation tool could support a performance instrument.

The road ahead 

My aim for AirStems is to be used to perform a song's stems live using only the hands. It works well as a practical practice tool, too, since a specific part can be muted so that the user performs it themselves; for example, muting the guitar to practise along with the track or muting the vocals for karaoke. That makes it accessible even to people who don’t own controllers or a DAW, so there’s a clear educational dimension, and it would also lend itself well to an interactive installation.

If I kept developing AirStems past the hackathon, the main additions would be per-stem soloing, controlled by a pinch gesture, and master volume controlled by the distance between the two hands, which would be a natural extension of the current design. 

Beyond that, I'd add the ability to record a remix directly to a WAV file, so that a performance could be saved as a finished track, along with word-by-word synchronized lyrics from Musixmatch.

As LALAL.AI adds further stem types [to its flagship model], such as guitar and piano, each additional stem provides another element to control.

My intention is to open-source the multi-stem player and beat-sync engine. The AirStems repository is already public, and Aetheric Geometry showed me the value of releasing this kind of work openly. 

I would like to refine the multi-stem player and the beat-synchronization engine as well as document them properly, so that others can build gesture-driven audio applications on top of them.

Follow LALAL.AI on Instagram, Facebook, Twitter, TikTok, Reddit, LinkedIn, and YouTube to keep up with all our updates and special offers.