How Fusion Karaoke Turns Every Song into a Game with Real-Time AI Visuals and LALAL.AI API
Working at one of Berlin’s most iconic karaoke clubs, Lorenzo Minneci noticed that the song videos had barely changed since 2005. His answer is Fusion Karaoke, where every lyric line turns into live visuals.
Our interview series with the winners of the Musicathon run by Musixmatch continues, and this time we’re talking to the creator of the project that the hackathon judges ranked first. Fusion Karaoke turns an ordinary karaoke night into an interactive audiovisual show where lyrics become real-time visuals and singers gradually morph into the original artist.
Behind it is Lorenzo Minneci, a Berlin-based developer, musician, and DJ who knows karaoke from the inside. We asked him about Fusion Karaoke, how this brilliant idea came about, which tools helped him bring it to life, and what role LALAL.AI plays under the hood.
“Developer, musician, DJ, or something in between? I never felt comfortable in just one box”
I started with a bachelor’s in electronic engineering, with the idea of building synthesizers and sound machines. I’d been learning instruments since I was a child: piano, electric bass, violin, and for the last 13 years Ableton Live and synthesizers, which turned out to be great for sparking creative ideas.
At that time, I wanted to think of engineers as inventors, but I soon had to realize that engineering school didn’t encourage any creative approach. The only way forward was to do it myself: learning and building projects on my own.
So I moved to the Netherlands to study Sonology, a course with a holistic and experimental approach to sound and everything around it. That experience helped me believe that my vision wasn’t crazy, and that what excited me wasn’t art, electronics, coding, or sound on their own, but the combination of all of them.
I then went back to Italy to solidify my coding skills with a master’s in Music and Acoustic Engineering at Politecnico di Milano.
Now I’m based in Berlin, where I moved almost four years ago. As a musician and DJ, I go by Haimzu, and I also do new media arts research and interactive, immersive installations for universities and private clients.
Unlike what I expected in the past, I actually enjoy keeping those two roads quite separate!
The love for audiovisual work started with music, and with the need to expand that language into other forms. I’d say it happened organically over the years and, even though I still don’t consider myself a visual artist, connecting sound, art, and technology sometimes ends up having a visual output too.
Last year, for example, I was asked by Kokoschka Revival to make an interactive installation for a permanent exhibition in a castle in Tuscany connected to Ludovico Ariosto, the Renaissance Italian poet. I ended up developing a magic mirror where people see themselves morphing, in real time, into the different characters of the poem.
That’s how I learned StreamDiffusion, which then fitted very well into the idea for Fusion Karaoke.
“I was surprised to realize what karaoke does to people”
Working as technical manager at probably the most iconic karaoke club in Berlin definitely inspired me to pull the ideas together.
But as an electronic musician and sound nerd, I was surprised to realize what karaoke does to people, and what they actually come to experience in a place like that. It’s socially a fascinating place: people are focused on being together and co-expressing through singing, and much less on music quality or hitting the right note. People like to be silly and free, and karaoke creates a perfect world to get lost in, or to reimagine the one we’re living in.
The karaoke club I work at opened in 2005, and the video quality of the songs has barely changed since then. People now spend their whole lives in front of computers, trying to optimize everything… so I noticed that, when they go out, they want the opposite: something simple that engages them actively in a funny way.
That’s when I started thinking about how to “gamify” karaoke. First, with something like Mario Kart-aoke, or a Google Dino version of it. In the end, the idea for Fusion Karaoke came up: playful, generative, and interactive, but still focused on the karaoke experience itself. It felt immediately like the right one.
Regarding Fusion Karaoke’s commercial potential, my initial instinct would push me straight towards a democratic tool that anyone could buy and use. Ideally, Fusion Karaoke would run as an app on a tablet or a phone, but that’s unrealistic right now, given the technical limitations: it needs a specific GPU to run. For now.
I’ve already received some requests to show Fusion Karaoke at venues in Berlin, and I’m definitely looking forward to bringing it to a real crowd! We’ll probably premiere it at our own karaoke club.
The Performer Literally Becomes the Song
Fusion Karaoke was meant to make karaoke more interactive, primarily for the singers and the crowd. With the camera, anyone standing next to the singer can play with it too, and because the images keep changing, everyone stays engaged and curious about what the next line (or the next song) will produce.
For the same reason I mentioned before, even the “translation” mistakes between lyrics and visuals often trigger funny moments for the crowd and the singers. And it’s great to see the moment when things like that stop being bugs and start becoming features people enjoy playing with.
Tech behind Fusion Karaoke and Where LALAL.AI Comes In
Fusion Karaoke’s main brain is TouchDesigner (a node-based visual programming platform used to create real-time interactive multimedia). Alongside it, there’s the user interface, the controller for the KJ (“Karaoke Jockey”, as we call the karaoke host), which runs locally and exchanges information with TouchDesigner.
The visual input is a simple USB camera. I use NVIDIA Broadcast to remove the background, since it’s more efficient than doing it inside TouchDesigner. The microphone signal comes in through an audio interface, ready to be analyzed.
Fusion Karaoke has two main visual modes: “Become the Lyrics” and “Become the Singer”. In both, once a song is searched and added to the queue, the lyrics are pulled from the Musixmatch API.
In “Become the Lyrics”, each line of lyrics is analyzed by a small language model running locally on my computer, which reinterprets it, line by line, into a prompt that makes visual sense. Those prompts feed StreamDiffusion’s image-to-image model, which generates the visuals in real time. There’s also the option to condition the generation with any style or theme, which helps customize the experience.
“Become the Singer” is a bit more complex technically, and this is where LALAL.AI comes in.
Unlike in the previous mode, an algorithm retrieves the name of the song’s original artist once a song is selected, and that becomes the core of the prompt. I first wanted to test the system on a small set of songs, so I split them into two stems with LALAL.AI beforehand: vocals and instrumentals. Having both the original vocal track and the live microphone from the stage, I could calculate a score function and map it to the transformation. The better the singing performance, the closer the morphing gets to the actual singer of the song. The score goes up when the two voices happen simultaneously, and down when they don’t.
On top of that, I also inject small “emotional state” prompts (“very happy”, “neutral”, “sad”) depending on the score, so that the generated singer is as enthusiastic as the performance deserves!
The visuals are generated at 8-9 FPS, and they’re quite reactive to the lyrics-prompt change, which happens with no noticeable latency. So luckily, visual lag wasn’t the biggest issue.
The processed audio is laggier, though. In a real venue, I’d duplicate the voice signal with a mixer or route it properly in the audio interface, sending it both into the interface and straight to the output, so it stays real-time. Then I’d process its RMS and compare it to the offset RMS of the original signal, compensating for the total latency of the system.
“The hardest part was synthesizing all the good ideas I had in my head into one simple, funny ‘pop’ tool to play with. Something anyone could understand immediately.”
If I really have to pick one engineering problem, it’d be getting a clear vision of the division of labor between the browser app and TouchDesigner. I wanted quite a lot of control over features and functionality, so I decided to let TouchDesigner be the brain of the system and keep the app purely as a controller interface.
“I used LALAL.AI to isolate the vocals and use them as ground truth for the scoring function”
The “one line of art direction” (for instance, “neon 80s”) modifies every per-line prompt. I wanted to keep the system as lightweight as possible, and that would have been impossible if every song needed a different model or architecture to run.
Local language models turned out to be good enough for this, and for the lyrics-to-prompt translation in general. This makes me happy, because it means I don’t have to rely on external “intelligences” via API, and there are no extra costs. Once the lyrics are downloaded for a given catalog, literally everything can run offline.
LALAL.AI API comes in exactly there! I used LALAL.AI to isolate the vocals and use them as ground truth for the scoring function. Initially, I also applied a pitch detector to measure the quality of the performance, but I removed it; it was too unreliable to perform well across every song automatically.
I also had a quick look at the semi-real-time LALAL.AI plugin, which looked very promising. But given the time constraint of the hackathon, I preferred to extract the stems beforehand and pull them from the database, to keep it as real-time as possible.
I’ve actually heard of LALAL.AI for many years, since the days when I was playing around with open-source stem separation repos on GitHub. Yours seemed to be the first product delivering it in a sleek, finished way, so I was very happy to see it among the partners.
The Road Ahead
I’ve got plenty of ideas for new projects! I’m finishing a music EP, and lately I’ve been learning 3D printing, as I want to be able to bring my digital ideas into the real world.
I’m a bit sick of screens and digital-only apps… I think people have had enough of those, and especially in the art and entertainment industry, it feels like everyone is leaning towards phygital experiences and products.
In the near future, I want to establish a creative studio working at the intersection of art, sound, and technology, and I’m fully committed to creating a positive impact in this messy world. That’s a responsibility I feel during these times, which is also the reason Fusion Karaoke is an experiment in how offline AI can be sustainable and fun.
Follow LALAL.AI on Instagram, Facebook, Twitter, TikTok, Reddit, LinkedIn, and YouTube to keep up with all our updates and special offers.