Lightnews — Scholar-powered news

Jesse Engel @jesseengel.bsky.social · Aug 8

Thanks again to @hollyherndon.bsky.social, it's always fun collaborating with you 🙏

Jesse Engel @jesseengel.bsky.social · Aug 8

These latest features take us one step closer to live interactive generative music models that are open, personalizable, and run locally on-device, so you can really trust them to hack them like any other instrument.

Next step, live on-device inference, stay tuned… 😀

1 2

Jesse Engel @jesseengel.bsky.social · Aug 8

As we’ve seen with Genie, real-time interactive generative models opens up new worlds, and we’re excited to keep pushing the boundaries of what’s possible.

1

Jesse Engel @jesseengel.bsky.social · Aug 8

This new wave of features is the culmination of effort from many folks at Google DeepMind, on the Lyria team and beyond. We are grateful to work with so many amazing collaborators to build the next generation of human-centered music AI tools. 🙏

1

Jesse Engel @jesseengel.bsky.social · Aug 8

We hope Magenta RT finetuning can serve that purpose, and are happy to have their model as a first example of personalizing live generative music.

1

Jesse Engel @jesseengel.bsky.social · Aug 8

For years, Holly and Mat Dryhurst have pioneered artist-centric ways in which AI can be used to push the boundaries of expression, and how artists and communities can have more agency over developing models for their own purposes (as they open sourced her voice to holly.plus for experimentation).

Holly+

Holly Herndon's digital twin

holly.plus

1

Jesse Engel @jesseengel.bsky.social · Aug 8

We're thrilled to have worked with Holly Herndon to train a model finetuned on her vocal style to share with all of you.

Playing with Holly’s model, you can use additional controls to get vocalizations that contour the music or "color" the instruments with a timbre that resembles Holly's voice

1

Jesse Engel @jesseengel.bsky.social · Aug 8

We’re also excited to add support for artists to customize Magenta RT on their own training data!

g.co/magenta/rt-finetuning-colab

1 1

Jesse Engel @jesseengel.bsky.social · Aug 8

Audio injection also supports *live* music input. You can jam along with it and use it as an intelligent live looper!

Try it out for yourself on free Colab TPUs!

g.co/magenta/rt-audio-injection-colab

1

Jesse Engel @jesseengel.bsky.social · Aug 8

Magenta RT now supports what we call *audio injection* - a novel method for real-time audio input

You can use audio injection to transform and add layers to prerecorded music!

1

Jesse Engel @jesseengel.bsky.social · Aug 8

We’ve since been hard at work preparing new features. Check out our tech report on *Live Music Models*, including both Lyria RT (API) and Magenta RT (open)

📜 Paper: arxiv.org/abs/2508.046...
🔊Samples: g.co/magenta/live...

Live Music Models

We introduce a new class of generative models for music called live music models that produce a continuous stream of music in real-time with synchronized user control. We release Magenta RealTime, an ...

arxiv.org

1

Jesse Engel @jesseengel.bsky.social · Aug 8

ICYMI, in June we shared a preview of Magenta RealTime - the first open weights model capable of *live* music generation

🤗 Model: huggingface.co/google/magen...
⭐ Code: github.com/magenta/mage...

google/magenta-realtime · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

1

Jesse Engel @jesseengel.bsky.social · Aug 8

Realtime interactive generative models FTW!

Announcing a new 🌊 of details and features for Magenta RealTime, the open weights live music AI model from GDM!

* Live Jamming with audio input 🎤🎸🎵
* Personalize your own models 🔧
* Tech report 📜

Links below in the 🧵...

1 1 3

Jesse Engel @jesseengel.bsky.social · Jul 9

Looking forward to seeing all sorts of interesting ways you mangle and twist the outputs into unique creations 🙂.

1

Jesse Engel @jesseengel.bsky.social · Jul 9

We are excited to make these models more accessible, and more tightly integrated into musician’s workflows as it will open up more opportunities to customize and personalize how you use them to come up with truly personal expression.

1 1

Jesse Engel @jesseengel.bsky.social · Jul 9

Built on the hard work of the Lyria Team, the VST plugin has been a labor of love of Ashutosh Desai. Thanks so much also to the support of artists Sidecar Tommy from Beats Antique, Toro y Moi in exploring the possibilities of DAW integration of our models.

1 1

Jesse Engel @jesseengel.bsky.social · Jul 9

The Infinite Crate is cross-platform, and currently supports Mac VST/AU, Windows VST, and a Mac Standalone application (no VST host required).

It is built on the Lyria RealTime API and thus requires an internet connection and API key (usage is free).

1 1

Jesse Engel @jesseengel.bsky.social · Jul 9

Be a Prompt DJ 🎧🎶: Play it like a live instrument. Use prompts and advanced controls to vary the instrumentation, texture, and genre. Incorporate with a DJ set to apply effects and mix with other loops.

1 1

Jesse Engel @jesseengel.bsky.social · Jul 9

Jam Along 🎸🥁: Create dynamic backing tracks that are different every time, varying and evolving as you play on top.

1 1

Jesse Engel @jesseengel.bsky.social · Jul 9

What can you do with it?

Dig in the crate 🎛️💡 : Explore new sounds by blending and merging genres, moods, instruments, or anything you can think of. Sample the stream of music to spark your own inspiration or help you get past a creative block.

1 2

Jesse Engel @jesseengel.bsky.social · Jul 9

New VST/AU Plugin! 🚨

Play with Lyria RealTime directly from inside your favorite DAW with “The Infinite Crate” 🎧🎶

With a VST plugin you can feed audio directly into your DAW for sampling, live performance, or even a practice partner to jam with.

💾 Get it here: g.co/magenta/infinite-crate

1 1 5

Jesse Engel @jesseengel.bsky.social · Jun 5

Try it out and fork it here:

bit.ly/3ZOp7KS

Don't forget you can type in new prompts,

Curious to see what else you can make :).

Jesse Engel @jesseengel.bsky.social · Jun 5

Pleasantly surprised with how easy/fun it is to "vibe code" new musical experiences in AI Studio.

I'm not much of a JS guy, but had Gemini help me make a "Kaoss Pad" emulator using the Lyria RealTime API to explore musical latent spaces in realtime.

1

Jesse Engel @jesseengel.bsky.social · Apr 23

How are they being impacted by the current approach to deploying/integrating AI in our societal habitat? How could we be more sustainably approaching things?

1

Jesse Engel @jesseengel.bsky.social · Apr 23

There's a lot of inspiration to be found in nature, seeing the interconnectedness of a habitat and the effects of disruption on one part of it. What are the core pillars of our societal habitat that we need to make sure survive going forward? How do support the broader system?

1