Check out the paper, plugins, and code for more details, and join the Discord server to stay up to date. Finally, a huge thank you to my collaborators at Neutone for the amazing work they’re doing!

At a time when many AI companies are competing with artists and training on their work without permission, the SDK democratizes this technology and provides a foundation for AI tools that enhance rather than replace human creativity.
neutone.ai/blog/neutone-on-ai-and-copyright
(8/8)

To date, the SDK has powered a wide variety of applications such as neural audio effects, timbre transfer, sample generation, and stem separation, as well as seen adoption by researchers, educators, industry, and artists alike.
neutone.ai/artists
(7/8)

Since early 2022, Neutone FX has made the latest realtime neural audio models accessible to artists around the world. It includes a model browser that allows one to search for and download user models that have been shared and uploaded to the Neutone servers via the SDK. (6/8)

We provide a technical overview of the interfaces needed to accomplish this, as well as the corresponding SDK implementations. Personally, I love prototyping neural audio models in Python with the SDK, and listening to the results in the DAW seconds later after exporting. (5/8)

By encapsulating common challenges like variable buffer sizes, sample rate conversion, and delay compensation within a model-agnostic interface, our framework enables seamless interoperability between neural models and host plugins while allowing users to work entirely
in Python. (4/8)

The Neutone SDK is an open source framework that streamlines the deployment of PyTorch neural audio models for both real-time and offline applications. It enables researchers to wrap their own PyTorch models and run them in the DAW using our free host plugins FX and Gen. (3/8)

We’re also releasing the beta version of Neutone Gen, the counterpart to Neutone FX that continues to bridge the gap between audio researchers and artists. Now, you can export heavy-weight, non-realtime models using the SDK and run them in the DAW via the free Gen plugin. (2/8)

Lastly, a huge thank you to my collaborator Hao Hao (github.com/gudgud96) and supervisor Josh (www.eecs.qmul.ac.uk/~josh/) for their help and contributions!

Our code is open source (github.com/christhetree/mod_discovery) and the trained synths are available as VST plugins via the @neutone.bsky.social platform and SDK.
Listening samples, visualizations, plugins, and more can be found at christhetr.ee/mod_discovery (7/7)

We evaluate our modulation discovery framework on unseen real-world modulation curves, highly modulated synthetic and real-world audio, and on white-box, gray-box, and black-box synth architectures. (6/7)

We investigate three modulation signal parameterizations:
• Framewise (Frame)
• Low-pass filtered (LPF)
• Piecewise 2D Bézier curves (Spline)
We find that LPF and Spline yield human-readable curves that trade sound-matching accuracy for interpretability. (5/7)

We apply our approach to a differentiable synthesizer inspired by the popular soft synths Serum and Vital with wavetable, filter, and envelope modulations. We also demonstrate its ability to generalize to other DDSP synth architectures. (4/7)

We propose a self-supervised neural sound-matching approach that leverages modulation extraction, constrained control signal parameterizations, and differentiable digital signal processing (DDSP) to discover the modulations present in a sound. (3/7)

Modulations are a critical part of sound design, enabling the creation of complex, evolving audio. However, finding the modulations in a sound is difficult and typical sound-matching / parameter estimation systems don’t consider the structure or routing of underlying modulations. (2/7)