Folks at #CCN2025, drop by my poster tomorrow to hear about the fun task I have been developing with Roland Fleming, to gauge internal models of diverse physical objects in a controlled way! There will be a demo as well 🕹️

Tuesday, August 12, 1:30 – 4:30 pm, de Brug & E‑Hall

This is o3-mini-high. Interestingly, in its reasoning you can see that it initially made the same mistake but then corrected itself.

...and a ConvNet combined with a differentiable sorting module (github.com/Felix-Peters... ), which learns to permute the rows and/or columns of the activation map. Seen here for prompts "Brutalist anime robot" and "A colorful tree on a blue sky painted by Cézanne"

We also experiment with a few different module architectures: a vanilla ConvNet (shown on the right), a ConvNet with spatial coordinates appended to the feature channels, which seems to help the model generate clearly segmented images (left)...

This is what the results look like, without (top) and with (bottom) diversity, for the prompts "Inflatable plastic bodybuilder in a colorful album cover painted by Magritte" and "A gang of biker pumpkins painted by Jan van Eyck".

The resulting "bending module" is a small-scale network that can be injected between any two layers of the generator, and gets trained while the rest of the model stays frozen. In our case, the base model is a GAN trained to generate butterflies! (huggingface.co/ceyda/butter...)

🚨1st Bluesky post + paper alert! 🚨
This project was a lot of fun, and I'm particularly happy to see it accepted in the NeurIPS ML4CD workshop! With @alinarogalska.bsky.social, @yoahmed.bsky.social @ohaijen.bsky.social we propose a method to 'hack' generative models... 🧵
arxiv.org/abs/2310.04816