Lightnews — Scholar-powered news

Mark Boss @markboss.bsky.social · 14d

Few examples from the demo. You can generate various styles from a single prompt. You can either pick semantic matching images, or not and get unexpected results.

[1] unsplash.com/photos/a-boa...
[2] unsplash.com/photos/mount...
[3] unsplash.com/photos/man-i...
[4] unsplash.com/photos/macro...

Mark Boss @markboss.bsky.social · 14d

Thanks to my co-authors Andreas Engelhardt, Simon Donné, Varun Jampani

Also check out the HF demo huggingface.co/spaces/stabi..., the code github.com/Stability-AI..., and the explainer youtu.be/ckcSgf0s-jI

ReSWD - a Hugging Face Space by stabilityai

Create images using color matching and guidance features. Upload your reference images and get generated images that match the colors and styles.

huggingface.co

1

Mark Boss @markboss.bsky.social · 14d

This can be used for multiple applications such as color matching or diffusion guidance. Here, we showcase the diffusion process of generating a medieval house with the reference to the right.

1

Mark Boss @markboss.bsky.social · 14d

Variance in MC is quite common in computer graphics so we combined ReSTIR -- more precisely the weighted reservoir sampling -- with SWD to keep more impactful random directions in the optimization.

1

Mark Boss @markboss.bsky.social · 14d

Happy to announce: ReSWD. Sliced Wasserstein Distances are quite powerful, but they perform a Monte Carlo (MC) integration (over random directions). During an optimization this can lead to noisy gradients due to variance.

Project page: reservoirswd.github.io

1 5 8

Mark Boss @markboss.bsky.social · Jun 11

I’ll also be sharing these and other works at the AI4CC workshop on the 12th at 11:00. ai4cc.net

AI for Content Creation Workshop

ai4cc.net

Mark Boss @markboss.bsky.social · Jun 11

There was SV3D where we mostly discarded any SDS loss (only for unseen areas). I was mainly working on the 3D part and it required quite a few tricks to make it work. sv3d.github.io

SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion

SV3D generates novel multi-view synthesis from a single input image.

sv3d.github.io

2

Mark Boss @markboss.bsky.social · Jun 10

3️⃣ MARBLE: Edit materials effortlessly using simple CLIP feature manipulation, supporting exemplar-based interpolation or parametric edits across various styles. Check it out: marblecontrol.github.io

Material Editing in CLIP Space

marblecontrol.github.io

1

Mark Boss @markboss.bsky.social · Jun 10

2️⃣ SPAR3D (follow-up to SF3D): Integrates a fast point diffusion module, enhancing depth, backside modeling, and enabling easier editing. Project page: spar3d.github.io

SPAR3D

SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images

spar3d.github.io

1

Mark Boss @markboss.bsky.social · Jun 10

1️⃣ SF3D: Generate textured, UV-unwrapped 3D assets with additional materials incredibly fast (<0.3s)! More details here: stable-fast-3d.github.io

SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement

SF3D generates high quality 3D assets from a single input image.

stable-fast-3d.github.io

1 1

Mark Boss @markboss.bsky.social · Jun 10

I’m at CVPR this week! Looking forward to connecting and discussing all things graphics, 3D, and gen AI. I'll be presenting 3 papers—stop by and chat!

1

Reposted by Mark Boss

Zhenjun Zhao @ericzzj.bsky.social · Mar 20

Stable Virtual Camera: Generative View Synthesis with Diffusion Models

Jensen (Jinghao)Zhou, Hang Gao, Vikram Voleti, @adyaman.bsky.social, Chun-Han Yao, @markboss.bsky.social, @philiptorr.bsky.social, Christian Rupprecht, Varun Jampani

arxiv.org/abs/2503.14489

1 3 5

Mark Boss @markboss.bsky.social · Jan 8

Check out the HF demo to test the model: huggingface.co/spaces/stabi.... The model (huggingface.co/stabilityai/...) is also available with code and Comfy Nodes github.com/Stability-AI.... We also have a project page available at spar3d.github.io

Stable Point-Aware 3D - a Hugging Face Space by stabilityai

Discover amazing ML apps made by the community

huggingface.co

9

Mark Boss @markboss.bsky.social · Jan 8

One neat implication is that we can edit the point cloud to fix missing features or wrong scaling. We even created a small gradio component for simple edits in the demo (pypi.org/project/grad...)

An image showcasing editing of the point cloud representation to add a cup to a mug or a tail to a plush toy.

1 9

Mark Boss @markboss.bsky.social · Jan 8

Happy to announce SPAR3D! A fast <1s single image to 3D reconstruction model that combines the best from diffusion and regression models by leveraging a point diffusion module to perform a fast initial point cloud. This aids 3D understanding for the mesh estimation
stability.ai/news/stable-...

Introducing Stable Point Aware 3D: Real-Time Editing and Complete Object Structure Generation — Stability AI

Stable Point Aware 3D (SPAR3D) introduces real-time editing and complete structure generation of a 3D object from a single image in less than a second.

stability.ai

1 18

Mark Boss @markboss.bsky.social · Dec 16

A single procedural modeling system is a huge undertaking when you aim for a high quality level. Take speed tree for example which combines procedural aspects with hand authored elements and it’s an entire company dedicated to that.

Mark Boss @markboss.bsky.social · Dec 16

Yes I agree for certain things it can work. Simple cities (Manhattan style) and natural landscapes are rather well fitting and are explored heavily in video games already. Going for interiors or any object is another beast.

2

Mark Boss @markboss.bsky.social · Dec 16

The realistic rendering is not the problem and even full path tracing scenes is doable for room scale scenes on GPU. It still requires some denoising tho as otherwise rendering times are too long to generate any meaningful amount of data. But even then data is the bottleneck

1

Mark Boss @markboss.bsky.social · Dec 16

It’s hard to scale 3D data similarly to image or video. We run around with capable cameras all the time. Only few people can model 3D and it’s takes time and isn’t offered for free (rightfully). So even if we would pay all artists in the world, we still won’t hit the scale of image and video.

3

Mark Boss @markboss.bsky.social · Dec 8

I recently went with recreating the rooms in Blender. A lot of furniture websites now have 3D viewers and you can download the models from devtools. They are also metric sized. Then blender becomes sims pro and you can iterate quite fast.