Lightnews — Scholar-powered news

Michael Tschannen @mtschannen.bsky.social · Feb 22

HF model collection for transformers:
huggingface.co/collections/...

HF model collection for OpenCLIP and timm:
huggingface.co/collections/...

And of course big_vision checkpoints:
github.com/google-resea...

SigLIP2 - a google Collection

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

1 2

Michael Tschannen @mtschannen.bsky.social · Feb 22

Paper:
arxiv.org/abs/2502.14786

HF blog post from @arig23498.bsky.social et al. with a gentle intro to the training recipe and a demo:
huggingface.co/blog/siglip2

Thread with results overview from Xiaohua (only on X, sorry - these are all in the paper):
x.com/XiaohuaZhai/...

SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

We introduce SigLIP 2, a family of new multilingual vision-language encoders that build on the success of the original SigLIP. In this second iteration, we extend the original image-text training obje...

arxiv.org

1 1 1

Michael Tschannen @mtschannen.bsky.social · Feb 22

📢2⃣ Yesterday we released SigLIP 2!

TL;DR: Improved high-level semantics, localization, dense features, and multilingual capabilities via drop-in replacement for v1.

Bonus: Variants supporting native aspect and variable sequence length.

A thread with interesting resources👇

1 1 12

Reposted by Michael Tschannen

Andreas Steiner @andreaspsteiner.bsky.social · Feb 19

Looking for a small or medium sized VLM? PaliGemma 2 spans more than 150x of compute!

Not sure yet if you want to invest the time 🪄finetuning🪄 on your data? Give it a try with our ready-to-use "mix" checkpoints:

🤗 huggingface.co/blog/paligem...
🎤 developers.googleblog.com/en/introduci...

7 19

Michael Tschannen @mtschannen.bsky.social · Dec 20

Check out our detailed report about *Jet* 🌊 - a simple, transformer-based normalizing flow architecture without bells and whistles.

Jet is an important part of JetFormer's engine ⚙️ As a standalone model it is very tame and behaves predictably (e.g. when scaling it up).

Alexander Kolesnikov @kolesnikov.ch · Dec 20

With some delay, JetFormer's *prequel* paper is finally out on arXiv: a radically simple ViT-based normalizing flow (NF) model that achieves SOTA results in its class.

Jet is one of the key components of JetFormer, deserving a standalone report. Let's unpack: 🧵⬇️

5

Reposted by Michael Tschannen

Ibrahim Alabdulmohsin @ibomohsin.bsky.social · Dec 7

Attending #NeurIPS2024? If you're interested in multimodal systems, building inclusive & culturally aware models, and how fractals relate to LLMs, we've 3 posters for you. I look forward to presenting them on behalf of our GDM team @ Zurich & collaborators. Details below (1/4)

1 5 12

Reposted by Michael Tschannen

Andreas Steiner @andreaspsteiner.bsky.social · Dec 5

🚀🚀PaliGemma 2 is our updated and improved PaliGemma release using the Gemma 2 models and providing new pre-trained checkpoints for the full cross product of {224px,448px,896px} resolutions and {3B,10B,28B} model sizes.

1/7

1 21 68

Michael Tschannen @mtschannen.bsky.social · Dec 3

It’s not, good catch.

Michael Tschannen @mtschannen.bsky.social · Dec 3

Very nice! I knew some soft-token TTS papers, but none so far using AR + normalizing flows. Thanks for sharing!

Michael Tschannen @mtschannen.bsky.social · Dec 3

The noise curriculum guides the (image generation) learning process to first learn high-level, global structure and later low-level structure/texture. Maximum likelihood “tends to focus” mostly on the latter.

1

Reposted by Michael Tschannen

Sander Dieleman @sedielem.bsky.social · Dec 2

In arxiv.org/abs/2303.00848, @dpkingma.bsky.social and @ruiqigao.bsky.social had suggested that noise augmentation could be used to make other likelihood-based models optimise perceptually weighted losses, like diffusion models do. So cool to see this working well in practice!

11 52

Reposted by Michael Tschannen

Alexander Kolesnikov @kolesnikov.ch · Dec 2

I always dreamed of a model that simultaneously

1. optimizes NLL of raw pixel data,
2. generates competitive high-res. natural images,
3. is practical.

But it seemed too good to be true. Until today!

Our new JetFormer model (arxiv.org/abs/2411.19722) ticks on all of these.

🧵

2 5 37

Reposted by Michael Tschannen

André Susano Pinto @asusanopinto.bsky.social · Dec 2

Did you ever try to get an auto-regressive transformer to operate in a continuous latent space which is not fixed ahead of time but learned end to end from scratch?

Enter JetFormer: arxiv.org/abs/2411.19722 -- joint work in a dream team: @mtschannen.bsky.social and @kolesnikov.ch

2 14

Michael Tschannen @mtschannen.bsky.social · Dec 2

Joint work with @asusanopinto.bsky.social and @kolesnikov.ch done @googledeepind.bsky.social.

8/8

1 3

Michael Tschannen @mtschannen.bsky.social · Dec 2

To our knowledge, JetFormer is the first model capable of generating high fidelity images and producing strong log-likelihood bounds.

So far we explored a simple setup (image/text pairs, no post-training), and hope JetFormer inspires more (visual) tokenizer-free models!

7/

1 4

Michael Tschannen @mtschannen.bsky.social · Dec 2

Finally, why getting rid of visual tokenizers/VQ-VAEs?
- They can induce information loss (e.g. small text)
- Removing specialized components was a key driver of recent progress (bitter lesson)
- Raw likelihoods are comparable across models (for hill climbing, scaling laws)

6/

1 3

Michael Tschannen @mtschannen.bsky.social · Dec 2

Importantly, this is simple additive Gaussian noise on the training images (i.e. a data augmentation). JetFormer does neither depend on it (or its parameters), nor is it trained for denoising like diffusion models.

5/

1 3

Michael Tschannen @mtschannen.bsky.social · Dec 2

Learning to generate high-fidelity images with maximum likelihood is tricky. To bias the model towards nicer-looking images we introduce a noise curriculum: Gaussian noise added to the input image and annealed to 0 during training, s.t. high-level details are learned first.

4/

2 4

Michael Tschannen @mtschannen.bsky.social · Dec 2

Conceptually, the normalizing flow serves as both an image encoder for perception tasks and an image decoder for image generation tasks during inference.

We train JetFormer to maximize the likelihood of the multimodal data, without auxiliary losses (perceptual or similar).

3/

1 5

Michael Tschannen @mtschannen.bsky.social · Dec 2

We leverage a normalizing flow (“jet”) to obtain a soft-token image representation that is end-to-end trained with a multimodal transformer for next-token prediction. The soft token distribution is modeled with a GMM à la GIVT.

arxiv.org/abs/2312.02116

2/

GIVT: Generative Infinite-Vocabulary Transformers

We introduce Generative Infinite-Vocabulary Transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose t...

arxiv.org

1 1 5

Michael Tschannen @mtschannen.bsky.social · Dec 2

Have you ever wondered how to train an autoregressive generative transformer on text and raw pixels, without a pretrained visual tokenizer (e.g. VQ-VAE)?

We have been pondering this during summer and developed a new model: JetFormer 🌊🤖

arxiv.org/abs/2411.19722

A thread 👇

1/

4 36 160

Michael Tschannen @mtschannen.bsky.social · Nov 25

Thank you!

1

Michael Tschannen @mtschannen.bsky.social · Nov 24

🙋‍♂️

1 1