Lightnews — Scholar-powered news

Reposted by Antoine Chaffin

LightOn @lightonai.bsky.social · Apr 30

✔️ Supporting enterprise-scale document processing
✔️ Enabling more accurate retrieval for AI-generated responses

Kudos to @nohtow.bsky.social for this new SOTA achievement!

🔗 Read the full blog article: www.lighton.ai/lighton-blog...

LightOn Releases GTE-ModernColBERT, First State-of-the-Art Late-Interaction Model Trained on PyLate! - LightOn

LightOn is proud to announce the release of GTE-ModernColBERT, our new state-of-the-art, open-source, multi-vector retrieval model. By leveraging ModernBERT architecture and our innovative PyLate libr...

www.lighton.ai

1 5

Reposted by Antoine Chaffin

Tom Aarsen @tomaarsen.com · Apr 30

ColBERT (a.k.a. multi-vector, late-interaction) models are extremely strong search models, often outperforming dense embedding models. And @lightonai.bsky.social just released a new state-of-the-art one: GTE-ModernColBERT-v1!

Details in 🧵

1 2 12

Reposted by Antoine Chaffin

Tom Aarsen @tomaarsen.com · Apr 30

I'm a big fan of the PyLate project for ColBERT models, and I'm glad to see these strong models coming out. Very nice work by the @lightonai.bsky.social folks, especially @nohtow.bsky.social.

Learn more about PyLate here: lightonai.github.io/pylate/

pylate

Neural Search

lightonai.github.io

1 6

Antoine Chaffin @nohtow.bsky.social · Apr 30

As per usual, thanks to my dear co-maintainer @raphaelsty.bsky.social for helping me make PyLate what it is 🫶

1

Antoine Chaffin @nohtow.bsky.social · Apr 30

In addition to knowledge distillation, we recently added features to allow large-scale contrastive pre-training, and this model has been released upon popular demand, but we are currently doing heavier training so stay tuned!

1 1

Antoine Chaffin @nohtow.bsky.social · Apr 30

PyLate makes downstream usage easy, but also facilitate training!
You can reproduce this SOTA training with <80 LoC and 2 hours of training and it'll run NanoBEIR during training, report it to W&B and create an informative model card!
Link to the gist: gist.github.com/NohTow/3030f...

1

Antoine Chaffin @nohtow.bsky.social · Apr 30

Besides, it also comes with the 8k context window of ModernBERT, which is very useful given that late interaction models generalize very well to longer context as highlighted in the ModernBERT paper
It is thus very suited to handle your very long documents!

1

Antoine Chaffin @nohtow.bsky.social · Apr 30

It is also the first model to outperform ColBERT-small on BEIR
While it is bigger, it is still a very lightweight model and benefits from the efficiency of ModernBERT!
Also, it has only been trained on MS MARCO (for late interaction) and should thus generalize pretty well!

1

Antoine Chaffin @nohtow.bsky.social · Apr 30

Model link: huggingface.co/lightonai/GT...
GTE-ModernColBERT is trained on top of the GTE-ColBERT model using knowledge distillation on the MS MARCO dataset and is the first SOTA model trained using PyLate!
Get started with PyLate using the documentation:
lightonai.github.io/pylate/

lightonai/GTE-ModernColBERT-v1 · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

1

Antoine Chaffin @nohtow.bsky.social · Apr 30

Among all those LLM releases, here is an important retrieval release:
To overcome limitations of awesome ModernBERT-based dense models, today @lightonai.bsky.social is releasing GTE-ModernColBERT, the very first state-of-the-art late-interaction (multi-vectors) model trained using PyLate 🚀

1 6

Reposted by Antoine Chaffin

LightOn @lightonai.bsky.social · Jan 14

ModernBERT-embed-large is released under Apache 2.0 and is available on Hugging Face:
huggingface.co/lightonai/mo...

Congrats to @nohtow.bsky.social for this great work!

lightonai/modernbert-embed-large · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

2 10

Antoine Chaffin @nohtow.bsky.social · Jan 14

When I saw the release of ModernBERT-embed during the holidays, I knew I had to build the large variant, so I wanted to thank Zach Nussbaum from Nomic AI for building and sharing it (as well as all the nomic-embed tools and data) and bearing with me during the training!

2

Antoine Chaffin @nohtow.bsky.social · Jan 14

ModernBERT-embed-large not only enables usage of ModernBERT-large out-of-the-box, but it should also be a very good starting point for strong fine-tunings on various tasks, so I can't wait to see what the community will build on top of it!

1 1

Antoine Chaffin @nohtow.bsky.social · Jan 14

Obviously, it comes at a slightly higher cost, but it is also trained with Matryoshka capabilities to reduce the footprint of embeddings
Notably, the performance with dimension 256 is only slightly worse than the base version with full dimension 768

1

Antoine Chaffin @nohtow.bsky.social · Jan 14

Model link: huggingface.co/lightonai/mo...
ModernBERT-embed-large is trained using the same (two-stage training) recipe as its smaller sibling and expectedly increases the performance, reaching +1.22 in MTEB average

1

Antoine Chaffin @nohtow.bsky.social · Jan 14

ModernBERT-embed-base is awesome because it allows to use ModernBERT-base for various tasks out-of-the-box
But the large variant of ModernBERT is also awesome...
So today, @lightonai.bsky.social is releasing ModernBERT-embed-large, the larger and more capable iteration of ModernBERT-embed!

1 2 12

Antoine Chaffin @nohtow.bsky.social · Jan 14

The multilingual version is not planned yet, there have been a work on adapting ModernBERT to other languages:
www.linkedin.com/posts/fremyc...

In the mean time, you could have a shot with mGTE (using xformers) or recent language-specific iteration of BERT such as CamemBERTv2!

François REMY on LinkedIn: Fast and multilingual embedding models are the key to many of the world's…

Fast and multilingual embedding models are the key to many of the world's RAG pipelines. Today marks my first step into the realm of ModernBERT, and certainly…

www.linkedin.com

2

Reposted by Antoine Chaffin

LightOn @lightonai.bsky.social · Dec 19

Today, LightOn releases ModernBERT, a SOTA model for retrieval and classification.

This work was performed in collaboration with Answer.ai and the model was trained on Orange Business Cloud Avenue infrastructure.

www.lighton.ai/lighton-blog...

1 4 9

Reposted by Antoine Chaffin

Benjamin Warner @benjaminwarner.dev · Dec 22

This week we released ModernBERT, the first encoder to reach SOTA on most common benchmarks across language understanding, retrieval, and code, while running twice as fast as DeBERTaV3 on short context and three times faster than NomicBERT & GTE on long context.

2 15 73

Reposted by Antoine Chaffin

Calvin McCarter @calvinmccarter.bsky.social · Dec 20

When one evaluates log-likelihood of a sequence of length L via the chain rule of probability, the first term has missingness fraction of 1, the second has missingness of (L-1)/L, etc. So the inference-time masking rate is ~ Uniform[0, 1].

2 1 1

Antoine Chaffin @nohtow.bsky.social · Dec 20

Oh, I see!
Having also worked a lot on causal models, I never thought of this kind of modelling because I always opposed MLM to open ended generation
I guess with papers such as this one arxiv.org/pdf/2406.04823, I should more!
Very interesting perspective, thanks!

arxiv.org

1

Antoine Chaffin @nohtow.bsky.social · Dec 20

Could you elaborate?
Or give me pointers?
Is it because having a fixed value bias the learning w.r.t the way we will sample downstream? (Like not mask 30% of the target?)

1

Antoine Chaffin @nohtow.bsky.social · Dec 19

But there is definitely some diggings to be made to find an optimal strategy in this regard
To me the logic would be to ramp up to have a kick off signal and make it harder and harder but papers seems to say otherwise
Maybe random is the optimal solution!

1 2

Antoine Chaffin @nohtow.bsky.social · Dec 19

Not really, we considered ramping up/down the masking ratio, but the findings from the literature (at least what we read at time) seemed counter-intuitive/not consensual
We ended up not really digging much into this particular aspect, again because we had so much to explore

1 2

Antoine Chaffin @nohtow.bsky.social · Dec 19

Definitely!
Again, the original goal of the project (besides cool models) was to convince some researchers to spend a bit of their GPUs hours on encoders pre-training again!

Hopefully we nailed it and will have the answers to a lot of questions in the future!

1