Long way to go 😅

This work is joint work with Dan Sattrup Nielsen and Peter Schneider-Kamp.

We share both the code and update the leaderboard with new releases:

🔗 Website: euroeval.com/leaderboards...


📄 Paper: arxiv.org/abs/2406.13469

👩‍💻 GitHub: github.com/EuroEval/Eur...

I especially like our dashboard, which allows the comparison of models of interest across target languages.

It notably includes both high, mid, and highly low-resource languages, which allow examining generalization even in areas where the available training data is minuscule in comparison to English:

This work is joint work with Dan Sattrup Nielsen and Peter Schneider-Kamp

📄 Paper: euroeval.com/leaderboards...
🔗 Website: euroeval.com
👩‍💻 GitHub: github.com/EuroEval/Eur...

One of the features that I really like is the ability to compare specific models of interest across target languages. Here, we show an example of Dutch, English, and German, but you can try out any combination:

euroeval.com/extras/radia...

This notably includes low-resource languages such as Faroese and Icelandic, which are great for checking generalizations to languages in which the available data is minuscule

Find out more or check the leaderboard here:

📑 Paper: arxiv.org/abs/2502.135...
📈 Leaderboard: huggingface.co/spaces/mteb/...
👩‍💻 GitHub: github.com/embeddings-b...

I would especially like to thank the managing team Isaac Chung, @imeneker.bsky.social, Márton Kardos, Roman Solomatin, @tomaarsen.com, Chenghao Xiao, @vaibhavadlakha.bsky.social, @orionweller.bsky.social, Siva Reddy. and @muennighoff.bsky.social, who all have done fantastic work 🙏

This work resulted from a large-scale collaboration, and I would like to thank all of the authors and contributors on MTEB.

This new release also comes with a whole new leaderboard, where it is possible to build benchmarks tailored to your use case using in-depth task selection.

Such an extensive collection of tasks comes with a considerable computational cost. Thus, we have added multiple optimizations to ensure the benchmark is accessible and quick to run. We see notable speedups for the English benchmark while maintaining relative rank.

Examining this claim, we see that the Mistral-derived models indeed perform better in languages on which the models are believed to be trained:

We use this collection of tasks to propose multiple benchmarks for multilingual, code, European and Indic languages, and many more.

We find that smaller multilingual models (~500M) outperform notably larger 7B models, likely due to a limited multilingual pre-training.

Would love to hear more, do you intend to expand existing metadata or utilise if the pretraining?

Desværre ikke