I want to reshare @brandfonbrener.bsky.social's @NeurIPSConf 2024 paper on CoLoR-Filter: A simple yet powerful method for selecting high-quality data for language model pre-training!

With @hlzhang109.bsky.social @schwarzjn.bsky.social @shamkakade.bsky.social

Cool work! Seems highly related to a paper we posted on arxiv a few weeks ago: arxiv.org/abs/2411.12925

I'm also headed to neurips if you want to find a time to chat, let me know

I’m heading to NeurIPS Wednesday through Sunday. DM me if you want to meet up!

Definitely one of my favorites too!

Maybe you just need a bar where everyone keeps their voices down…
maps.app.goo.gl/vdLCiFsUbapp...

NEW: we have an exciting opportunity for a tenure-track professor at the #KempnerInstitute and the John A. Paulson School of Engineering and Applied Sciences (SEAS). Read the full description & apply today: academicpositions.harvard.edu/postings/14362
#ML #AI

Thanks to co-authors Nikhil Anand, Nikhil Vyas, Eran Malach, and Sham Kakade!

Arxiv: arxiv.org/abs/2411.12925
Plots: github.com/KempnerInsti...
Code: github.com/KempnerInsti...
Models: huggingface.co/KempnerInsti...

Loss-to-loss prediction lets us do things like translate a scaling law from only 8 models on a new dataset by leveraging data from a prior run. Full results and more applications in the paper!

Generalizing further, we can predict from test-to-test. These predictions pair up models trained on two different datasets with the same budget and then compare test loss on a third dataset.

Next, we can use a similar methodology to go from train-to-test. These predictions describe how a single function transfers from performance on the training loss to performance on any test loss.

We can fit these curves to sets of 88 models of varying model sizes and dataset sizes. In total we fit over 500 models for these experiments and also release all of the models.

The fits extrapolate well to models with 20x more FLOPs

First, we consider how to translate scaling laws from one dataset to another and from one loss to another.

We find that we can fit a curve to map loss on dataset 0 to loss on dataset 1 for N-parameter models on D tokens (where E_i is the estimated irreducible loss on dataset i)

Check out the paper “Loss-to-loss prediction: scaling laws for all datasets” on arxiv

arxiv.org/abs/2411.12925

Or more details in the thread below!

How does test loss change as we change the training data? And how does this interact with scaling laws?

We propose a methodology to approach these questions by showing that we can predict the performance across datasets and losses with simple shifted power law fits.