Thank you again to all my collaborators for their contributions and thoughtful feedback.

Madeline Hughes
Akshaya Thoutam
Anay Gupta
Andrew Navia
@nfusi.bsky.social
Srivatsan Raghavan
Peter Winter
@avapamini.bsky.social
@lcrawford.bsky.social

I appreciate any feedback!

Revised Manuscript:
www.biorxiv.org/content/10.1...

Interestingly, we saw improved zero-shot performance when increasing model size (but still no data scaling) for both scVI and Geneformer

The Nicheformer authors observed a similar phenomenon, that when Nicheformer was pre-trained on 1% of their 110M cell dataset performance did not decrease dramatically:

There is an implicit assumption that scaling the pre-training dataset size is inherently better, but the only demonstrated scaling law we know of is in terms of data quality:
arxiv.org/abs/2503.02726

This work addresses a critical consideration in training large-scale models: the size and diversity of the pre-training corpus.

And for out-of-distribution perturbation response prediction.

We also observed similar results for zero-shot batch integration.

The learning saturation points were always 25% or less when evaluating the models on zero-shot classification and were always 10% or less when evaluating the models on fine-tuned classification.

To assess the extent to which this plateauing generalized across datasets and tasks, we identified the "learning saturation point" for each model. This is the minimum pre-training dataset size for which a model surpassed 95% of the maximum performance observed.

Across all model architectures, model performance at cell type classification (both zero-shot and fine-tuned) plateaued at a small fraction of the total pre-training dataset size, regardless of dataset diversity. When fine-tuning, pre-training has almost no impact on performance.

We assessed five model architectures pre-trained to perform as single-cell foundation models (scFMs) in the context of single-cell RNA-seq: PCA, scVI, SSL, Geneformer, and SCimilarity. We pre-trained these models on subsets of the scTab corpus using three downsampling schemes.

In our expanded analysis, we show that single-cell foundation models tend to plateau in downstream task performance with pre-training subsets that are a small fraction of the size of current pre-training datasets.

Thank you to all my collaborators for their contributions and thoughtful feedback.

Madeline Hughes
Akshaya Thoutam
@anaygupta.bsky.social
Andrew Navia
@nfusi.bsky.social
Srivatsan Raghavan
Peter Winter
@avapamini.bsky.social
@lcrawford.bsky.social

I welcome any comments!

Code: github.com/microsoft/sc...
Paper: www.biorxiv.org/content/10.1...