Lightnews — Scholar-powered news

Donatella Genovese

@donatellag.bsky.social

60 followers 170 following 7 posts

PhD Student | Works on Explainable AI | https://donatellagenovese.github.io/

donatellagenovese.github.io

Posts Media Videos Starter Packs

Reposted by Donatella Genovese

Pasquale Minervini @neuralnoise.com · Mar 13

Please share it within your circles! edin.ac/3DDQK1o

9 14

Donatella Genovese @donatellag.bsky.social · Feb 28

Really cool paper by @kayoyin.bsky.social about interpretability of In Context Learning, they found that Function Vectors (FV) heads are crucial for few-shot ICL.
www.arxiv.org/abs/2502.14010

1 2

Donatella Genovese @donatellag.bsky.social · Feb 21

A really nice resource for understanding how to parallelize LLM training.

Thomas Wolf @thomwolf.bsky.social · Feb 19

After 6+ months in the making and over a year of GPU compute, we're excited to release the "Ultra-Scale Playbook": hf.co/spaces/nanot...

A book to learn all about 5D parallelism, ZeRO, CUDA kernels, how/why overlap compute & coms with theory, motivation, interactive plots and 4000+ experiments!

The Ultra-Scale Playbook - a Hugging Face Space by nanotron

The ultimate guide to training LLM on large GPU Clusters

hf.co

Donatella Genovese @donatellag.bsky.social · Feb 14

3/ Interleaving Concepts with Token Embeddings

🔹 Predicted concepts are compressed into a continuous vector 🎯
🔹 They are then inserted into hidden states alongside token embeddings 🧩

Donatella Genovese @donatellag.bsky.social · Feb 14

2/ Training the Model with Dual Objectives

🔹 Next-token prediction – the standard LLM training objective.
🔹 Concept prediction – the model learns to reproduce extracted concepts from its hidden state.

Donatella Genovese @donatellag.bsky.social · Feb 14

1/ Concept Extraction with SAE

🔹 A Sparse Autoencoder (SAE) extracts high-level concepts from the hidden states of a pretrained LLM.
🔹 Only the most important concepts are selected based on their attribution score (impact on model output).

Donatella Genovese @donatellag.bsky.social · Feb 14

🚀 Meta’s new LLM pretraining framework predicts concepts and integrates them into its hidden state to enhance next-token prediction. 🚀

It achieves the same performance with 21.5% fewer tokens and better generalization! 🎯

📝: arxiv.org/abs/2502.08524

3 1

Donatella Genovese @donatellag.bsky.social · Feb 9

A very interesting work that explores the possibility of having a unified interpretation across multiple models

Harry Thasarathan @hthasarathan.bsky.social · Feb 7

🌌🛰️🔭Wanna know which features are universal vs unique in your models and how to find them? Excited to share our preprint: "Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment"!

arxiv.org/abs/2502.03714

(1/9)

1 2

Reposted by Donatella Genovese

Simone Scardapane @sscardapane.bsky.social · Jan 10

*MoE Graph Transformers for Interpretable Particle Collision Detection*
by @alessiodevoto.bsky.social @sgiagu.bsky.social et al.

We propose a MoE graph transformer for particle collision analysis, with many nice interpretability insights (e.g., expert specialization).

arxiv.org/abs/2501.03432

4 12