🎯 These patterns hold across all 16 models - despite huge differences (encoder/decoder, 100M->8B params, instruction-tuning)

Despite rapid advances since BERT, certain aspects of how LMs process language remain remarkably consistent💡

Paper: arxiv.org/abs/2506.02132
Code: github.com/ml5885/model...

We also tested how tokenization affects linguistic representations using analogy tasks (king - man + woman = ?) 👑

Whole-word embeddings consistently outperform averaged subtoken representations - linguistic regularities are stored at the word level, not compositionally!

🔬 We also measured intrinsic dimensionality across layers using PCA.

🎢 Some models (GPT-2, OLMo-2) compress their middle layers to just 1-2 dimensions capturing 50-99% of variance, then expand again! This bottleneck aligns with where grammar is most accessible & lexical info is most nonlinear.

To understand when these patterns emerge, we analyze OLMo-2 & Pythia checkpoints throughout pre-training. 👶👦👨👨‍🦳

We find that models learn this linguistic organization in the first few thousand steps! But this encoding slowly degrades as training progresses. 📉

🤔 But are classifiers actually learning linguistic patterns or just memorizing?

📈 We ran control tasks with random labels - inflection classifiers show high selectivity (real learning!) while lemma classifiers don't (memorization).

Key findings 📊:
- 📉 Lexical info concentrates in early layers & becomes increasingly nonlinear in deeper layers
- ✨ Inflection (grammar) stays linearly accessible throughout ALL layers
- Models memorize word identity but learn generalizable patterns for inflections!

🧐 How do modern LMs encode linguistic information? Do they represent words grouped by meaning (walk/walked) or grammar (walked/jumped)?

We trained classifiers on hidden activations from 16 models (BERT -> Llama 3.1) to find out how they store word identity (lexemes) vs. grammar (inflections).

🚨New #interpretability paper with @nsubramani23.bsky.social: 🕵️ Model Internal Sleuthing: Finding Lexical Identity and Inflectional Morphology in Modern Language Models