another plausible argument (to me) is that llms can learn higher-level skills without lower-level skills because evolution did not kill off llms with missing foundational skills.

if this is your model of intelligence, then missing the ability to copy strings, or do multi-digit addition implies that LLMs cannot reach superhuman intelligence because they lack the foundation to do so

why doesn't it bother me? I think a popular mental model of intelligence is building skills on top of each other like a pyramid. through evolution or schooling, humans learned lots of new skills, but each skill depended on previous building blocks.

🏖️ For semantic segmentation, we can suppress specific concepts like "sand" across the entire image. The model then predicts the next most likely class (like "earth" or "water") while leaving other parts of the scene unchanged:

🐦 In this bird classification example, when we suppress the "spotted" feature (technically mottling) on this bird's breast and neck, the ViT switches from predicting "Canada Warbler" to "Wilson Warbler", a similar bird species but without necklace pattern:

So we built interactive demos where you can suppress specific features and watch model predictions change.

osu-nlp-group.github.io/SAE-V/#demos

See below for examples of what you can do.

The difference between CLIP and visual-only models like DINOv2 is striking. CLIP forms country-specific visual representations, while DINOv2 doesn't see these cultural connections. Here are examples from a USA feature and a Brazil feature.

What's actually different between CLIP and DINOv2? CLIP knows what "Brazil" looks like: Rio's skyline, sidewalk patterns, and soccer jerseys.

We mapped 24,576 visual features in vision models using sparse autoencoders, revealing surprising differences in what they understand.