andimara.bsky.social
Andi
@andimara.bsky.social
790 followers 130 following 49 posts
Multimodal research @huggingface
Posts Media Videos Starter Packs
andimara.bsky.social
Andi @andimara.bsky.social · May 21
Read the blog: huggingface.co/blog/nanovlm
nanoVLM: The simplest repository to train your VLM in pure PyTorch
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
2
andimara.bsky.social
Andi @andimara.bsky.social · May 21
Train your Vision-Language Model in just two commands:
> git clone github.com/huggingface/...
> python train.py
GitHub - huggingface/nanoVLM: The simplest, fastest repository for training/finetuning small-sized VLMs.
The simplest, fastest repository for training/finetuning small-sized VLMs. - huggingface/nanoVLM
github.com
1 2
andimara.bsky.social
Andi @andimara.bsky.social · May 21
New Blog📖✨:
nanoVLM: The simplest way to train your own Vision-Language Model in pure PyTorch explained step-by-step!
Easy to read, even easier to use. Train your first VLM today!
1 3
andimara.bsky.social
Andi @andimara.bsky.social · May 14
Link: webml-community-smolvlm-realtime-webgpu.static.hf.space/index.html
Camera Interaction App
webml-community-smolvlm-realtime-webgpu.static.hf.space
1 1
andimara.bsky.social
Andi @andimara.bsky.social · May 14
Real-time SmolVLM in a web browser with transformers.js.

All running locally with no installs. Just open the website.
1 1 3
andimara.bsky.social
Andi @andimara.bsky.social · Apr 8
If you’re into efficient multimodal models, you’ll love this one.
Check out the paper: huggingface.co/papers/2504....
Paper page - SmolVLM: Redefining small and efficient multimodal models
Join the discussion on this paper page
huggingface.co
1
andimara.bsky.social
Andi @andimara.bsky.social · Apr 8
📱 Real-world Efficiency: We've created an app using SmolVLM on an iPhone 15 and got real-time inference directly from its camera!
🌐 Browser-based Inference? Yep! We get lightning-fast inference speeds of 40-80 tokens per second directly in a web browser. No tricks, just compact, efficient models!
1 2
andimara.bsky.social
Andi @andimara.bsky.social · Apr 8
🌟 State-of-the-Art Performance, SmolVLM comes in three powerful yet compact sizes—256M, 500M, and 2.2B parameters—each setting new SOTA benchmarks for their hardware constraints in image and video understanding.
1
andimara.bsky.social
Andi @andimara.bsky.social · Apr 8
✨ Less CoT, more efficiency: Turns out, too much Chain-of-Thought (CoT) data actually hurts performance in small models. They dumb
✨ Longer videos, better results: Increasing video length during training enhanced performance on both video and image tasks.
1 2
andimara.bsky.social
Andi @andimara.bsky.social · Apr 8
✨ System prompts and special tokens are key: Introducing system prompts and dedicated media intro/outro tokens significantly boosted our compact VLM’s performance—especially for video tasks.
1
andimara.bsky.social
Andi @andimara.bsky.social · Apr 8
✨ Pixel shuffling magic: Aggressively pixel shuffling helped our compact VLMs "see" better, same performance with sequences 16x shorter!
✨ Learned positional tokens FTW: For compact models, learned positional tokens significantly outperform raw text tokens, enhancing efficiency and accuracy.
1 2
andimara.bsky.social
Andi @andimara.bsky.social · Apr 8
✨ Smaller is smarter with SigLIP: Surprise! Smaller LLMs didn't benefit from the usual large SigLIP (400M). Instead, we use the 80M base SigLIP that performs equally well at just 20% of the original size!
1 1
andimara.bsky.social
Andi @andimara.bsky.social · Apr 8
Here are the coolest insights from our experiments:
✨ Longer context = Big wins: Increasing the context length from 2K to 16K gave our tiny VLMs a 60% performance boost!
1 1
andimara.bsky.social
Andi @andimara.bsky.social · Apr 8
Today, we share the tech report for SmolVLM: Redefining small and efficient multimodal models.
🔥 Explaining how to create a tiny 256M VLM that uses less than 1GB of RAM and outperforms our 80B models from 18 months ago!
huggingface.co/papers/2504....
Paper page - SmolVLM: Redefining small and efficient multimodal models
Join the discussion on this paper page
huggingface.co
2 8 22
andimara.bsky.social
Andi @andimara.bsky.social · Mar 17
SmolDocling is available today 🏗️

🔗 Model: huggingface.co/ds4sd/SmolDo...
📖 Paper: huggingface.co/papers/2503....
🤗 Space: huggingface.co/spaces/ds4sd...

Try it and let us know what you think! 💬
ds4sd/SmolDocling-256M-preview · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
6
andimara.bsky.social
Andi @andimara.bsky.social · Mar 17
At only 256M parameters, SmolDocling outperforms much larger models on key document conversion tasks:
🖋️ Full-page transcription: Beats models 27× bigger!
📑 Equations: Matches or beats leading models like GOT
💻 Code recognition: We introduce the first benchmark for code OCR
1 1
andimara.bsky.social
Andi @andimara.bsky.social · Mar 17
What makes it unique?
📌 Handles everything a document has: tables, charts, code, equations, lists, and more
📌 Works beyond scientific papers—supports business docs, patents, and forms
📌 It runs with less than 1GB of RAM, so running at large batch sizes is super cheap!
1 1
andimara.bsky.social
Andi @andimara.bsky.social · Mar 17
How does SmolDocling beat models 27× bigger? SmolDocling transforms any document into structured metadata with DocTags, being SOTA in:

✅ Full-page conversion
✅ Layout identification
✅ Equations, tables, charts, plots, code OCR
1 1
andimara.bsky.social
Andi @andimara.bsky.social · Mar 17
🚀 We just dropped SmolDocling: a 256M open-source vision LM for complete document OCR! 📄✨
Lightning fast, process a page in 0.35 sec on consumer GPU using < 500MB VRAM ⚡
SOTA in document conversion, beating every competing model we tested (including models 27x more params) 🤯
But how? 🧶⬇️
3 10 35
andimara.bsky.social
Andi @andimara.bsky.social · Mar 5
Extremely bullish on @CohereForAI's Aya Vision (8B & 32B) - new SOTA open-weight VLMs

- 8B wins up to 81% of the time in its class, better than Gemini Flash
- 32B beats Llama 3.2 90B!
- Integrated on @hf.co from Day 0!

Check out their blog! huggingface.co/blog/aya-vis...
A Deepdive into Aya Vision: Advancing the Frontier of Multilingual Multimodality
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
1
andimara.bsky.social
Andi @andimara.bsky.social · Jan 31
Me too! Highlight of my career so far :)
1 1
andimara.bsky.social
Andi @andimara.bsky.social · Jan 31
And that was why we didn't release this before. It's live research code. Most gets rewritten fairly often, and some parts have been the same for years.
It works, it manages to produce SOTA results at 256M and 80B sizes, but it's not production code.
Go check it out:
github.com/huggingface/...
smollm/vision at main · huggingface/smollm
Everything about the SmolLM2 and SmolVLM family of models - huggingface/smollm
github.com
1 1 3
andimara.bsky.social
Andi @andimara.bsky.social · Jan 31
And it also has a bunch of bugs like this one in our modeling_vllama3.py file. We start from a pretrained LLM, but for some reason the weights of the head are not loaded from the model. I still don't know why :(
2 3
andimara.bsky.social
Andi @andimara.bsky.social · Jan 31
The codebase is full of interesting insights like this one in our dataset.py file: How do you get reproducible randomness in different processes across different machines?
Start different random number generators based on a tuple (seed, rank)!
2 3
andimara.bsky.social
Andi @andimara.bsky.social · Jan 31
Post training, you can run the evaluation on all of these tasks by running:
sbatch vision/experiments/evaluation/vloom/async_evals_tr_346/run_evals_0_shots_val_2048 . slurm
1 3