Lightnews — Scholar-powered news

Paul Gavrikov @paulgavrikov.bsky.social · 17h

Agree 100%! I think this paper does a great job of outlining issues in the original paper.

Paul Gavrikov @paulgavrikov.bsky.social · 20h

If you think of texture as the material/surface property (which I think is the original perspective), then the ablation in this paper is insufficient to suppress the cue.

1 2

Paul Gavrikov @paulgavrikov.bsky.social · 20h

I really liked the thoroughness of this paper but I’m afraid that the results are building on a shaky definition of „texture“. If you replace texture in the original paper with local details it’s virtually the same finding.

1 2

Paul Gavrikov @paulgavrikov.bsky.social · 1d

lol, a #WACV AC just poorly rephrased the weaknesses I raised in my review as their justification and ignored all other reviews ... I feel bad for the authors ...

Paul Gavrikov @paulgavrikov.bsky.social · 8d

See more details and try out your own model here:
paulgavrikov.github.io/visualoverlo...

VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes

The paper introduces VisualOverload, a new visual question answering (VQA) benchmark designed to test vision-language models (VLMs) on densely populated, detail-rich scenes using public-domain paintin...

paulgavrikov.github.io

Paul Gavrikov @paulgavrikov.bsky.social · 8d

4) Models answer consistently for easy questions ("Is it day?": yes, "Is it night?": no) but fall back to guessing for hard tasks such as reasoning. Concerningly, some models even fall below random chance, hinting at shortcuts.

1

Paul Gavrikov @paulgavrikov.bsky.social · 8d

3) Similar trends for OCR. Our OCR questions contain constraints (e.g., the fifth word) that models often fail to consider. Minor errors include a strong tendency to autocorrect typos or to hallucinate more common spellings, especially in non-Latin/English.

1 1

Paul Gavrikov @paulgavrikov.bsky.social · 8d

2) Models cannot count in dense scenes, and the performance gets worse the larger the number of objects; they typically "undercount" and errors are massive. Here is the distribution over all models:

1

Paul Gavrikov @paulgavrikov.bsky.social · 8d

1) Our benchmark is hard: the best model (o3) achieves an accuracy of 69.5% in total, but only 19.6% on the hardest split. We observe significant performance drops on some tasks.

1

Paul Gavrikov @paulgavrikov.bsky.social · 8d

Our questions are built on top of a fresh dataset of 150 high-resolution and detailed scenes probing core vision skills in 6 categories: counting, OCR, reasoning, activity/attribute/global scene recognition. The ground truth is private, and our eval server is live!

1

Paul Gavrikov @paulgavrikov.bsky.social · 8d

🚨 New paper out!
"VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes"
👉 arxiv.org/abs/2509.25339
We test 37 VLMs on 2,700+ VQA questions about dense scenes.
Findings: even top models fumble badly—<20% on the hardest split and key failure modes in counting, OCR & consistency.

1 2 6

Paul Gavrikov @paulgavrikov.bsky.social · 8d

Paper out now: arxiv.org/abs/2509.25339

VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes

Is basic visual understanding really solved in state-of-the-art VLMs? We present VisualOverload, a slightly different visual question answering (VQA) benchmark comprising 2,720 question-answer pairs, ...

arxiv.org

Paul Gavrikov @paulgavrikov.bsky.social · Sep 8

Joint work with Wei Lin, Jehanzeb Mirza, Soumya Jahagirdar, Muhammad Huzaifa, Sivan Doveh, James Glass, and Hilde Kuehne.

Paul Gavrikov @paulgavrikov.bsky.social · Sep 8

Paper coming soon! In the meantime:
• Try your model: huggingface.co/spaces/paulg...
• Dataset: huggingface.co/datasets/pau...
• Code: github.com/paulgavrikov...

1

Paul Gavrikov @paulgavrikov.bsky.social · Sep 8

🤖 We tested 37 models. Results?
Even top VLMs break down on “easy” tasks in overloaded scenes.

Best model (o3):
• 19.8% accuracy (hardest split)
• 69.5% overall

1

Paul Gavrikov @paulgavrikov.bsky.social · Sep 8

📊 VisualOverload =
• 2,720 Q–A pairs
• 6 vision tasks
• 150 fresh, high-res, royalty-free artworks
• Privately held ground-truth responses

1

Paul Gavrikov @paulgavrikov.bsky.social · Sep 8

Is basic image understanding solved in today’s SOTA VLMs? Not quite.

We present VisualOverload, a VQA benchmark testing simple vision skills (like counting & OCR) in dense scenes. Even the best model (o3) only scores 19.8% on our hardest split.

2 5 16

Reposted by Paul Gavrikov

Keuper Labs @keuper-labs.bsky.social · Jul 8

Congratulations to @paulgavrikov.bsky.social for an excellent PhD defense today!

2 3

Paul Gavrikov @paulgavrikov.bsky.social · May 28

Yesterday, I had the great honor of delivering a talk on feature biases in vision models at the VAL Lab at the Indian Institute of Science (IISc). I covered our ICLR 2025 paper and a few older works in the same realm.
youtu.be/9efpCs1ltcM

Paul Gavrikov - Feature Biases in Vision Models (Research Talk @ IISc, Bengaluru)

YouTube video by Paul Gavrikov

youtu.be

Paul Gavrikov @paulgavrikov.bsky.social · May 3

Paper: arxiv.org/abs/2403.09193
Code: github.com/paulgavrikov...

Paul Gavrikov @paulgavrikov.bsky.social · May 3

It was truly special reconnecting with old friends and making so many new ones. Beyond the conference halls, we had some unforgettable adventures — exploring the city, visiting the woodlands, and singing our hearts out at karaoke nights. 🎤🦁🌳

1

Paul Gavrikov @paulgavrikov.bsky.social · May 3

What an incredible week at #ICLR 2025! 🌟
I had an amazing time presenting our poster "Can We Talk Models Into Seeing the World Differently?" with
@jovitalukasik.bsky.social. Huge thanks to everyone who stopped by — your questions, insights, and conversations made it such a rewarding experience.

1 3

Paul Gavrikov @paulgavrikov.bsky.social · Apr 24

Looking forward to meet you!

1

Paul Gavrikov @paulgavrikov.bsky.social · Apr 24

Today at 3pm - poster #328. See you there!

Paul Gavrikov @paulgavrikov.bsky.social · Apr 21

On Thursday, I'll be presenting our paper "Can We Talk Models Into Seeing the World Differently?" (#328) at ICLR 2025 in Singapore! If you're attending or just around Singapore, l'd love to connect—feel free to reach out! Also, I’m exploring postdoc or industry opportunities–happy to chat!

1 4

Paul Gavrikov @paulgavrikov.bsky.social · Apr 21

On Thursday, I'll be presenting our paper "Can We Talk Models Into Seeing the World Differently?" (#328) at ICLR 2025 in Singapore! If you're attending or just around Singapore, l'd love to connect—feel free to reach out! Also, I’m exploring postdoc or industry opportunities–happy to chat!

1 5 4