🧵 7/7

📢 Shoutout to my amazing co-authors and to ServiceNow Research and Mila for making this happen! 🚀

📄 Read the full paper: arxiv.org/abs/2502.15210

#PairBench #LLMs #VLMs #GenAI #AutoEval

🧵 6/7

✅ Beyond benchmarking, PairBench can be used during VLM training & fine-tuning to detect biases early and improve evaluation methods!

This could lead to more trustworthy, consistent AI systems for real-world tasks. 🚀

🧵 5/7

✅ PairBench correlates strongly with existing benchmarks, meaning it can serve as a low-cost alternative to expensive human-annotated benchmarks!

This makes it easier to compare and rank models efficiently—without excessive computational costs.

🧵 4/7

Instead of blindly picking a judge model, we should ask:
🔹 What task is being evaluated?
🔹 What metric matters most?

✅ PairBench helps match the right VLM to the right task, improving fairness & reliability in auto-evaluation.

🧵 3/7

🚨 No single VLM is the best! Models vary drastically across PairBench metrics.

Although some align well with human judgements, they may struggle at symmetry, smoothness, or controllability—making their scores unreliable!

📄 More failure cases in our paper’s appendix!

🧵 2/7

✅ Surprising (and concerning) result: Most VLMs lack symmetry! 🤯

In theory, sim(A, B) = sim(B, A)—but in practice? Many models fail!

For example, simply swapping the order of the input images makes GPT-4o and Gemini 1.5 Pro change their decision and scores drastically. 🔄

🧵 1/7

Vision language models (VLMs) are widely used as automated evaluators, but can they actually compare data reliably? 🤔

✅ PairBench systematically tests how well VLMs judge similarity across modalities, revealing key strengths & weaknesses in their decisions.

🚨 Excited to introduce PairBench! 🚨

💡 TL;DR: VLM-judges can fail at data comparison!

✅ PairBench helps you pick the right one by testing alignment, symmetry, smoothness & controllability—ensuring reliable auto-evaluation.

📄 Paper: arxiv.org/abs/2502.15210

🧵 Thread: 👇