In collaboration with @ema-ridopoco.bsky.social Tommaso Carraro @paolomorettin.bsky.social @emilevankrieken.com @nolovedeeplearning.bsky.social @looselycorrect.bsky.social @andreapasserini.bsky.social

Want to know more?

1️⃣ Learn more about RSs: Why they appear, their root causes, and mitigation: arxiv.org/abs/2305.19951

2️⃣ Make NeSy models aware of their shortcuts: arxiv.org/abs/2402.12240

For other details regarding rsbench, datasets, and experiments, check the links below:

Website: unitn-sml.github.io/rsbench/
Paper: openreview.net/forum?id=5Vt...
GitHub: github.com/unitn-sml/rs...

Easy to set up and use!

1️⃣ Configurable: can be easily configured with YAML/JSON files.
2️⃣ Intuitive: straightforward to use:

📊 8 challenging tasks, all with predefined settings.

3 new benchmarks:
🔢 MNMath for arithmetic reasoning
🛃 MNLogic for SAT-like problems
🚖 SDD-OIA, a synthetic self-driving task!

They can all be made easier or harder with our data generator!

🧪 Test your models!

- 🌍 Evaluate concepts in in- and out-of-distribution scenarios.
- 🎯 Ground-truth concept annotations are available for all tasks.
- 📊 Visualize how your models handle different learning & reasoning tasks!

🔍 rsbench allows you to:

- 🧮 Run algorithmic, logical, and high-stakes tasks w/ known reasoning shortcuts (RSs).
- 📊 Eval concept quality via F1, accuracy & concept collapse.
- 🛠️ Easily customize the tasks and count RSs a priori using our countrss tool!

🤔 What are reasoning shortcuts?

NeSy models might learn wrong concepts but still make perfect predictions!

Example: A self-driving car 🚗 stops in front of a 🚦🔴 or a 🚶. Even if it confuses the two, it outputs the right prediction!

🌐 rsbench allows you to evaluate the concepts learned by:

1️⃣ Neuro-Symbolic models (#NeSy)
2️⃣ Concept Bottleneck Models (#CBMs)
3️⃣ Black-box Neural Networks (NNs*)
4️⃣ Vision-Language Models (#VLMs*)

* through post-hoc concept-based explanations (e.g., TCAV)

📣 Does your model learn high-quality #concepts, or does it learn a #shortcut?

Test it with our #NeurIPS2024 dataset & benchmark track paper!

rsbench: A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts

What's the deal with rsbench? 🧵

by @ema-ridopoco.bsky.social @looselycorrect.bsky.social @andreapasserini.bsky.social @samubortolotti.bsky.social

eg

👉 proceedings.neurips.cc/paper_files/...

👉 openreview.net/forum?id=pDc...

👉 unitn-sml.github.io/rsbench/