Lightnews — Scholar-powered news

Ali Modarressi @amodarressi.bsky.social · Jul 20

Details on poster times and locations coming soon.

Would love to meet and chat ☕️💬

If you’re attending #ACL2025, feel free to stop by and say hi! 👋
🧵[4/4]

Ali Modarressi @amodarressi.bsky.social · Jul 20

⏱️🔎 Time Course MechInterp
We track how factual knowledge forms in OLMo over training by analyzing the evolving roles of Attention Heads and FFNs.
Heads are dynamic and often repurposed; FFNs are stable and keep refining facts.
By: A. Dawar Hakimi
arxiv.org/abs/2506.03434
🧵[3/4]

Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Models

Understanding how large language models (LLMs) acquire and store factual knowledge is crucial for enhancing their interpretability and reliability. In this work, we analyze the evolution of factual kn...

arxiv.org

1

Ali Modarressi @amodarressi.bsky.social · Jul 20

🌐 MEXA: Multilingual Evaluation of English-Centric LLMs

A method for assessing the multilingual capabilities of English-centric LLMs using parallel sentences. It estimates how many languages an LLM covers and at what level.

By: @kargaranamir.bsky.social

x.com/amir_nlp/sta...
🧵[2/4]

Amir H. Kargaran on X: "Excited to introduce MEXA, a method for assessing the multilingual capabilities of English-centric LLMs using parallel sentences. It estimates how many languages an LLM covers and at what level. Paper: https://t.co/awRq0Y4SCl Code: https://t.co/M3UVh2F9J1 https://t.co/xBOQ1DJmWx" / X

Excited to introduce MEXA, a method for assessing the multilingual capabilities of English-centric LLMs using parallel sentences. It estimates how many languages an LLM covers and at what level. Paper: https://t.co/awRq0Y4SCl Code: https://t.co/M3UVh2F9J1 https://t.co/xBOQ1DJmWx

x.com

1 1

Ali Modarressi @amodarressi.bsky.social · Jul 20

Leaving Vancouver after ICML’s closing fireworks 😁🎆

Heading to Toronto for a few days, then off to
@aclmeeting.bsky.social to present:

"Collapse of Dense Retrievers"
A work by @mohsen-fayyaz.bsky.social that I was fortunate to collaborate on.

Also co-presenting two other papers…🧵 [1/4]

1

Ali Modarressi @amodarressi.bsky.social · Jul 9

Full NoLiMa post thread (X / Twitter): x.com/AModarressi/...

Ali Modarressi on X: "🚀 Introducing NoLiMa Paper 🚀 Most long-context benchmarks have literal overlaps between the questions and the context—but what if they didn’t? 🤔 Turns out, it’s a tough challenge! Powerful models like GPT-4o performance drops from 99.3% to 69.7% at 32K context length. 📉 https://t.co/Fo3YsGCBsi" / X

🚀 Introducing NoLiMa Paper 🚀 Most long-context benchmarks have literal overlaps between the questions and the context—but what if they didn’t? 🤔 Turns out, it’s a tough challenge! Powerful models like GPT-4o performance drops from 99.3% to 69.7% at 32K context length. 📉 https://t.co/Fo3YsGCBsi

x.com

Ali Modarressi @amodarressi.bsky.social · Jul 9

Check out the paper & our GitHub repo (with results on recent models 🆕✨)!
📄: arxiv.org/abs/2502.05167
🔗: github.com/adobe-resear...
🤗: huggingface.co/datasets/amo...
This work was my internship project at
@adobe.com, in collaboration with my mentors there and Hinrich Schütze.

NoLiMa: Long-Context Evaluation Beyond Literal Matching

Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens. A popular method for evaluating these capabilities is the needle-in-a-haystack (NIAH) test, which involves ret...

arxiv.org

1 1

Ali Modarressi @amodarressi.bsky.social · Jul 9

I’ll be at @icmlconf.bsky.social next week presenting NoLiMa!
Poster on Tue July 15, 4:30–7pm (E-2312).

Happy to grab a coffee and chat about long-context, memory, research, or just to catch up.

I’ll be in Toronto for a couple of days after the conference, let me know if you’re around!

1 2 4

Reposted by Ali Modarressi

TMLR Published Papers @tmlr-pub.bsky.social · May 19

MemLLM: Finetuning LLMs to Use Explicit Read-Write Memory

Ali Modarressi, Abdullatif Köksal, Ayyoob Imani, Mohsen Fayyaz, Hinrich Schuetze

Action editor: Greg Durrett

https://openreview.net/forum?id=dghM7sOudh

#memory #memorizing #memllm

1 2

Ali Modarressi @amodarressi.bsky.social · May 17

The takeaway? we need robust retrievers that prioritize answer relevance, not just heuristic shortcuts.

work with an amazing team:
@mohsen-fayyaz.bsky.social,
Hinrich Schütze,
@violetpeng.bsky.social

paper: arxiv.org/abs/2503.05037
dataset 🤗: t.co/QZFyCLqP0P

Cross-post from x.com/mohsen_fayyaz

Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence

Dense retrieval models are commonly used in Information Retrieval (IR) applications, such as Retrieval-Augmented Generation (RAG). Since they often serve as the first step in these systems, their robu...

arxiv.org

3

Ali Modarressi @amodarressi.bsky.social · May 17

We also analyze RAG: biased retrievers can mislead LLMs, degrading their performance by 34%, worse than retrieving nothing! 😮

1 1

Ali Modarressi @amodarressi.bsky.social · May 17

When multiple biases combine, retrievers fail catastrophically:
📉 Answer-containing docs ranked <3% of the time over a synthetic biased doc with no answer!

1 1

Ali Modarressi @amodarressi.bsky.social · May 17

Dense retrievers are crucial for RAG and search, but do they actually retrieve useful evidence? 🤔
We design controlled experiments by repurposing a relation extraction dataset, exposing serious flaws in models like Dragon+ and Contriever.

1 2

Ali Modarressi @amodarressi.bsky.social · May 17

📄 Collapse of Dense Retrievers

Accepted to #ACL2025 main conference 🎉🎉

In this paper we uncover major vulnerabilities in dense retrievers like Contriever, showing they favor:
📌 Shorter docs
📌 Early positions
📌 Repeated entities
📌 Literal matches
...all while ignoring the answer's presence!

1 2 9