Lightnews — Scholar-powered news

Peng Qi @qi2peng2.bsky.social · Jul 25

How do we prove that #AI can't do #maths?

Real Mathematics (yes, "real" is a pun here):

a+b+c = (a+b)+c = a+(b+c)

AI Mathematics (well, floating point maths, really):

>>> 0.1+0.2+0.3
0.6000000000000001
>>> 0.1+(0.2+0.3)
0.6

QED.

Peng Qi @qi2peng2.bsky.social · Jul 16

This project was joint work with my Amazon colleagues (led by Yumo Xu), and it's great to see it finally published. Hope this helps motivate more careful eval work in the near future!

#AI #agent #evaluation #RAG #NLP

1 1

Peng Qi @qi2peng2.bsky.social · Jul 16

b) as builders, we evaluate the technology soberly and help users navigate these risks in product design.

Want to learn more? Checkout
Our paper: arxiv.org/pdf/2506.01829
Open-source code: github.com/amazon-scien...

1

Peng Qi @qi2peng2.bsky.social · Jul 16

Why should you care? As businesses / individuals leverage AI more and more to speed up research and decision-making, it is important that, a) as users, we examine the tools we are using to understand their limitations and avoid pitfalls with significant potential downsides, and

1

Peng Qi @qi2peng2.bsky.social · Jul 16

With a new, carefully annotated dataset and an automated evaluation metric we designed, we find that although LLMs are reasonably good at citing accurate sources most of the time, SOTA LLMs still cite incorrectly 5-28% of the time, and miss citations anywhere from 16% to an alarming 95% of the time.

1 1

Peng Qi @qi2peng2.bsky.social · Jul 16

"𝐂𝐢𝐭𝐞𝐄𝐯𝐚𝐥: 𝐏𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞-𝐃𝐫𝐢𝐯𝐞𝐧 𝐂𝐢𝐭𝐚𝐭𝐢𝐨𝐧 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐟𝐨𝐫 𝐒𝐨𝐮𝐫𝐜𝐞 𝐀𝐭𝐭𝐫𝐢𝐛𝐮𝐭𝐢𝐨𝐧", we propose a framework to systematically study citation accuracy by considering previously neglected contexts such as user-provided information and LLMs' parametric knowledge.

1

Peng Qi @qi2peng2.bsky.social · Jul 16

As 🔎 AI deep research agents 🔎 become an essential part of many people's day-to-day work, it is more essential than ever before that we can trust what they produce.

When these agents cite sources they claim the report is based on, how much can we actually trust them? In our new #ACL2025 paper, ...

1

Peng Qi @qi2peng2.bsky.social · Jul 2

AI agents research, and share a skeleton of a research proposal if I ever find the time / need to write one from my perspective today: qipeng.me/blog/stop-us...

Why You Should Stop Using HotpotQA for AI Agents Evaluation in 2025 | Peng Qi

We published HotpotQA, a groundbreaking multi-step question answering dataset in 2018, which has since motivated and facilitated numerous AI agent research works. But you should probably reconsider…

qipeng.me

Peng Qi @qi2peng2.bsky.social · Jul 2

a good problem in its historical context, some of my own research attempts at solving this problem that I believe are on the critical path to autonomous agents, and what's changed today to make the dataset less relevant in its original form. I also reflect on the possible paths forward for ...

Why You Should Stop Using HotpotQA for AI Agents Evaluation in 2025 | Peng Qi

We published HotpotQA, a groundbreaking multi-step question answering dataset in 2018, which has since motivated and facilitated numerous AI agent research works. But you should probably reconsider…

qipeng.me

1 1

Peng Qi @qi2peng2.bsky.social · Jul 2

Seven years ago, I co-led a paper called 𝗛𝗼𝘁𝗽𝗼𝘁𝗤𝗔 that has motivated and facilitated many #AI #Agents research works since. Today, I'm asking that you stop using HotpotQA blindly for agents research in 2025 and beyond.

In my new blog post, I revisit the brief history of 𝗛𝗼𝘁𝗽𝗼𝘁𝗤𝗔, why it defined ...

Why You Should Stop Using HotpotQA for AI Agents Evaluation in 2025 | Peng Qi

We published HotpotQA, a groundbreaking multi-step question answering dataset in 2018, which has since motivated and facilitated numerous AI agent research works. But you should probably reconsider…

qipeng.me

1

Peng Qi @qi2peng2.bsky.social · May 22

The longer employers don’t acknowledge and embrace this discrepancy, the faster they lose the top candidates they spent enormous efforts to hire and retain, leaving the organization in self-fulfilling mediocrity.

Peng Qi @qi2peng2.bsky.social · May 22

but not perfectly aligned with those of the employer. They ask: Will I get the opportunity to build a career beyond what is immediately required of me? Will I learn and grow, be part of a great team and culture? Will I make a name for myself while doing great work? Will I remain competitive?
3/

1

Peng Qi @qi2peng2.bsky.social · May 22

We are never settling for a candidate that does exactly the thing that needs to be done right now, since that thing itself can change before you know it.

But too often employers and managers forget, that highly motivated and capable candidates also hold expectations parallel to these, ...
2/

1

Peng Qi @qi2peng2.bsky.social · May 22

When making great hashtag#hiring decisions, we often look for growth potential in a candidate. Will they rise to the occasion when unforeseen challenges arise? Will they grow in the role, and lift up others in the team? Will they still be able to contribute if business direction changes?
1/

1