Somewhat frustrated yesterday to once again read ACL paper which did all sorts of complex things (including the usual results tables showing best approach) on garbage data. With minimal ack of this in limitations. Most fundamental rule of CS is Garbage In, Garbage Out
Really interesting paper on real-world evaluation in IR. I should learn more about eval in IR, its not something Ive ever properly looked at dl.acm.org/doi/10.1145/...
Several people have asked me recently if I will still be able to contribute to research projects after I retire in summer 2026. Absolutely! I will have emeritus statius, and am very hapy to remain involved in research projects at Aberdeen amd elsewhere.
I am often asked about my experience blogging, sometimes by people who are considering writing their own blog. In this “meta” blog, I summarise my thoughts and experiences about my blog.
New blog: Defining hallucination is not straightforward
Many researchers assume that hallucination is a binary feature; either something is a hallucination or it is not. This is too simplistic. I describe some of the issues I have seen below.
At ACL, I engaged with 50 papers (went to oral, talked to poster person). Decided (looked at paper sometimes), that 3 of these robust, interesting, relevant to me; 2 of these 3 won awards. Hum, maybe in future I should focus on 40 award papers, ignore the other 3000?
Excited by recent positive evaluations of NLG apps developed by my students to encourage safer driving in UK and Nigeria. We see stat sig reductions in unsafe driving incidents in both countries.
Last week I had to deal with two cases of papers containing hallucinated references. This is not acceptable! Shows complete disdain for understand prev work, and suggests rest of paper may be fabricated.
Ok to use LLM to suggest related work, but read (or at least skim) them!
Watched recording of ACL panel on generalisability (recommended to me). I share concerns about "LLM popcorn", but my biggest concern about NLP is lack of research diversity. Everyone does LLM, few people do impact or qual eval, little interest in genuine collab with other fields
The academic world has changed since I got my PhD in 1990. One of the worst changes is that researchers now often pay thousands of pounds to publish their work. Unfair to researchers with limited funding, and bad for science.
Very interesting meta-analysis of human-AI collab. Shows more effective in content creation (eg report writing) than in decision making, which does not surprise me
When combinations of humans and AI are useful: A systematic review and meta-analysis
Excited to announce the first-ever Workshop for Young Researchers in Natural Language Generation (YNLG), supported by @siggen.bsky.social, taking place on October 29, 2025 in Hanoi, Vietnam, co-located with INLG 2025. Call for Submissions is out now!
I got great feedback from recent paper and talk on eval impact, and summarise some of the suggested papers (including more examples of impact eval) and insightful comments (eg, about eval “ecosystem”) I received.
I'll be at ACL next week (Tue-Thur, not Sun/Mon). Look forward to meeting old friends and new people who want to connect! Ill also be giving an invited talk on impact evaluation at the GEM workshop on Thur 31 July
Really happy that this survey of NLP in cancer care, from my student Mengxuan Sun , has finally appeared (its been a saga). One key but depressing finding is that evaluation quality is uniformly dreadful by medical standards; NLP researchers just dont seem to care...
Motivated by recent discussion with my group: Ignore subjective statements such as "I find LLMs to be incredibly useful for XX", especially when made by people (such as AI companies or gurus) who have strong biases/incentives/COI .
We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers.
The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.
Good point, in some cases I have struggled to convince companies to publish. But in other cases we could publish. I guess depends on the company and the people who make this decision, and also on what is being published (eg very hard to publish negative result about company's product!)
Ive written a "Last Word" opinion piece for CL about evaluating real-world impact. It * looks at how impact can be evaluated * shows via a structured survey that perhaps 0.1% of ACL Anth papers measure real-world impact * discusses why this is the case
