Lightnews — Scholar-powered news

Dario @dariogargas.bsky.social · Jun 3

Are you into Chip Design, EDA or just like to do RTL code for fun? Check out the largest benchmarking of LLMs for Verilog generation: TuRTLe 🐢

It includes 40 open LLMs evaluated on 4 benchmarks, following 5 tasks. And its only growing!

huggingface.co/spaces/HPAI-...

arxiv.org/abs/2504.01986

2 2

Dario @dariogargas.bsky.social · May 22

So many healthcare LLMs, and yet so little information! Check out this table summarizing contributions, and find more details in our latest pre-print: arxiv.org/abs/2505.04388

Dario @dariogargas.bsky.social · May 21

Biology needs to be reduced to fundamental components so that mysticism and religion do not corrupt its might.
Same happens to LLMs. These are not thinking machines, in any definition of thinking we may agree on. And the NTP reduction clearly shows that.

1

Dario @dariogargas.bsky.social · May 21

Mostly agree here, though I would rather use the word mimic than persuade. Persuade entails a purpose, which I'm not sure LLMs have. That, is, does a mathematical function have a purpose?

9

Dario @dariogargas.bsky.social · May 21

Exactly! The most effective control measure, RAG, is still a patch that can provide no technical guarantee. Just a strong bias that models may not follow.

The sooner we understand the limits of LLMs, the sooner we'll learn to deploy them properly.

1

Dario @dariogargas.bsky.social · May 21

The Aloe Beta preprint includes full details on data & training setup.
Plus four different evaluation methods (including medical expert).
Plus a risk assessment of healthcare LLMs.

Two years of work condensed in a few pages, figures and tables.

Love open research!
huggingface.co/papers/2505....

Dario @dariogargas.bsky.social · May 6

We just opened two MLOps Engineer positions at @bsc-cns.bsky.social

Our active and young research team needs someone to help sustain and improve our services, including HPC clusters, automated pipelines, artifact managements and much more!

Are you up for the challenge?
www.bsc.es/join-us/job-...

350_25_CS_AIR_RE2

Reference: 350_25_CS_AIR_RE2 Job title: Research Engineer - AI Factory (RE2) About BSC The Barcelona Supercomputing Center - Centro Nacional de Supercomputación (BSC-

www.bsc.es

Dario @dariogargas.bsky.social · May 6

Last week our team presented this at NAACL. Check out the beautiful poster they put together 😍

1

Dario @dariogargas.bsky.social · Apr 11

Though both data sources have the same origin (visual inspection of embryo change) I'd expect features found by humans and features found by a neural net to be complementary.

I guess the intrinsic variance is what dominates here. We can only know so much about an embryo by just looking at it.

Dario @dariogargas.bsky.social · Apr 11

Working on a project for evaluating embryo quality using in-vitro fertilization data.

A random forest using morphokinetic features of embryo evolution visually annotated by experts, and a CNN directly using static images get similar performance. Separately AND together.

I find it surprising...

1

Dario @dariogargas.bsky.social · Apr 9

There are quite a lot of researchers who a so preoccupied with whether or not they could get the funding, they don't stop to think if they should.

Being chased by dinosaurs and writing grants. Same thing.

1

Dario @dariogargas.bsky.social · Apr 4

The recipe is simple 🧑‍🍳 :
1. A good open model 🍞
2. A properly tuned RAG pipeline 🍒

And you will be cooking a five star AI system ⭐ ⭐ ⭐ ⭐ ⭐

See you on the AIAI 2025 conference, where we will be presenting this work, done at @bsc-cns.bsky.social and @hpai.bsky.social

2

Dario @dariogargas.bsky.social · Apr 4

How expensive 🫰 is it to get the best LLM performance? How much cash needs to burn 💸 to get reliable responses? Pareto optimal plots answer these questions.

Our research shows it is economically feasible and scalable to achieve O1 level performance at a fraction of the cost.
buff.ly/ji1VHiV

1 2

Dario @dariogargas.bsky.social · Apr 1

Our LLM safety project, Egida, reached 2K downloads 😀
It includes +60K safety questions expanded with jailbreaking prompts.
The four models trained (and released) show strong signs of safety alignment and generalization capacity. Check out the 🤗 HF page and the paper for details!
buff.ly/kxFVyl2

Dario @dariogargas.bsky.social · Apr 1

Today we release the TuRTLe leaderboard! 🐢

Are you in the Chip Design or EDA business? Wanna know which LLMs are best for the task? By integrating 4 benchmarks, TuRTLe evaluates:

* Syntax
* Functionality
* Synthesizability
* Power, Performance and Area metrics

huggingface.co/spaces/HPAI-...

TuRTLe Leaderboard - a Hugging Face Space by HPAI-BSC

A Unified Evaluation of LLMs for RTL Generation.

huggingface.co

1

Dario @dariogargas.bsky.social · Mar 3

Disclaimer: Only text questions were used to evaluate LLMs, unlike students. Student's score computed under the assumption that all questions were answered, which may not be the case.
buff.ly/3Xa9gFc

Dario @dariogargas.bsky.social · Mar 3

MIR is Spain's medical entrance exam. Best students reach an estimate accuracy of +90. Two or three every year.
We took MIR, '20-'24 to test open LLMs. Llama 3.1 based models, like Aloe, reach +80 in accuracy.
Deepseep R1 reaches +88. Boosted by a RAG system, 92.
buff.ly/4bbbXMw
buff.ly/4hLrhBV

HPAI-BSC/CareQA · Datasets at Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

buff.ly

1

Dario @dariogargas.bsky.social · Feb 28

After listening to the latest @fallofcivspod.bsky.social episode about the Mongolian Empire, by @paulcooper34.bsky.social , I realized Mongols and the Fremen from Dune share remarkable similarities.
Skilled warriors adapted to harsh environments, taking over a society they don't want to adopt.

1 1

Dario @dariogargas.bsky.social · Feb 23

I like the cell one 👍

I'm an empiricist, so we attack metrics by developing adversarial benchmarks that expose model shortcuts. Plus, its a lot of fun to show how fragile LLMs can be.

1

Dario @dariogargas.bsky.social · Feb 23

While writing a paper I consistently learn general insights that are too general or not tested enough to be sold as paper contributions, but are great for conversation :)

1 1

Dario @dariogargas.bsky.social · Feb 21

Human evaluation of LLMs is close to saturation. Models have been optimized so much for plausibility, that we are unable to tell good from bad. Only experts in expert domains can see a meaningful difference.

Dario @dariogargas.bsky.social · Feb 21

After a year working on LLM evaluation, our benchmarking paper is finally out (to be presented at NAACL 2025). Main lessons:
* All LLM evals are wrong, some are slightly useful.
* Goodhart's law. All the time. Everywhere.
* Do lots of different evals and hope for the best.

Automatic Evaluation of Healthcare LLMs Beyond Question-Answering

Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor. Close-ended measurements evaluate the factuality…

buff.ly

1 2

Dario @dariogargas.bsky.social · Feb 21

Evaluating LLMs is a bit like paleontology. Trying to understand the behavior of very complex entities by observing only noisy and partial evidence. How do paleontologists deal with the uncertainty and frustration? Do they also feel like doing alchemy instead of science?

Dario @dariogargas.bsky.social · Feb 19

Remarkable effort. Questionable motivation.

Dario @dariogargas.bsky.social · Feb 19

Wisdom from my 6y old daughter: "A king is a just person disguised as king."