Lightnews — Scholar-powered news

Aida Nematzadeh @aidanematzadeh.bsky.social · 11d

This result is particularly interesting for capabilities that are harder for models (like numerical reasoning or text rendering) as prompt-aware guidance boosts performance on these failure modes without retraining.

Aida Nematzadeh @aidanematzadeh.bsky.social · 11d

Most diffusion-based models use a fixed (model-tuned) guidance schedule. We show that picking the guidance value during inference, conditioned on the prompt/capability, significantly improves performance.

arxiv.org/abs/2509.16131

1 1 2

Reposted by Aida Nematzadeh

A. Sophia Koepke @askoepke.bsky.social · Jun 11

Our #CVPR2025 workshop on Emergent Visual Abilities and Limits of Foundation Models (EVAL-FoMo) is taking place this afternoon (1-6pm) in room 210.

Workshop schedule: sites.google.com/view/eval-fo...

EVAL-FoMo 2 - Schedule

Date: June 11 (1:00pm - 6:00pm)

sites.google.com

3 3 7

Aida Nematzadeh @aidanematzadeh.bsky.social · Apr 23

Also at #ICLR2025: See this work in action! We're demoing "Gecko" showing how we use capability-based evaluators to help users customize evals & select models. 🦎

Find us at the Google booth, Fri 4/26, 12:00-12:30 PM.

1

Aida Nematzadeh @aidanematzadeh.bsky.social · Apr 23

At #ICLR2025, we're diving into what makes prompt adherence evaluators work for image/video generation.
Check out our poster Friday at 3 PM: iclr.cc/virtual/2025/p… 🦎

https://iclr.cc/virtual/2025/p…

1 1

Aida Nematzadeh @aidanematzadeh.bsky.social · Apr 23

Generative models are powerful evaluators/verifiers, impacting evaluation and post-training. Yet, making them effective, particularly for highly similar models/checkpoints, is challenging. The devil is in the details.

1 1 5

Reposted by Aida Nematzadeh

Dr. Katharine Dickson @metaomicsnerd.bsky.social · Mar 10

I know multiple people who need to hear this piped into their offices during working hours

hospicerocks.bsky.social @hospicerocks.bsky.social · Mar 10

1 10 33

Reposted by Aida Nematzadeh

Jess Hamrick @jhamrick.bsky.social · Mar 6

I wrote about why funding for the NSF and NIH is important to me and to my hometown (Charlottesville, VA) at @cvilletomorrow.bsky.social. Thank you to @sciencehomecoming.bsky.social for inspiring me to do this!

www.cvilletomorrow.org/if-federal-f...

If federal funding for science is cut, we won't just be losing the research

Jessica B. Hamrick is a Virginia success story for her career in science. That career, she writes, wouldn't have been possible without federal funding for science.

www.cvilletomorrow.org

7 21

Reposted by Aida Nematzadeh

A. Sophia Koepke @askoepke.bsky.social · Feb 13

Our 2nd Workshop on Emergent Visual Abilities and Limits of Foundation Models (EVAL-FoMo) is accepting submissions. We are looking forward to talks by our amazing speakers that include @saining.bsky.social, @aidanematzadeh.bsky.social, @lisadunlap.bsky.social, and @yukimasano.bsky.social. #CVPR2025

Ameya P. @bayesiankitten.bsky.social · Feb 12

🔥 #CVPR2025 Submit your cool papers to Workshop on
Emergent Visual Abilities and Limits of Foundation Models 📷📷🧠🚀✨

sites.google.com/view/eval-fo...

Submission Deadline: March 12th!

EVAL-FoMo 2

A Vision workshop on Evaluations and Analysis

sites.google.com

3 7

Reposted by Aida Nematzadeh

Pablo Samuel Castro @pcastr.bsky.social · Jan 21

if you would like to attend #ICLR2025 but have financial barriers, apply for financial assistance!

our priority categories are student authors, and contributors from underrepresented demographic groups & geographic regions.

deadline is march 2nd.

iclr.cc/Conferences/...

ICLR 2025 Financial Assistance

iclr.cc

11 14

Reposted by Aida Nematzadeh

Erin Grant @eringrant.me · Jan 16

Our representational alignment workshop returns to #ICLR2025! Submit your work on how ML/cogsci/neuro systems represent the world & what shapes these representations 💭🧠🤖

w/ @thisismyhat.bsky.social @dotadotadota.bsky.social, @sucholutsky.bsky.social @lukasmut.bsky.social @siddsuresh97.bsky.social

Dota Tianai Dong @dotadotadota.bsky.social · Jan 16

🚨Call for Papers🚨
The Re-Align Workshop is coming back to #ICLR2025

Our CfP is up! Come share your representational alignment work at our interdisciplinary workshop at
@iclr-conf.bsky.social

Deadline is 11:59 pm AOE on Feb 3rd

representational-alignment.github.io

9 20

Aida Nematzadeh @aidanematzadeh.bsky.social · Jan 8

The RE application is now open: boards.greenhouse.io/deepmind/job...

And here is the link to the RS position:
boards.greenhouse.io/deepmind/job...

3

Reposted by Aida Nematzadeh

Thomas Wolf @thomwolf.bsky.social · Jan 6

What was the most impactful/visible/useful release on evaluation in AI in 2024?

2 3 11

Reposted by Aida Nematzadeh

kyunghyuncho.bsky.social @kyunghyuncho.bsky.social · Jan 2

bye, felix

kyunghyuncho.me/bye-felix/

Bye, Felix – Kyunghyun Cho

kyunghyuncho.me

6 93

Reposted by Aida Nematzadeh

Jane Wang @janexwang.bsky.social · Jan 3

A brilliant colleague and wonderful soul Felix Hill recently passed away. This was a shock and in an effort to sort some things out, I wrote them down. Maybe this will help someone else, but at the very least it helped me. Rest in peace, Felix, you will be missed. www.janexwang.com/blog/2025/1/...

Felix — Jane X. Wang

From the moment I heard him give a talk, I knew I wanted to work with Felix . His ideas about generalization and situatedness made explicit thoughts that had been swirling around in my head, incohe...

www.janexwang.com

2 11 63

Reposted by Aida Nematzadeh

Andrew Lampinen @lampinen.bsky.social · Jan 2

Felix Hill was such an incredible mentor — and occasional cold water swimming partner — to me. He's a huge part of why I joined DeepMind and how I've come to approach research. Even a month later, it's still hard to believe he's gone.

Felix Hill and some other DMers and I after cold water swimming at Parliament Hill Lido a few years ago

7 17 120

Reposted by Aida Nematzadeh

Carl T. Bergstrom @carlbergstrom.com · Aug 20

It seems to me that the time is ripe for a Bluesky thread about how—and maybe even why—to befriend crows.

(1/n)

Beautiful crow against a black background

470 2.6K 7.9K

Reposted by Aida Nematzadeh

Sander Dieleman @sedielem.bsky.social · Dec 16

Here's Veo 2, the latest version of our video generation model, as well as a substantial upgrade for Imagen 3 🧑‍🍳🚢

(Did I mention we are hiring on the Generative Media team, btw 👀)

blog.google/technology/g...

State-of-the-art video and image generation with Veo 2 and Imagen 3

We’re rolling out a new, state-of-the-art video model, Veo 2, and updates to Imagen 3. Plus, check out our new experiment, Whisk.

blog.google

17 60

Reposted by Aida Nematzadeh

Sander Dieleman @sedielem.bsky.social · Dec 14

I've been getting a lot of questions about autoregression vs diffusion at #NeurIPS2024 this week! I'm speaking at the adaptive foundation models workshop at 9AM tomorrow (West Hall A), about what happens when we combine modalities and modelling paradigms.
adaptive-foundation-models.org

NeurIPS 2024 Workshop on Adaptive Foundation Models

adaptive-foundation-models.org

2 6 46

Aida Nematzadeh @aidanematzadeh.bsky.social · Dec 9

The released benchmark can also be used as a counting VQA benchmark and evaluation of auto-metrics.

Paper: arxiv.org/abs/2406.14774

Evaluating Numerical Reasoning in Text-to-Image Models

Text-to-image generative models are capable of producing high-quality images that often faithfully depict concepts described using natural language. In this work, we comprehensively evaluate a range o...

arxiv.org

Aida Nematzadeh @aidanematzadeh.bsky.social · Dec 9

We design 3 main tasks with varying degrees of difficulty and evaluate 13 models across different families. Models show rudimentary numerical reasoning skills, limited to small numbers and simple prompt formats; many models are affected by non-numerical prompt manipulations.

1

Aida Nematzadeh @aidanematzadeh.bsky.social · Dec 9

What do text-to-image models know about numbers? Find out in our new paper 🦎 "Evaluating Numerical Reasoning in text-to-image Models" to be presented at #NeurIPS2024 (Wed 4:30-7:30 PM, #5304).

Dataset: github.com/google-deepm... (1386 prompts, 52,721 images, 479,570 annotations)

GitHub - google-deepmind/geckonum_benchmark_t2i: GeckoNum Benchmark for T2I Model Eval.

GeckoNum Benchmark for T2I Model Eval. Contribute to google-deepmind/geckonum_benchmark_t2i development by creating an account on GitHub.

github.com

1 1 6

Reposted by Aida Nematzadeh

Jennifer Hu @ COLM (recruiting PhDs and postdocs!) @jennhu.bsky.social · Dec 7

Stop by our #NeurIPS tutorial on Experimental Design & Analysis for AI Researchers! 📊

neurips.cc/virtual/2024/tutorial/99528

Are you an AI researcher interested in comparing models/methods? Then your conclusions rely on well-designed experiments. We'll cover best practices + case studies. 👇

NeurIPS Tutorial Experimental Design and Analysis for AI ResearchersNeurIPS 2024

neurips.cc

6 13 86

Reposted by Aida Nematzadeh

Marc Lanctot @sharky6000.bsky.social · Dec 6

If you will be at #NeurIPS2024 @neuripsconf.bsky.social and would like to come see our models in action, come say hi 👋 and check out our demo at the GDM booth!

Wednesday, Dec. 11th @ 9:30-10:00.

Lots of other great things to see as well! Check it out: 👇
deepmind.google/discover/blo...

1 6 41

Aida Nematzadeh @aidanematzadeh.bsky.social · Dec 6

I am hiring for RS/RE positions! If you are interested in language-flavored multimodal learning, evaluation, or post-training apply here 🦎 boards.greenhouse.io/deepmind/job...

I will also be #NeurIPS2024 so come say hi! (Please email me to find time to chat)

Research Scientist, Language

London, UK

boards.greenhouse.io

1 7 28