Lightnews — Scholar-powered news

Reposted by Ravid Shwartz Ziv

NYU Center for Data Science @nyudatascience.bsky.social · Mar 10

CDS Research Scientist Ravid Shwartz-Ziv (@shwartzzzivravid.bsky.social) recently provided expert analysis on DeepSeek's latest AI developments in TechCrunch.

techcrunch.com/2025/01/30/h...

How DeepSeek changed Silicon Valley's AI landscape | TechCrunch

Chinese AI lab DeepSeek provoked the first Silicon Valley freak-out of 2025. Here's what it could mean for American AI policy.

techcrunch.com

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Feb 12

I have an awesome idea that no one had tried before - RL on math datasets 🤯
You will have a natural verifier!

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Jan 31

Now that the ICML deadline is over, well done to all students! And next time, please, please, please don't wait for the last moment, I'm too old for that... 🙏

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Jan 23

Go read our paper about lazy layers!

NYU Center for Data Science @nyudatascience.bsky.social · Jan 22

CDS Research Scientist @shwartzzzivravid.bsky.social and collaborators at UT Austin found "lazy layers" in deep LLMs where attention patterns degenerate.

Their solution: Inheritune - a way to create smaller but equally capable models by inheriting layers from larger ones.

arxiv.org/abs/2404.08634

Inheritune: Training Smaller Yet More Attentive Language Models

Large Language Models (LLMs) have achieved remarkable performance across various natural language processing tasks, primarily due to the transformer architecture and its self-attention mechanism. Howe...

arxiv.org

1 3

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Jan 14

Check out our paper for detailed experiments and explanations on how we're making AI systems more reliable by helping them better express their uncertainty!

Thank you to Tal Zeevi (who did all the work!) @yann-lecun.bsky.social , Stain Lawrence and John Onofrey
The Paper - arxiv.org/abs/2412.07169

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Jan 14

The results? In medical imaging, Rate-In maintains sharp uncertainty estimates around critical anatomical boundaries, while traditional methods get fuzzy. We demonstrate superior performance across different noise levels and benchmarks!

1 2

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Jan 14

Rate-In's approach: We dynamically adjust dropout rates by measuring information loss in each layer. Where features are critical, we preserve more; where they're redundant, we drop more. Like adaptive noise, guided by information theory!

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Jan 14

So, how do we make AI express uncertainty during inference without special training?

Current uncertainty prediction methods (like Monte Carlo Dropout) use fixed dropout rates everywhere. They don't adapt to specific images or tasks - it's a one-size-fits-all approach!

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Jan 14

Imagine you're a doctor looking at an MRI scan. Would you rather have an AI that:
A) Says "There's a tumor" with blind confidence
B) Points out exactly which areas it's uncertain about, helping focus your expertise.

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Jan 14

A new paper 🥳🥳🥳
We present "Rate-In" - a technique that helps neural networks better express their uncertainty during inference, which is especially crucial for medical applications!
with Tal Zeevi, @yann-lecun.bsky.social , H. Stain Lawrence and John Onofrey

1 7

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Jan 6

Apparently, I reached 4000 citations 🤓
Thank you all my collaborators! 🎉
In 5K, I will give my secret to amazing papers titles 😎

5

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 18

Having lunch with @ylecun.bsky.social. Talking about cool science ideas
Suddenly, a phone call from school: Your kid doesn't feel good
Me: I can't come
School: He feels bad
Me: Ok, coming right away
The kid when I arrive: I feel great!
Me: 😡You are cute, but he invented CovNet and I-JEPA!

2

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 13

Conference hack: Pitch your ideas to brilliant minds. Most of the time, they'll break it, but if they're (really!) nice, they'll help you fix it 🤯

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 11

I'm on a flight from NYC to Vancouver. There are so many researchers on the plane that if it crashes, the AGI will be postponed for at least 10 years...

3

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 10

I have 6 hours flight. Hit me up with the recent papers that I must read...

3

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 9

UNBELIEVABLE Apple Customer Service: My 6-month-old Mac just died! At the store, they demand my Apple password (not even the computer login!!) but I can't remember it.
Their brilliant solution? recover it with my DEAD laptop🤦 Or wait 3 DAYS for a request. Then 5 days for repair! Such great service 😡

1 3

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 4

Want to help organize something similar? Let me know! (We have all the materials - notebooks/datasets ready, so it shouldn't be too much work)
Thanks to everyone who helped, especially
@cbbruss.bsky.social , Will Calandra and
@ylecun.bsky.social

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 4

It was incredible seeing them think through problems together and try different approaches I would never think about. They were creative and fast (except for LLM training 🕧). I have no doubt they'll take progress in the field to the next level and change the world.

1 1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 4

It was fantastic - beyond NYU's administration, the students were amazing.
I may sound old (I'm old!), but today's students are much smarter than in my time! They have great approaches and know how to learn and solve problems quickly.

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 4

Teams tackled identical challenges using either LLMs (what is LLM? a great question!) or classical ML algorithms while tracking metrics like performance, memory usage, and compute time over time🧐

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 4

We had a hackathon yesterday at NYU "Beyond the Hype" where participants solved problems with and without LLMs to analyze which problems are better suited for LLM solutions in real-world environments 🧑‍💼

1 1 3

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Dec 3

Hi! I'll be at NeurIPS next week (Wednesday - Friday) and would love to meet! You can DM or email me if you'd like to grab a coffee and talk. If we haven't talked before, please share a bit about yourself

2

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Nov 29

This is such a cool project and I hope to see more like that 😱

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Nov 29

They tricked Freysa by:
Creating a fake "new admin session."
Redefining what "approveTransfer" meant
Convincing it that receiving money REQUIRED using approveTransfer
The result was that $47K was transferred to p0pular.eth

1

Ravid Shwartz Ziv @shwartzzzivravid.bsky.social · Nov 29

By attempt 482, the prize was $50K, and each try cost $450. Then someone cracked it with genius social engineering:

1