Lightnews — Scholar-powered news

Dan Roy

@roydanroy.bsky.social

8.2K followers 550 following 150 posts

Research Director, Founding Faculty, Canada CIFAR AI Chair @VectorInst. Full Prof @UofT - Statistics and Computer Sci. (x-appt) danroy.org I study assumption-free prediction and decision making under uncertainty, with inference emerging from optimality.

Posts Media Videos Starter Packs

Dan Roy @roydanroy.bsky.social · Apr 21

Tian and Karolina and team are at ICLR. Come say hi.

Tian Jin @tjin.bsky.social · Apr 21

📣 The Journey Matters: Our #ICLR2025 paper shows how to pretrain sparse LLMs with half the size of dense LLMs while maintaining quality. We found that the average parameter count during sparse pre-training predicts quality, not final size. An MIT/Rice/Google/ISTA collab 🧵 1/N

1 7

Dan Roy @roydanroy.bsky.social · Apr 7

Curious. Didn’t know meta had a PPL team.

Dan Roy @roydanroy.bsky.social · Apr 7

I like to think about non-reasoning model responses as vibes.

Dan Roy @roydanroy.bsky.social · Apr 7

So who’s read the 2027 article? What do you think?

3 1

Dan Roy @roydanroy.bsky.social · Mar 23

Someone has suggested I check out bsky again. So I'm back looking around here. Notification list is kinda boring. So any good conversations going on? Perhaps about LLM/AI reasoning?

8 34

Dan Roy @roydanroy.bsky.social · Mar 23

Of course.

Dan Roy @roydanroy.bsky.social · Feb 3

Anyone else have the worry that a lot of LLM research is .... just bad psychology?

10 2 69

Dan Roy @roydanroy.bsky.social · Feb 3

And, to achieve the results in this paper, what was the most challenging part? Why had previous attempts fallen short? What was your key new insight?

2 2

Dan Roy @roydanroy.bsky.social · Feb 2

Very interesting. So, what was the biggest hole to fill, in terms of hypotheses?

1 3

Reposted by Dan Roy

Mary Beth Hunzaker @mbhunzaker.bsky.social · Jan 8

Okay, so just a few* thoughts (*this got longer as I wrote 😅….long thread)-

Mary Beth Hunzaker @mbhunzaker.bsky.social · Jan 7

Having a lot of thoughts & feelings today as someone who’s worked both on FBs misinfo interventions (back when they were making investments) and Birdwatch (pre-Musk Community Notes) 💔. Debating if a thread is worth the inevitable headache 😓

1 32 53

Dan Roy @roydanroy.bsky.social · Jan 8

Acknowledgments.

Dan Roy @roydanroy.bsky.social · Jan 8

I got to ski Revelstoke this winter break.

Couple observations: the price of receiving 600 cm of snow by Jan 8 is that it is constantly snowing. Saw almost no sun the whole time and the peak was often in whiteout conditions (though North Bowl was always clear…).

See image for more.

Dan Roy @roydanroy.bsky.social · Jan 8

Multiple friends have likely lost their homes in Los Angeles. Can’t imagine how disorienting this would be. They had only minutes to flee and grab belongings.

Dan Roy @roydanroy.bsky.social · Dec 30

What are the key papers to read?

1 6

Dan Roy @roydanroy.bsky.social · Dec 22

OK. Practical question times. How are you adjusting your research given progress in reasoning style models? Also how are you adjusting the way you work?

11 4 61

Dan Roy @roydanroy.bsky.social · Dec 15

A $100,000,000 experiment is no longer "consequence" free. Ilya is saying "scaling is over", but this may simply be that the scaling "laws" (not laws) are no longer accurate. Also, those laws are tied to hyperparameter tunings.

1 3

Dan Roy @roydanroy.bsky.social · Dec 14

Sure some were empirical. Some were not.

Dan Roy @roydanroy.bsky.social · Dec 14

I'd say no in a sense. Xavier-He initialization was theoretical work. And that was absolutely critical.

1 2

Dan Roy @roydanroy.bsky.social · Dec 13

Pretraining is not done. It's just that theorists haven't told the hackers how to do it better.

Nathan Lambert @natolambert.bsky.social · Dec 13

ILYA: "PRETRAINING IS DONE. WE ARE NOW IN THE POST TRAINING ERA."

3 35

Dan Roy @roydanroy.bsky.social · Dec 13

Annoying. If it could be automatic, sure.

Dan Roy @roydanroy.bsky.social · Dec 12

I'd say wait then.

Dan Roy @roydanroy.bsky.social · Dec 12

That's part of the spec. I don't think this is too problematic. The example they give is problems in NP, where there is a polynomial time checker (i.e., a polytime EV), but generating an instance that passes the checker is hard in the worst case.

Dan Roy @roydanroy.bsky.social · Dec 12

Now that I've had a taste of X without post length limitations, I've got to say that it is quite annoying have to fit tweets into 256 characters here on bsky. On X, when they get to long, they go below the fold, and so you're still incentivized to make it short. Can't we have that here?

9 1 24

Dan Roy @roydanroy.bsky.social · Dec 11

Lottery ticket?

1 3

Dan Roy @roydanroy.bsky.social · Dec 11

@gkdziugaite.bsky.social. Works at GDM and Mila. Influential, technical work.

1 3