Lightnews — Scholar-powered news

Hokin

@hokin.bsky.social

38 followers 66 following 53 posts

Philosopher, Scientist, Engineer https://hokindeng.github.io/

hokindeng.github.io

Posts Media Videos Starter Packs

Hokin @hokin.bsky.social · Sep 8

I would love to review more papers. Could you send me some invites at times?

Hokin @hokin.bsky.social · Aug 20

I totally agree with you!

Reposted by Hokin

Tobias Gerstenberg @tobigerstenberg.bsky.social · Aug 1

Josh Tenenbaum's inspiring keynote at #cogsci2025 on growing vs scaling AI, the big questions of cognitive science, and the many open questions for the field.

Reposted by Hokin

Jake Quilty-Dunn @quiltydunn.bsky.social · Aug 2

this was a hilarious final slide that just hung up there during q&a

David Barner @drbarner.bsky.social · Aug 2

Susan Carey mic-drop at #cogsci2025. "There are no innate concepts: Discuss"

Reposted by Hokin

David Barner @drbarner.bsky.social · Aug 2

Susan Carey mic-drop at #cogsci2025. "There are no innate concepts: Discuss"

Hokin @hokin.bsky.social · Jul 22

I had two Private Equity internships in freshman & sophomore summers and a BCG PTA internship in my junior year. But it's the same thing for people who went straight into a research lab since the first semester of their college.

Hokin @hokin.bsky.social · Jun 30

🚀 Dive in 👇

🌐 Project: williamium3000.github.io/core-knowledge/
📄 Paper: arxiv.org/abs/2410.10855
📝 OpenReview: openreview.net/forum?id=EIK6xxIoCB
📊 Dataset: huggingface.co/datasets/williamium/CoreCognition
👨‍💻 Code: tinyurl.com/4ak2ryat
🐣 Twitter: x.com/DengHokin/status/1939207699910058405

Hokin @hokin.bsky.social · Jun 30

🙌 Super grateful to @fredashi.bsky.social, @marstin.bsky.social @tomerullman.bsky.social @siyuansong.bsky.social @gershbrain.bsky.social @zoryzhang.bsky.social @tonyfeng.bsky.social, and everyone @growai.bsky.social for all the 🔥 discussions and wisdom 🧠

Hokin @hokin.bsky.social · Jun 30

💥 Huge thanks to my amazing collaborators @williamium3000.bsky.social, Kaia Gao, Icy Wang, Tianwei Zhao, Haoran Sun, Zoey Lyu, Robert Hawkins, Nuno Vasconcelos, @talgolanneuro.bsky.social & @carrot0817.bsky.social 💐

Hokin @hokin.bsky.social · Jun 30

7) The result is astonishing 😱 No models are able to get both the manipulation tasks and the control tasks right at the same time, which seems to completely trivial to humans ...

This suggests MLLMs completely lack core knowledge 🦜 and purely rely on shortcuts 🤫 ... (11/n)

Hokin @hokin.bsky.social · Jun 30

6) Last, we introduce "Concept Hacking" to reveal core knowledge deficiencies in the control experiment set-up.

Concept Hacking systematically manipulates the task-relevant features while preserving all task-irrelevant conditions ... (10/n)

Hokin @hokin.bsky.social · Jun 30

5) Does Reasoning Help 🤔 We further compared the reasoning-augmented models and their corresponding instruction-tuned counterparts.

Scaling test-time compute doesn't seem to be a solution 😮‍💨 ... (9/n)

Hokin @hokin.bsky.social · Jun 30

4) 🧐 Would Core Knowledge Emerge from Pure Scaling? ‼️‼️‼️Nope‼️‼️‼️

Regressions on performance of 230 models with diff. parameters & data sizes, we quantify the scaling effects.

The observation is very intriguing: "higher-level" abilities seem to be more "scalable" than "lower-level" abilities (8/n)

Hokin @hokin.bsky.social · Jun 30

3) Performance on core cognition abilities serves as a reliable predictor for achieving top results on high-level benchmarks.

Concretely, we compute the correlation matrices of MLLMs abilities on our dataset and another 26 public benchmarks and 9 higher-level abilities defined by SEED-Bench (7/n)

Hokin @hokin.bsky.social · Jun 30

2) The topology of internal cognitive representation of MLLMs is akin to human, suggesting "cognition" as a natural kind.

Like humans, physical vs. intention understanding are orthogonal, while tool-use and mechanics co-emerge.

🤔 "Artificial model organisms" for "cognition lesion study" ? 🤔 (6/n)

Hokin @hokin.bsky.social · Jun 30

‼️ RESULTS ‼️ First, MLLMs exhibit a reversed developmental trajectory 📉 compared to humans 📈: they excel at "high-level" tasks that we learn later in life 🙀 but struggle with "basic" ones that we develop in infancy 👶

This observation is statistically significant (5/n) 📊

Hokin @hokin.bsky.social · Jun 30

Living in the AI world, we sample from classic experiments built as physical setups or simulations & convert stimuli to 4 VQA formats (<img>, <vid>, <multi w/wo interleave>) for inference on 230 MLLMs. Evals use exact, in, template, LLM, and hybrid-match w/ filters to ensure method integrity (4/n)

Hokin @hokin.bsky.social · Jun 30

For example, we have integrated Elizabeth Spelke and Renée Baillargeon's object permanence tasks, Susan Carey's core knowledge framework, Josh Tenenbaum and Michael McCloskey's intuitive physics tasks, Mary Hegarty's mechanical reasoning tasks, among many others ... (3/n)

Hokin @hokin.bsky.social · Jun 30

Acknowledging Piaget's shortcomings and efforts of coming generations of brilliant scholars and researchers, our CoreCognition framework aims to be an integration of 100 years of human cognitive developmental research with a backbone of the Piagetian 4-stage theory of cognitive development ... (2/n)

Hokin @hokin.bsky.social · Jun 30

#CoreCognition #LLM #multimodal #GrowAI We spent 3 years to curate 1503 classic experiments spanning 12 core concepts in human cognitive development and evaluated on 230 MLLMs with 11 different prompts for 5 times to get over 3.8 millions inference data points.

A thread (1/n) - #ICML2025 ✅

Hokin @hokin.bsky.social · Jun 14

🤣🤣🤣

Hokin @hokin.bsky.social · Jun 13

2 years of efforts and mighty @zoryzhang.bsky.social finally bring it to the finish-line ❗️

Also huge thanks for @tomerullman.bsky.social advices, very inspiring for us to design the control experiments and dimensions we looked at 💐 💐

🏄 Kudos to to awesome people at GrowAI @growai.bsky.social

Hokin @hokin.bsky.social · Jun 13

What is the best testbed for human-like cognition in machines?🤔 Hardly anything comes better than gaze understanding.

Humans are extremely good at gaze reading 👁️ , and this ability develops extremely early in childhood.

Here, we tested 111 VLMs and 65 humans ⬇️

Zory Zhang @zoryzhang.bsky.social · Jun 12

👁️ 𝐂𝐚𝐧 𝐕𝐢𝐬𝐢𝐨𝐧 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬 (𝐕𝐋𝐌𝐬) 𝐈𝐧𝐟𝐞𝐫 𝐇𝐮𝐦𝐚𝐧 𝐆𝐚𝐳𝐞 𝐃𝐢𝐫𝐞𝐜𝐭𝐢𝐨𝐧?
Knowing where someone looks is key to a Theory of Mind. We test 111 VLMs and 65 humans to compare their inferences.
Project page: grow-ai-like-a-child.github.io/gaze/
🧵1/11

Reposted by Hokin

Paul Linton @lintonvision.bsky.social · Jun 11

Beautiful to see this initiative from a group of like minded PhD students collaborating together! 🚀

Hokin @hokin.bsky.social · Jun 11

New Paper Alert ‼️ Current VLMs completely fail human gaze understanding 🙀 and scaling does NO help ‼️

However, humans, since an extremely age 🧒, are extremely sensitive to other people's gaze 🙄 👀

No mentors, no labs, only pre-doc students, 111 VLMs, and we did it 😎

Hokin @hokin.bsky.social · Jun 11

Thank you for your very nice words, Paul!