rtommccoy.bsky.social
Tom McCoy
@rtommccoy.bsky.social
1.9K followers 320 following 160 posts
Assistant professor at Yale Linguistics. Studying computational linguistics, cognitive science, and AI. He/him.
Posts Media Videos Starter Packs
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · 2d
Beginning a Grand Tour of California!
- Oct 6: Colloquium at Berkeley Linguistics
- Oct 9: Workshop at Google Mountain View
- Oct 14: Talk at UC Irvine Center for Lg, Intelligence & Computation
- Oct 16: NLP / Text-as-Data talk at NYU

Say hi if you'll be around!
4
Reposted by Tom McCoy
begus.bsky.social
Gasper Begus @begus.bsky.social · 2d
Exciting talk in the linguistics department at UC Berkeley tomorrow!
@rtommccoy.bsky.social
1 10
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · 7d
Yes!! An excellent point!!
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · 8d
🤖 🧠 NEW BLOG POST 🧠 🤖

What skills do you need to be a successful researcher?

The list seems long: collaborating, writing, presenting, reviewing, etc

But I argue that many of these skills can be unified under a single overarching ability: theory of mind

rtmccoy.com/posts/theory...
Illustration of the blog post's main argument, summarized as: "Theory of Mind as a Central Skill for Researchers: Research involves many skills.If each skill is viewed separately, each one takes a long time to learn. These skills can instead be connected via theory of mind – the ability to reason about the mental states of others. This allows you to transfer your abilities across areas, making it easier to gain new skills."
2 2 19
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Sep 1
Totally. I think one key question is whether you want to model the whole developmental process or just the end state. If just the end state, LLMs have a lot to offer; but if the whole development (which is what we ultimately should aim for!) there are many issues in how LLMs get there
1 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Sep 1
The conversation that frequently plays out is:

A: "LLMs do lots of compositional things!"
B: "But they also make lots of mistakes!"
A: "But so do humans!"

I don't find that very productive, so would love to see the field move toward more detailed/contentful comparisons.
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Sep 1
They're definitely not fully systematic, so currently it kinda comes down to personal opinion about how systematic is systematic enough. And one thing I would love to see is more systematic head-to-head comparisons of humans and neural networks so that we don't need to rely on intuitions.
1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Sep 1
Yeah, I think that's a good definition! I also believe that some LLM behaviors qualify as this - they routinely generate sentences with a syntactic structure that never appeared in the training set.
1 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 31
"Hello world!" sounds like a word followed by a crossword clue for that word: "Hell = Low world"
3
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 31
And although models still make lots of mistakes on compositionality, that alone also isn't enough because humans do too. So, if we want to make claims about models being human-like or not, what we really need are finer-grained characterizations of what human-like compositionality is.
1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 31
Agreed with these points broadly! But though being less “bad at compositionality” isn’t the same as compositional like humans, it does mean that we can no longer say "models completely fail at compositionality and are thus non human like" (because they no longer completely fail).
1 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 31
I agree that garden paths & agreement attraction could be explained with fairly superficial statistics. For priming, what I had in mind was syntactic priming, which I do think requires some sort of structural abstraction.
1 2
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 31
What would you view as evidence for true productivity?
1 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
Definitely true that LLM-style models can't go gather new data (they're restricted to focusing on a subset of their input), but it doesn't feel outside the spirit of ML to allow the system to seek new data which it then applies statistical learning over, if seeking is also statistically-driven
1 2
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
E.g., in ML, datapoint importance is determined by some inscrutable statistics, while in more nativist approaches it's determined by a desire to build a high-level causal model of the world?
1 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
It feels like a false dichotomy to me? In ML models, some training examples are more influential than others, so you could say an ML model can "decide" to ignore some data. In that sense both model types decide which data to learn from, but they differ in what criteria they use to do so.
1 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
Yes, this is a great point! I do think language (which is the domain I mainly study) gets around these concerns a bit: for language, human children primarily have to rely on being fed data, and that data is symbolic in nature. But I agree these properties don't hold for all cognitive domains!
1 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
In other words, our argument is very much based on the available evidence. New, stricter evidence could very well push the needle back toward needing symbols at the algorithmic level - and that would be exciting if so!
1 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
One key next step, then, is stricter diagnostics of symbolic behavior that go beyond “can humans/models be compositional” into “in what specific ways are we compositional”, “what types of errors are made”, etc., and then comparing humans & models head-to-head

(cont.)
1 2
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
A broader comment: LLMs are definitely far from perfect. But there has been important progress. For a while, we could say “neural nets are so bad at compositionality that they’re obviously different from humans.” I’m no LLM fanboy, but I do think such sweeping arguments no longer apply

(cont.)
2 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
FWIW, the types of productivity that we look at go beyond n-grams; there’s also novelty in syntactic tree structures, and in things like “using a word as the subject of a sentence when the LLM has only ever seen it as the direct object”
2 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
I completely agree that the differences in training/learning between LLMs and humans are a major shortcoming of LLMs as cognitive models - probably the biggest edge that symbolic models have over neural networks. Meta-learning seems promising here, but is still in early stages.
1 2
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
Maybe no one talks about AlphaGo as a cognitive model because there’s no history of research on “here are the particularly informative behavioral quirks that humans show when playing Go”, such that it’s not clear what evidence we would look for to argue a model is playing Go in a human-like way?
1 1
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
E.g., for psycholinguistics, LLMs show garden path effects, agreement attraction, & priming. I would also put compositionality, productivity, & rapid learning in this category of “particularly informative cognitive phenomena.”
2 2
rtommccoy.bsky.social
Tom McCoy @rtommccoy.bsky.social · Aug 30
I completely agree that matching human performance is not all that matters - models should be human-like, not just human-level. That said, certain behaviors are viewed in CogSci as particularly illuminating about the mind, and one exciting thing about LLMs is that they display many such properties
1 1