Justin DuJardin
@justindujardin.com
I like open source, games, anime, and solving problems with machine learning.
Reposted by Justin DuJardin
They find that RoPE (the positional encoding used in most modern LLMs) has a fundamental flaw. It entangles "what" (content) and "where" (position) information.
They propose PoPE (Polar Coordinate Position Embeddings), which eliminates the what-where.
They propose PoPE (Polar Coordinate Position Embeddings), which eliminates the what-where.
December 26, 2025 at 2:27 AM
They find that RoPE (the positional encoding used in most modern LLMs) has a fundamental flaw. It entangles "what" (content) and "where" (position) information.
They propose PoPE (Polar Coordinate Position Embeddings), which eliminates the what-where.
They propose PoPE (Polar Coordinate Position Embeddings), which eliminates the what-where.
Reposted by Justin DuJardin
Physical Intelligence has a recipe for real world RL on top of VLAs and it looks impressive: www.pi.website/blog/pistar06
A VLA that Learns from Experience
A method for training our generalist policies with RL to improve success rate and throughput on real-world tasks.
www.pi.website
November 18, 2025 at 12:41 AM
Physical Intelligence has a recipe for real world RL on top of VLAs and it looks impressive: www.pi.website/blog/pistar06
Reposted by Justin DuJardin
Sif - watercolour and acryla gouache
November 13, 2025 at 4:02 PM
Sif - watercolour and acryla gouache
get you a partner who loves you like claude loves hasattr
get you a partner who loves you like claude loves adding inline styles
October 31, 2025 at 4:13 PM
get you a partner who loves you like claude loves hasattr
Reposted by Justin DuJardin
Reposted by Justin DuJardin
Reposted by Justin DuJardin
Reposted by Justin DuJardin
We got a public domain Pinocchio Dark Souls game so it only stands to reason that a public domain Winnie the Pooh one would also work and I just want to suggest the boss names Pontiff Piglet and Holy Blade Roo
September 23, 2025 at 3:30 AM
We got a public domain Pinocchio Dark Souls game so it only stands to reason that a public domain Winnie the Pooh one would also work and I just want to suggest the boss names Pontiff Piglet and Holy Blade Roo
I love this work. Started training and evaluating all my models deterministically a few years back when I realized I lacked the compute/time to be effective with nondeterministic models. You pay a perf cost, but never wonder if changing that hparam made it better or if it was just nondeterminism.
Defeating Nondeterminism in LLM Inference by Horace He (Thinking Machines' blog, an AI lab founded by Mira Murati)
Connectionism will cover topics as varied as their research is: from kernel numerics to prompt engineering. Here, they share what they are working on.
Connectionism will cover topics as varied as their research is: from kernel numerics to prompt engineering. Here, they share what they are working on.
Defeating Nondeterminism in LLM Inference
Reproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language models.
For example, you might observe that asking ChatGPT the...
thinkingmachines.ai
September 10, 2025 at 10:32 PM
I love this work. Started training and evaluating all my models deterministically a few years back when I realized I lacked the compute/time to be effective with nondeterministic models. You pay a perf cost, but never wonder if changing that hparam made it better or if it was just nondeterminism.
I had to say goodbye to the best cat yesterday. His name was MorTon. He was very sweet, handsome, brave, and definitely a fierce hunter. I'll miss him bunches. Please enjoy his floof 😿
September 10, 2025 at 5:13 PM
I had to say goodbye to the best cat yesterday. His name was MorTon. He was very sweet, handsome, brave, and definitely a fierce hunter. I'll miss him bunches. Please enjoy his floof 😿
working on a new project and just finished the classic developer journey
- research the best way to do x
- decide your system is "special" and reject it
- fight through days of debugging and refactoring
- "independently" arrive at research conclusion
- research the best way to do x
- decide your system is "special" and reject it
- fight through days of debugging and refactoring
- "independently" arrive at research conclusion
a cartoon of robin hood leaning against a tree with the word nbd above him
ALT: a cartoon of robin hood leaning against a tree with the word nbd above him
media.tenor.com
August 11, 2025 at 3:47 PM
working on a new project and just finished the classic developer journey
- research the best way to do x
- decide your system is "special" and reject it
- fight through days of debugging and refactoring
- "independently" arrive at research conclusion
- research the best way to do x
- decide your system is "special" and reject it
- fight through days of debugging and refactoring
- "independently" arrive at research conclusion
Reposted by Justin DuJardin
What is "lossing"? (with audio)
August 2, 2025 at 5:16 PM
What is "lossing"? (with audio)
Really reflecting on what it means to be professional and act ethically this morning..
a cute cartoon cat is sitting down and looking at the camera .
ALT: a cute cartoon cat is sitting down and looking at the camera .
media.tenor.com
July 24, 2025 at 6:06 PM
Really reflecting on what it means to be professional and act ethically this morning..
I shared some of my findings recently.
BlueSky: *crickets*
r/MachineLearning: mostly constructive feedback
HackerNews: this guy sucks
TechRxiv: we’re backed by IEEE, and ghost people who ask for status updates after we miss our self-imposed deadlines.
Glad I burned billable hours to work on this
BlueSky: *crickets*
r/MachineLearning: mostly constructive feedback
HackerNews: this guy sucks
TechRxiv: we’re backed by IEEE, and ghost people who ask for status updates after we miss our self-imposed deadlines.
Glad I burned billable hours to work on this
a man is carrying a suitcase in a living room and says i got nothin ' left
Alt: George Michael Bluth collapsing to the floor, saying “I got nothin’ left”
media.tenor.com
July 13, 2025 at 5:37 PM
I shared some of my findings recently.
BlueSky: *crickets*
r/MachineLearning: mostly constructive feedback
HackerNews: this guy sucks
TechRxiv: we’re backed by IEEE, and ghost people who ask for status updates after we miss our self-imposed deadlines.
Glad I burned billable hours to work on this
BlueSky: *crickets*
r/MachineLearning: mostly constructive feedback
HackerNews: this guy sucks
TechRxiv: we’re backed by IEEE, and ghost people who ask for status updates after we miss our self-imposed deadlines.
Glad I burned billable hours to work on this
TechRxiv is ghosting me, so here's my paper:
I fixed neural arithmetic. Division works. Extrapolation works. 10^-16 error.
Turns out: training distribution matters, complex numbers > log space, and those NALU weights were calculable all along.
Proof: hillspace.justindujardin.com
#MachineLearning
I fixed neural arithmetic. Division works. Extrapolation works. 10^-16 error.
Turns out: training distribution matters, complex numbers > log space, and those NALU weights were calculable all along.
Proof: hillspace.justindujardin.com
#MachineLearning
Hill Space is All You Need
hillspace.justindujardin.com
July 11, 2025 at 11:33 PM
TechRxiv is ghosting me, so here's my paper:
I fixed neural arithmetic. Division works. Extrapolation works. 10^-16 error.
Turns out: training distribution matters, complex numbers > log space, and those NALU weights were calculable all along.
Proof: hillspace.justindujardin.com
#MachineLearning
I fixed neural arithmetic. Division works. Extrapolation works. 10^-16 error.
Turns out: training distribution matters, complex numbers > log space, and those NALU weights were calculable all along.
Proof: hillspace.justindujardin.com
#MachineLearning
Reposted by Justin DuJardin
Reposted by Justin DuJardin
Castle Troll ballin’ out
July 8, 2025 at 4:57 PM
Castle Troll ballin’ out
Me waiting for TechRxiv to assign my DOI
a cartoon character sitting in front of a coleco computer with the word refresh below him
ALT: a cartoon character sitting in front of a coleco computer with the word refresh below him
media.tenor.com
July 7, 2025 at 8:29 PM
Me waiting for TechRxiv to assign my DOI
Reposted by Justin DuJardin
These characters will release soon! ✨ They come with an easy-to-edit UV map, animations, in 3 file formats, compatible with all game engines and above all; CC0 and free to use!
May 30, 2025 at 7:51 PM
These characters will release soon! ✨ They come with an easy-to-edit UV map, animations, in 3 file formats, compatible with all game engines and above all; CC0 and free to use!
Reposted by Justin DuJardin
The method in this paper was designed to find an optimal data mixture. But researchers in the human sciences who are training models *in order to understand the effect of the data* might also consider this as a clever way of evaluating hundreds of subsets without training hundreds of models. #MLSky
May 5, 2025 at 9:33 PM
The method in this paper was designed to find an optimal data mixture. But researchers in the human sciences who are training models *in order to understand the effect of the data* might also consider this as a clever way of evaluating hundreds of subsets without training hundreds of models. #MLSky
Reposted by Justin DuJardin
There was a rather charming trend in the 1910s to introduce the actor at the beginning of the film by showing them first in their street or evening clothes and then dissolving into the costume of their characters. Here we see recent stage import William S. Hart become a bandit in THE BARGAIN (1914).
May 3, 2025 at 10:47 PM
There was a rather charming trend in the 1910s to introduce the actor at the beginning of the film by showing them first in their street or evening clothes and then dissolving into the costume of their characters. Here we see recent stage import William S. Hart become a bandit in THE BARGAIN (1914).
Me, a genius, shortly before an unrelated run restart
April 20, 2025 at 8:19 PM
Me, a genius, shortly before an unrelated run restart
Sounds like a skill issue
The hallucinations will show options that aren’t available to stream on Netflix, just like Netflix’s current search does now.
Netflix is testing an OpenAI-powered search engine that lets users find shows via inquiries that go beyond genres or actors' names, like the subscriber's mood (Bloomberg)
Main Link | Techmeme Permalink
Main Link | Techmeme Permalink
April 12, 2025 at 4:55 AM
Sounds like a skill issue
When OLMo came out, I thought, "Wow, that's cool, but where's the real magic in its approach that would make me use it over [my existing LLM preference]?". The trace feature showing documents from the training data that match the answer it gave you is unexpected and a very attractive feature ✨
For years it’s been an open question — how much is a language model learning and synthesizing information, and how much is it just memorizing and reciting?
Introducing OLMoTrace, a new feature in the Ai2 Playground that begins to shed some light. 🔦
Introducing OLMoTrace, a new feature in the Ai2 Playground that begins to shed some light. 🔦
April 9, 2025 at 5:36 PM
When OLMo came out, I thought, "Wow, that's cool, but where's the real magic in its approach that would make me use it over [my existing LLM preference]?". The trace feature showing documents from the training data that match the answer it gave you is unexpected and a very attractive feature ✨
Reposted by Justin DuJardin
Large language models store vast amounts of knowledge, but how exactly do they learn it?
Excited to share my Google DeepMind internship results, which reveal the fascinating dynamics behind factual knowledge acquisition in LLMs!
Excited to share my Google DeepMind internship results, which reveal the fascinating dynamics behind factual knowledge acquisition in LLMs!
April 3, 2025 at 12:21 PM
Large language models store vast amounts of knowledge, but how exactly do they learn it?
Excited to share my Google DeepMind internship results, which reveal the fascinating dynamics behind factual knowledge acquisition in LLMs!
Excited to share my Google DeepMind internship results, which reveal the fascinating dynamics behind factual knowledge acquisition in LLMs!