Lightnews — Scholar-powered news

Reposted by Houjun Liu

Houjun Liu @jemoka.com · 8d

Introducing 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝗯𝘂𝗯𝗯𝗹𝗲𝘀: a *fully unsupervised* LM for input-adaptive parallel latent reasoning

✅ Learn yourself a reasoning model with normal pretraining
✅ Better perplexity compared to fixed thinking tokens

No fancy loss, no chain of thought labels 🚀

1 3 7

Houjun Liu @jemoka.com · 8d

I'm really excited about this. Because this model is trained with literally nothing but LM loss, it helps create a new reasoning paradigm where reasoning capabilities are baked right in at pretraining, unifying train and test time behaviors.
Look ma, no distribution shift! 🙏

Houjun Liu @jemoka.com · 8d

Better yet, without us teaching the model to do this at all, it learned to allocate more compute at tokens of higher entropy (even as measured by an independently trained model of the same architecture), and use less compute where there's either too little or too much entropy. 🤯

1 1

Houjun Liu @jemoka.com · 8d

By just using our approach, you don't have to do any extra work to get pretraining gains! We show across scale AND computation match that our approach performs better in pretraining perplexity than both regular transformers and manually inserting non-adaptive thinking tokens. 🥳

1 1

Houjun Liu @jemoka.com · 8d

We design an transformer variant that uses a score-attenuated "forking" mechanism to clone useful residuals the model wants to update and attend to, thus creating a 𝗯𝘂𝗯𝗯𝗹𝗲 of latent computation for those highly-informative tokens.

1 3

Houjun Liu @jemoka.com · 8d

Current approaches in scaling inference-time compute require supervising with explicit chain-of-thought data, which limits thoughts to be sequential and in human language only. 😔
Wouldn't it be nice if you can do normal pretraining, and somehow get latent thinking for free? 🤔

1 1

Houjun Liu @jemoka.com · 8d

Joint work with my wonderful collaborators @shikharmurty.bsky.social, @robertcsordas.bsky.social, and @chrmanning.bsky.social.

Paper: arxiv.org/abs/2510.00219.
Code and Package: github.com/stanfordnlp/....

Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space

Current approaches for scaling inference-time compute in transformers rely on training them to emit explicit chain-of-thought tokens before producing an answer. While these methods are powerful, they…

arxiv.org

1 1

Houjun Liu @jemoka.com · 8d

Introducing 𝘁𝗵𝗼𝘂𝗴𝗵𝘁𝗯𝘂𝗯𝗯𝗹𝗲𝘀: a *fully unsupervised* LM for input-adaptive parallel latent reasoning

✅ Learn yourself a reasoning model with normal pretraining
✅ Better perplexity compared to fixed thinking tokens

No fancy loss, no chain of thought labels 🚀

1 3 7

Reposted by Houjun Liu

Houjun Liu @jemoka.com · Aug 20

New Paper Day! For EMNLP findings—in LM red-teaming, we show you have to optimize for **both** perplexity and toxicity for high-probability, hard to filter, and natural attacks!

1 2 6

Houjun Liu @jemoka.com · Aug 20

Thanks to @schmidtsciences.bsky.social and Lambda Labs for generously supporting our work :)

Houjun Liu @jemoka.com · Aug 20

🤔 think this is all too much? No worries, we are also dropping a **PACKAGE** to do this for you. Check it out: github.com/sisl/astra-rl

GitHub - sisl/astra-rl: The Adaptive Stress Testing for Robust AI (ASTRA) toolbox provides tooling to support model developers and testing in the full life cycle of making more robust AI Systems through the application of adaptive stress testing and adversarial training.

The Adaptive Stress Testing for Robust AI (ASTRA) toolbox provides tooling to support model developers and testing in the full life cycle of making more robust AI Systems through the application of...

github.com

1

Houjun Liu @jemoka.com · Aug 20

☝️ And so.... You should optimize for **BOTH** attack success and perplexity to get the most effective attacks!

1

Houjun Liu @jemoka.com · Aug 20

Even across baseline methods, low-perplexity prompts result in more effective attacks, but optimizing for attack success alone results in high-perplexity prompts.

1

Houjun Liu @jemoka.com · Aug 20

In fact, our method allows us to discover a Pareto tradeoff (🤯) between attack success and prompt likelihood; tuning a single parameter in our method travels along the Pareto-optimal front.

1

Houjun Liu @jemoka.com · Aug 20

Using the Adaptive Stress Testing (AST) framework as a reward signal for an online DPO-based optimization, we present a method to discover **both** high-probability prompts that are also successful in attacks.

1

Houjun Liu @jemoka.com · Aug 20

Most approaches in gradient-based red-teaming result in very low-probability prompts, which previous work have shown are both easier to filter and bad negative examples for downstream hardening.

1

Houjun Liu @jemoka.com · Aug 20

Done at Stanford Intelligent Systems Laboratory — my joint first author Amelia Hardy, along with our wonderful collaborators Allie Griffith, @bernardlange.bsky.social, Duncan Eddy, Mykel Kochenderfer.

Paper:
arxiv.org/pdf/2407.09447
Python package to do this for yourself:
github.com/sisl/astra-rl

arxiv.org

1

Houjun Liu @jemoka.com · Aug 20

New Paper Day! For EMNLP findings—in LM red-teaming, we show you have to optimize for **both** perplexity and toxicity for high-probability, hard to filter, and natural attacks!

1 2 6

Reposted by Houjun Liu

FOCS 2025 @focs2025.bsky.social · Jul 13

The list of accepted papers at #FOCS2025 is up!

focs.computer.org/2025/accepte...

Accepted Papers – FOCS 2025

focs.computer.org

15 37

Reposted by Houjun Liu

Haskell programming language @haskell.org · Jun 21

You're not too dumb for Haskell, you just need a reason to practice. :)

3 14

Reposted by Houjun Liu

Journal of Open Source Software @joss-openjournals.bsky.social · Jul 3

Just published in JOSS: 'Turftopic: Topic Modelling with Contextual Representations from Sentence Transformers' https://doi.org/10.21105/joss.08183

1 1

Reposted by Houjun Liu

Martin Kleppmann @martin.kleppmann.com · Jun 27

OCaml @ocaml.org is in The Economist!

The Economist @economist.com · Jun 27

Jane Street is the quant shop's quant shop. The goose that lays the golden egg is its tech system, which is built rather unusually https://econ.trib.al/MPdov6Y

Jane Street’s sneaky retention tactic

It involves the use of an obscure, French programming language

econ.trib.al

4 9 54

Reposted by Houjun Liu

TTIC @tticconnect.bsky.social · Jun 27

We’re proud to announce three new tenure-track assistant professors joining TTIC in Fall 2026: Yossi Gandelsman, Will Merrill, and Nick Tomlin (@nickatomlin.bsky.social). Meet them here: buff.ly/JH1DFtT

2 7

Reposted by Houjun Liu

Mathurin Massias @mathurinmassias.bsky.social · Jun 18

New paper on the generalization of Flow Matching www.arxiv.org/abs/2506.03719

🤯 Why does flow matching generalize? Did you know that the flow matching target you're trying to learn *can only generate training points*?

w @quentinbertrand.bsky.social @annegnx.bsky.social @remiemonet.bsky.social 👇👇👇

2 17 55

Reposted by Houjun Liu

Houjun Liu @jemoka.com · Jun 2

New Paper Day! For ACL 2025 Findings:

You should **drop dropout** when you are training your LMs AND MLMs!

1 4 11