I gave a talk earlier today as Stanford NLP seminar. Here are the slides if you are interested: deqingfu.github.io/_docs/202505...

At @naaclmeeting.bsky.social this week! I’ll be presenting our work on LLM domain induction with @thomason.bsky.social on Thu (5/1) at 4pm in Hall 3, Section I.

Would love to connect and chat about LLM planning, reasoning, AI4Science, multimodal stuff, or anything else. Feel free to DM!

It seems I haven't posted any research related posts on this platform. Starting to do it now.
bsky.app/profile/deqi...

I would like to thank my intern mentor Lawrence Chen from Meta, and all other peers Tong Xiao, Rui Wang, Guan Pang, and Pengchuan Zhang. Big thanks to my lab mate @billzhu.bsky.social for valuable discussions and my advisor @robinjia.bsky.social for thoughtful inputs.

Finally, token-level annotations given by TLDR model could speedup human annotators to fix image captions that are slightly off. In fact, it can speed up human annotation by 3 times!

Next, there is something interesting. After finishing training the TLDR model, one can simply remove the reward model head and re-attach the original language model head, to, obviously, become a new vision-language model. It's shown that these new models become better.

TLDR has rich usefulness. First, it can serve as a hallucination rate evaluation metric. As shown in the table, GPT-4o is still the best vision language model in the token level while open-weight models such as Llama-3.2-90B is catching up in the sentence and response level.

TLDR is trained on synthetic hard negatives generated via a perturbation-based method. The architecture is very simple. Instead of applying the reward model head to the last token, as many RMs are doing, TLDR applies the reward model head to every token.

Excited to share that my intern work at Meta GenAI is accepted to @iclr-conf.bsky.social #ICLR2025

Introducing TLDR: Token-Level Detective Reward Model For Large Vision Language Models.

TLDR provides fine-grained annotations to
each text token.

🔗arXiv: arxiv.org/abs/2410.04734

I think it may come from pretraining data and how numbers are presented by humans. We are still investigating how/why these features emerge from LLMs and will keep you updated with any new findings!

we have a very much similar results in NeurIPS 2024: arxiv.org/abs/2406.03445

I'll be at #NeurIPS2024! My group has papers analyzing how LLMs use Fourier Features for arithmetic and how TFs learn higher-order optimization for ICL (led by @deqing.bsky.social), plus workshop papers on backdoor detection and LLMs + PDDL (led by @billzhu.bsky.social)

Can add add me please? Thanks!

Thanks for making this pack. Can you add me please? Thank you!

🙌

USC NLP folks are on Bluesky!
Follow my amazing colleagues here

go.bsky.app/KUwSZ6W

Happy to join a new social media platform. I work on theory/science behind modern LLMs, and how to make them more robust and explainable.