Lightnews — Scholar-powered news

Shikhar Murty @shikharmurty.bsky.social · Feb 14

“casual interception” as defined in \citep{}…

Reposted by Shikhar Murty

Hao Zhu 朱昊 @zhuhao.me · Feb 6

Ever dreamed of AI agents learning through interacting with the open world unsupervisedly? Our latest preprint introduces NNetNav-Live which collects training data through exploration on real websites and hindsight labeling, which produces a SOTA OSS agent.

1 2 4

Shikhar Murty @shikharmurty.bsky.social · Feb 6

controlling a browser / computer!
but requires a bit more tooling to set it up.

1

Shikhar Murty @shikharmurty.bsky.social · Feb 6

Please check out our paper for more details: arxiv.org/pdf/2410.02907

And our code if you want a NNetNav-ed model for your own domain:
github.com/MurtyShikhar...

Done with collaborators: @zhuhao.me, Dzmitry Bahdanau and @chrmanning.bsky.social

arxiv.org

Shikhar Murty @shikharmurty.bsky.social · Feb 6

We find that cross-website robustness is limited, and almost always, performance goes up from incorporating in-domain nnetnav data. This makes it even more important to work on unsupervised learning for agents - how are you going to collect human data for *any* website? [6/n]

1 1

Shikhar Murty @shikharmurty.bsky.social · Feb 6

We use this data for SFT-ing LLama3.1-8b. Our best models outperform zero-shot GPT-4 on both WebArena and WebVoyager, and reach SoTA performance among unsupervised methods for both datasets [5/n]

1

Shikhar Murty @shikharmurty.bsky.social · Feb 6

We use NNetNav to collect around 10k workflows for over 20 websites including 15 live websites, and 5 self-hosted websites.

Data is available on 🤗: huggingface.co/datasets/sta...
huggingface.co/datasets/sta...
[4/n]

stanfordnlp/nnetnav-live · Datasets at Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

1

Shikhar Murty @shikharmurty.bsky.social · Feb 6

Main ideas behind NNetNav exploration
1 complex goals have intermediate subgoals thus complex trajectories must have meaningful sub-trajectories
2 Use an LM instruction relabeler + judge to test if trajectory-so-far is meaningful. If yes, continue exploring, otherwise prune [3/n]

1

Shikhar Murty @shikharmurty.bsky.social · Feb 6

NNetNav uses a structured exploration method to efficiently search and collect traces on live-websites, which are retroactively labeled into instructions, finding a strikingly diverse set of workflows for any website (e.g. like this plot) [2/n]

1

Shikhar Murty @shikharmurty.bsky.social · Feb 6

Want to make a browser agent for *any* domain like banking or healthcare?
We propose methods for training LLMs with open-ended, unsupervised interaction on live websites:
✅ OSS SoTA on WebVoyager
✅ world's smallest high-performing web-agent
Try it here: nnetnav.dev

1 2 9

Shikhar Murty @shikharmurty.bsky.social · Dec 28

going to stay off twitter for my own mental health. something has gone horribly wrong with that platform.

5

Shikhar Murty @shikharmurty.bsky.social · Dec 15

Couldn't make it to NeurIPS due to work, but do check out our workshop happening in West Ballroom B. Lots of cool things to come, including a very fun panel!

Nouha Dziri @nouhadziri.bsky.social · Dec 15

Super excited today for the System 2 Reasoning at Scale workshop, come join us to discover how to equip AI systems with reasoning that's optimized for renewable energy and not fossil fuel 🔥🚀

⏰When? today, 9am-5:30pm
📍West Ballroom B

s2r-at-scale-workshop.github.io
#NeurIPS2024

2

Reposted by Shikhar Murty

Robert Csordas @robertcsordas.bsky.social · Dec 12

Come visit our poster "MoEUT: Mixture-of-Experts Universal Transformers" on Friday at 4:30 in East Exhibit Hall A-C #1907 on #NeurIPS2024. With Kazuki Irie, Jürgen Schmidhuber, Christopher Potts and @chrmanning.bsky.social.

1 5 14

Reposted by Shikhar Murty

Stanford NLP Group @stanfordnlp.bsky.social · Dec 9

The extraordinary recent takeover of ML/AI by #NLP is well-known but insufficiently reflected on.

Look at the @neuripsconf.bsky.social tutorials in 2024!

neurips.cc/virtual/2024...

14 tutorials; 6 have "LLM" in the title; 4 more cover foundation models, with large NLP coverage. That's > 70% 😲

NeurIPS 2024 TutorialsNeurIPS 2024

neurips.cc

1 14 64

Reposted by Shikhar Murty

Paul Soulos @paulsoulos.bsky.social · Dec 9

🚨 Thrilled to share that Compositional Generalization Across Distributional Shifts with Sparse Tree Operations received a spotlight award at #NeurIPS2024! 🌟 I'll present a poster on Tuesday and give an invited lightning talk at the System 2 Reasoning Workshop on Sunday. 🧵👇

1 4 11

Reposted by Shikhar Murty

Alexandre Lacoste @alex-lacoste.bsky.social · Dec 3

🧵-1
We are thrilled to release #AgentLab, a new open-source package for developing and evaluating web agents. This builds on the new #BrowserGym package which supports 10 different benchmarks, including #WebArena.

AgentLab diagram.

The image describes AgentLab, a framework for efficient parallel experiments with agents. It highlights:

Core Agent Features:

Dynamic Prompting and a Unified LLM API for interacting with large language models.
BrowserGym Platform:

A tool for testing agents on benchmarks like WebArena, WorkArena, MiniWoB, and others.
Key Features:

Reproducibility, a Unified Leaderboard, an analysis tool called Xray, and a Dataset for sharing agent traces.
Blue elements represent AgentLab components.

2 15 18

Shikhar Murty @shikharmurty.bsky.social · Dec 3

Folks, I'm not going to be at Neurips this year, but we have an *awesome* workshop that i'm super proud of.

Go attend, and use the link below to ask all of your burning questions about LLM reasoning, agents and compositionality!

Nouha Dziri @nouhadziri.bsky.social · Dec 3

🎊Excited for #neurips2024 and our "System 2 Reasoning at Scale" workshop. We have an excited lineup of speakers who will answer your most burning questions about AI and reasoning 🚀

🔥Got spicy questions? Submit & vote here:
app.sli.do/event/dJNU63...

Join Slido: Enter #code to vote and ask questions

Participate in a live poll, quiz or Q&A. No login required.

app.sli.do

1

Reposted by Shikhar Murty

Nouha Dziri @nouhadziri.bsky.social · Dec 3

🎊Excited for #neurips2024 and our "System 2 Reasoning at Scale" workshop. We have an excited lineup of speakers who will answer your most burning questions about AI and reasoning 🚀

🔥Got spicy questions? Submit & vote here:
app.sli.do/event/dJNU63...

Join Slido: Enter #code to vote and ask questions

Participate in a live poll, quiz or Q&A. No login required.

app.sli.do

1 3 4

Shikhar Murty @shikharmurty.bsky.social · Nov 27

I also wear the AI agents researcher hat. Can't say i'm similarly impressed by reviewers in that community...

Shikhar Murty @shikharmurty.bsky.social · Nov 27

ACL syntax track reviewers >> almost any other conference.

These folks care about their sub-field and i learn something new every time!

1

Shikhar Murty @shikharmurty.bsky.social · Nov 27

ACL syntax track reviewers >> almost any other conference.

These folks care about their sub-field and i learn something new every time!

1 2 12

Shikhar Murty @shikharmurty.bsky.social · Nov 26

Now, reviewers are upset if we only finetune sub 10B parameter models!

1

Shikhar Murty @shikharmurty.bsky.social · Nov 25

for more context: we are training the probe on sentences from PTB / BLIMP

1

Shikhar Murty @shikharmurty.bsky.social · Nov 25

thx for sharing, though semantic parsing almost certainly benefits from modeling syntax :)

1 1

Shikhar Murty @shikharmurty.bsky.social · Nov 24

SRL probe still rewards hidden states that model dependency relations, no? would like a probe thats agnostic to how well the underlying network models syntax

1 1

Shikhar Murty @shikharmurty.bsky.social · Nov 24

could i get added? thx for making this!!

2