Lightnews — Scholar-powered news

Stone Tao @stonet2000.bsky.social · 19d

1/1 neurips submission accepted! Will share more details later, covers an often overlooked aspect of on policy RL training when scaling to large scale parallelized environments

2

Reposted by Stone Tao

Nathan Lambert @natolambert.bsky.social · 19d

Rumors going around are that NeurIPS PCs had to reject ~400 papers because of venue constraints that were in the accept pile.

Hard things like this happen but the PCs and conference should explicitly tell all the authors who had this happen to them. Or create something like accept w/o presentation.

1 1 19

Stone Tao @stonet2000.bsky.social · 20d

my partner just hopped online, very talented with a great eye for design. She’s also behind all the beautiful graphics in the Lux AI Challenge that helped us gain 100s of competitors! She’s on the job market right now so DM her if you are looking for a designer/marketer/PM!

Isa Pan 🥐 @p-isa.bsky.social · 20d

New to Bluesky, so here's my lil' intro:

Hi, I'm Isa 👋 part designer, part marketer, part product wrangler 🧩

Built Dora AI and more with the rad team at Dora (dora.run) and 3 seasons of the Lux AI Challenge as CDO 🤩

Here to meet fellow product enthusiasts!

1 3

Stone Tao @stonet2000.bsky.social · 22d

what’s your next blog going to be?

i’ve been also meaning to pick it up again but writers block lol

1 1

Stone Tao @stonet2000.bsky.social · 29d

It works best if the object meshes you use for diff. rendering have a corresponding segmentation mask. If there are occlusions then the default option of using SAM2 to generate segmentations will perform a little worse. That being said even if the mask has some gaps the optimization can work

1

Stone Tao @stonet2000.bsky.social · Sep 9

Big thanks to Linghao Chen (author of EasyHEC) for helping out. I simply made some examples and simplified some code for reproducibility/readability.

3

Stone Tao @stonet2000.bsky.social · Sep 9

Opensourcing a tool to calibrate camera extrinsics painlessly in a minute, no checkerboards! It's based on EasyHEC, using diff. rendering to optimize extrinsics given object meshes+poses. Crazy that even a piece of paper works too.

Code:
github.com/StoneT2000/s...
(paper example in next post)

2 7 26

Stone Tao @stonet2000.bsky.social · Aug 20

Big thanks to Jeremy Morgan (PhD @ USC) for helping massively accelerate the IK solvers in ManiSkill! Open source really helps grow this project (and i get to learn tons of new things like IK optimization)
github.com/haosulab/Man...

[Feature] Faster IK based controllers using levenberg-marqurdt and align pd ee pos controllers in gpu/cpu sims by StoneT2000 · Pull Request #1213 · haosulab/ManiSkill

Cleaned up version of #1148 to close #955. Updates the original code to permit various other pd ee controller options with the new solver. LM solver is now the default. Users can modify the IK sol...

github.com

3

Stone Tao @stonet2000.bsky.social · Jul 19

ManiSkill3 simple example, Mujoco Playground, and the DextrahRGB paper all do very similar things.

I still think transfer techniques are useful, i’d think of these sim2real attempts as just explorations in the large scale visual DR space with parallel rendering. Both could be used together probably

1

Stone Tao @stonet2000.bsky.social · Jul 19

see section D of maniskill3 paper: arxiv.org/abs/2410.00425

tldr: Green screen background, segment out robot and cube in sim and only render that. Large scale visual domain randomization (robot, cube textures, camera parameters)

very simple trick that is fairly reproducible for simple tasks

ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI

Simulation has enabled unprecedented compute-scalable approaches to robot learning. However, many existing simulation frameworks typically support a narrow range of scenes/tasks and lack features crit...

arxiv.org

1 3

Stone Tao @stonet2000.bsky.social · Jul 19

might be the first time I made it personally on the trending developers list on github! One of the contributing factors being the lerobot sim2real code

1 17

Reposted by Stone Tao

Chris Paxton @cpaxton.bsky.social · Jul 5

Combining sim to real with cheap hardware like the so-100 really is making cutting edge robot learning so accessible and its wonderful

Stone Tao @stonet2000.bsky.social · Jul 4

Awesome to see people reproducing the accessible LeRobot zero-shot sim2real project! Trained for just 90 min in ManiSkill and deployed directly in real. Sim2real is not easy, but very rewarding when it works

Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...

7 7 49

Stone Tao @stonet2000.bsky.social · Jul 5

That doesn’t seem to make sense, you don’t need a RGBD camera, phone works fine. Happy to address it more if details can be shared on github if needed

1 1

Stone Tao @stonet2000.bsky.social · Jul 5

what bug did you encounter with OpenCVCameraConfig?

1 1

Stone Tao @stonet2000.bsky.social · Jul 4

For papers that aren’t ready / i think should be rejected due to many weaknesses, i often struggle to put anything in strengths and tend to just say some general thing. I only then elaborate more detailed (i hope) in the weaknesses section

1 1

Reposted by Stone Tao

Nathan Lambert @natolambert.bsky.social · Jul 4

My latest post: The American DeepSeek Project

Build fully open models in the US in the next two years to enable a flourishing, global scientific AI ecosystem to balance China's surge in open-source and an alternative to building products ontop of leading closed models.
buff.ly/kvJQE3I

1 6 34

Stone Tao @stonet2000.bsky.social · Jul 4

Awesome to see people reproducing the accessible LeRobot zero-shot sim2real project! Trained for just 90 min in ManiSkill and deployed directly in real. Sim2real is not easy, but very rewarding when it works

Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...

6 28

Stone Tao @stonet2000.bsky.social · Jun 28

Excited to announce that I will be interning at @nvidia research this summer on robotics/embodied AI! I’ll be in seattle for the summer, let me know if you want to meet up and chat! 🦾

10

Stone Tao @stonet2000.bsky.social · Jun 22

June 25 EEB 248 @ 3pm - Towards Embodiment Scaling Laws in Robot Locomotion by @BoAi0110

June 25 OHE 122 @ 11am - ImVR: Immersive VR Teleoperation System for General Purpose by Yulin Liu

2

Stone Tao @stonet2000.bsky.social · Jun 22

Rest of my lab has various presentations at RSS2025, please check out their awesome work!

June 25 EEB 248 @ 10am - Hardware Optimization for In-Hand Rotation by K. Fay

1 3

Stone Tao @stonet2000.bsky.social · Jun 22

~1 hour give or take maybe 20 mins (depends how close your camera is technically), and only 8gb of GPU memory!

someone even reproduced our sim2real tutorial with lerobot during the lerobot hackathon with a google colab gpu and a macbook for robot deployment

1 4

Stone Tao @stonet2000.bsky.social · Jun 22

The sim2real demo had some mixed success, hampered primarily by the lighting conditions of the outdoors.

At least it worked sometimes! Hindsight says that despite the weather, low-cost nature, only 1 hour of training, anything working is a miracle

1 2 9

Stone Tao @stonet2000.bsky.social · Jun 20

I’ll be at #RSS2025 from June 21 to June 23!

I’ll be giving a presentation on ManiSkill3 on June 21, 5:30 PM

We will also have two live demo sessions, on June 21, 12:30-2:00PM and 6:30-8:00PM. Swing by to see live demos of zero shot RGB sim2real, cool sim demos, and VR teleop!

1 3

Reposted by Stone Tao

Phillip Isola @phillipisola.bsky.social · Jun 15

Our computer vision textbook is now available for free online here:
visionbook.mit.edu

We are working on adding some interactive components like search and (beta) integration with LLMs.

Hope this is useful and feel free to submit Github issues to help us improve the text!

Foundations of Computer Vision

The print version was published by

visionbook.mit.edu

3 31 110

Stone Tao @stonet2000.bsky.social · Jun 15

Email so far ahead is interesting. Is this just substack emails?

1 2