LightNews
ankurhandos.bsky.social
Ankur Handa
@ankurhandos.bsky.social
1.2K followers 78 following 28 posts
Training robots in simulation.
Posts Media Videos Starter Packs
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Apr 29
This work was mainly a collaboration for GTC so it all came together quickly in 2 months and we didn't want to change much :)

We are working on improving the system and will release a tech report in a few months.
1 3
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Apr 29
*us being lazy not "using" :)
1 2
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Apr 29
The robot is rewarded to lift the object beyond a certain height to ensure that the grasp is stable. So it lifts it first and takes it to a certain height and then does the dropping. This was using being lazy for not changing the reward - vestigial stuff. The lift reward here.
1 3
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Apr 29
Stereo camera images that the networks use as input. They go directly into the network without any pre-processing and out comes action that is sent to the robot as target.
1 3
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Apr 29
Collaboration with Boston Dynamics on extending our DextrAH-RGB work on their robot.

The head movement is deliberate and learned thru training.

Each arm has its own independent controller. Which one to invoke when is determined by its proximity to the object.

www.youtube.com/watch?v=dFOb...
Arm you glad to see me, Atlas? | Boston Dynamics
YouTube video by Boston Dynamics
www.youtube.com
1 2 11
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Apr 7
Nice substack post that has a funny legend about Brunelleschi and his challenge to make an egg stand on its end. Whoever can make it stand gets to build the dome of santa maria del fiore.

www.james-lucas.com/p/it-always-...
2
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Apr 7
*cupola
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 13
I have been using cursor and it's great. It's a vscode fork so I like it as I have been using vscode for years now.
1 1
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 12
Just remembered this session "Discussion for Direct versus Features Session" link.springer.com/chapter/10.1... that I cited in my phd thesis but can't download now as it is behind paywall. But I remember it being very interesting and fun to read.
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 11
The message here is you should try to stay as close to raw pixels as possible - it just works out much better in the long run.

I love this hacker news comment that I saw on twitter few years ago.
1 5
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 11
Michal Irani, Michael Black, Padmanabham Anandan, Rick Szeliski, Harpreet Sawhney were all looking at recovering camera pose transformation directly from image pixels. And the tracking in ARIA glasses is uses "direct".
1 2
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 11
We have been using "direct" image to action mapping for a while now and the word "direct" took me back to tracking in SLAM. Back in the 90s many people were looking at recovering camera pose directly from the images rather than doing feature tracking first and then recovering homography afterwards.
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
Our new work has made a big leap moving away from depth based end-to-end to raw rgb pixels based end-to-end. We have two versions: mono and stereo, all trained entirely in simulation (IsaacLab).
1 5
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
Great work by my colleagues Ritvik Singh, Karl Van Wyk, Arthur Allshire and Nathan Ratliff.
.
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
This is the next one in the line in our dex-series of work where we started off with pose estimation as the representation of the object and gradually moved towards more general end-to-end image based direct pixels to action mapping.
1
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
When doing distillation, we also regress to the location of the object, which serves as a diagnostic tool for us to see what the network is predicting and use it for any state machine on top.
1
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
Our (stereo) vision network takes inspiration from the dust3r/mast3r work (with no explicit epipolar geometry imposed) where image embeddings are passed to a transformer with cross attention.
1
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
This approach differs from the common two-stage pipeline, where grasping or pick location is first regressed and then followed by motion planning. Instead, it integrates both stages into a single process and trains the entire system end-to-end using RL.
1
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
Scene Synthesizer: scene-synthesizer.github.io/getting_star...

ControlNets: github.com/lllyasviel/C...
Installation — Scene Synthesizer 1.12.6 documentation
scene-synthesizer.github.io
1 1
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
The benefits of training with parallel tiled rendering in simulation are still underappreciated. With modern tools like Scene Synthesizer and ControlNets, which transform synthetic images into photorealistic ones, the value of simulation-based training will only continue to grow.
1
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
Depth sensors are noisy and haven’t seen major improvements in a while and pure vision-based systems have caught up and most frontier models today use raw rgb pixels. We always wanted to move towards direct rgb based control and this work is our first attempt at doing so.
1
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
We train a teacher via RL with state vectors and student via distillation on images for learning control for 23 DoF multi-fingered hand and arm system. Doing end-to-end with real data as in BC-like systems is hard already but doing end-to-end with simulation is even harder due to the sim-to-real gap
1
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Feb 10
Our new work has made a big leap moving away from depth based end-to-end to raw rgb pixels based end-to-end. We have two versions: mono and stereo, all trained entirely in simulation (IsaacLab).
1 2 21
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Dec 29
Love the AI cartoons here:

timoelliott.com/blog/cartoon...
Artificial Intelligence Cartoons
Please feel free to use these cartoons in your presentations, university courses, books etc, for free, as long as you drop me a line to let me know what you have planned (telliott at timoelliott do…
timoelliott.com
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Nov 20
Can you share the other ones that you liked?
ankurhandos.bsky.social
Ankur Handa @ankurhandos.bsky.social · Nov 20
Why were you not convinced before? Is it because it uses images and did a lot more scaling than the previous work?
1