what’s your next blog going to be?

i’ve been also meaning to pick it up again but writers block lol

It works best if the object meshes you use for diff. rendering have a corresponding segmentation mask. If there are occlusions then the default option of using SAM2 to generate segmentations will perform a little worse. That being said even if the mask has some gaps the optimization can work

Big thanks to Linghao Chen (author of EasyHEC) for helping out. I simply made some examples and simplified some code for reproducibility/readability.

ManiSkill3 simple example, Mujoco Playground, and the DextrahRGB paper all do very similar things.

I still think transfer techniques are useful, i’d think of these sim2real attempts as just explorations in the large scale visual DR space with parallel rendering. Both could be used together probably

see section D of maniskill3 paper: arxiv.org/abs/2410.00425

tldr: Green screen background, segment out robot and cube in sim and only render that. Large scale visual domain randomization (robot, cube textures, camera parameters)

very simple trick that is fairly reproducible for simple tasks

That doesn’t seem to make sense, you don’t need a RGBD camera, phone works fine. Happy to address it more if details can be shared on github if needed

what bug did you encounter with OpenCVCameraConfig?

For papers that aren’t ready / i think should be rejected due to many weaknesses, i often struggle to put anything in strengths and tend to just say some general thing. I only then elaborate more detailed (i hope) in the weaknesses section

June 25 EEB 248 @ 3pm - Towards Embodiment Scaling Laws in Robot Locomotion by @BoAi0110

June 25 OHE 122 @ 11am - ImVR: Immersive VR Teleoperation System for General Purpose by Yulin Liu

~1 hour give or take maybe 20 mins (depends how close your camera is technically), and only 8gb of GPU memory!

someone even reproduced our sim2real tutorial with lerobot during the lerobot hackathon with a google colab gpu and a macbook for robot deployment