One more thing — if you are looking to replace the Franka gripper with something more real-time friendly, we've got you covered: Our dynamixel-api package lets you control any Dynamixel-based gripper directly from Python. 🔗 github.com/TimSchneider... Special thanks to Erik Helmut!
Also, franky lets you access some of the functionality accessible only on the web interface, such as enabling FCI and unlocking the brakes from Python! But please don't tell Franka Robotics 🤫, because using their API like that is probably illegal.
But wait, there is more! franky exposes most libfranka functionality in its Python API: 🔧 Redefine end-effector properties ⚖️ Tune joint impedance 🛑 Set force/torque thresholds …and much more!
franky supports position & velocity control in both joint and task space — plus gripper control, contact reactions, and more! 🤖 With franky, you get real-time control both in C++ & Python: commands are fully preemptible, and Ruckig replans smooth trajectories on the fly.
5️⃣ In conclusion, we find that complementing vision with tactile sensing helps to train more robust policies under more challenging settings. In the future we plan to extend our extend our analysis to even more challenging tasks, such as screw or lightbulb insertion.
4️⃣ We also find that even for wider holes, the resulting vision-only policy is significantly less robust to changes in the environment (different hole sizes or angles) when tested zero-shot style. In contrast, the vision-tactile policy is robust even under unseen conditions.
3️⃣ Results? Turns out, as long as the hole is wide enough, a vision-only agent learns to solve the task just as well as a vision-tactile agent. However, once we make the hole tighter, we see that the vision-only agent fails to solve the task and gets stuck in a local minimum.
2️⃣ What did we do? We built a real, fully autonomous, and self-resetting tactile insertion setup and trained model-based RL directly in the real world. Using this setup, we run extensive experiments to understand the role of vision and touch in this task.
1️⃣ Robotic insertion in the real world is still a challenging task. Humans use a combination of vision and touch to exhibit dexterous behavior in the face of uncertainty. We wanted to know: What role do vision and touch play when RL agents learn to solve real-world insertion?
Stoked to present another work at RLDM 2025! If you’re into dexterous robotics, multimodal RL, or tactile sensing, swing by Poster 100 today to see what we cooked up 🦾✨ #Robotics#TactileSensing#RL#DexterousManipulation @ias_tudarmstadt
9️⃣ Come chat with us! Interested in active perception, transformers, or tactile robotics? Stop by poster 105 at RLDM this afternoon and let’s connect! 🗓️ 16:30 - 19:30 📍 Poster 105
8️⃣ Limitations & Future Directions: Like all deep RL, TAP needs a lot of data. Next steps: - Improve sample efficiency (think: pre-trained models) - Apply TAP on real robots (sim2real transfer) - Scale up to multi-finger/multi-modal (vision+touch) perception
7️⃣ What about baselines? TAP outperformed both random and prior state-of-the-art (HAM) baselines, highlighting the value of attention-based models and off-policy RL for tactile exploration.
6️⃣ We observe that TAP learns reasonable strategies. E.g., when estimating the pose of a wrench, TAP first scans the surface to find the handle and then moves towards one of the ends to determine pose and orientation.
5️⃣ Key Experiments: We tested TAP on a variety of ap_gym (github.com/TimSchneider...) tasks from the TactileMNIST benchmark (sites.google.com/robot-learni...). In all cases, TAP learns to actively explore & infer object properties efficiently.
4️⃣ How Does TAP Work? TAP jointly learns action and prediction with a shared transformer encoder using a combination of RL and supervised learning. We show that TAP's formulation arises naturally when optimizing a supervised learning objective w.r.t action and prediction.
3️⃣ Introducing TAP: We propose TAP (Task-agnostic Active Perception) — a novel method that combines RL and transformer models for tactile exploration. Unlike previous methods, TAP is completely task-agnostic, i.e., it can learn to solve a variety of active perception problems.
