[Question] Seeking Advice: Training Nao Robot with RL using IsaacLab – Mimic Joints, Textures, and Physics Issues

[louislelay]

Hello everyone,

I'm working on a project to train the Nao robot using a rl agent. I've made some progress but encountered obstacles and would greatly appreciate your insights to ensure I'm on the right track.

Steps I Followed

1. Asset Import

   • I followed the Importing a new asset tutorial.
   • Used the URDF from this repository and meshes from this link.
   • Removed gazebo and transmission tags as instructed, and the modified URDF is attached at the end.
   • Converted the URDF to USD, I encountered the following error but when I followed this tutorial, I had also this error so I'm not really worried for this:

     Runtime Error: Failed to open layer @.../instanceable_meshes.usd@
     

     Despite this, the model rendered correctly in Isaac Sim.

3. Environment Setup

   • Created an RL environment inspired by this tutorial.
   • Adapted the humanoid direct environment and took inspiration from the Franka-Cabinet example to import my custom asset. The environment files are attached in a zip.

4. Testing with a Dummy Agent

   • Ran the following command to test the setup:

     ./isaaclab.sh -p source/standalone/environments/zero_agent.py --task Isaac-Nao-Direct-v0 --num_envs 32

   • The robots appeared stuck in the ground and generated these types of errors:

     [Error] [omni.physx.plugin] Usd Physics: the revolute joint at prim /World/envs/env_0/Nao/LFinger22_link/LFinger23 does need a finite limit set to be used by the mimic joint feature.
     [Error] [omni.hydra] [UsdToMdl] Prim '/__Prototype_38/Looks/material_RThumb2UV/Shader' parameter 'diffuse_texture': References an asset that can not be found: './Props/materials/textureNAO.png'

Issues

1. Mimic Joint Error

   [Error] [omni.physx.plugin] Usd Physics: the revolute joint at prim /World/envs/env_0/Nao/LFinger22_link/LFinger23 does need a finite limit set to be used by the mimic joint feature.
   

   • This error occurred for each finger joint. I tried changing the joints to revolute and setting limits but had no success, it just changed my errors to:

   [Error] [omni.physx.plugin] Usd Physics: PhysxMimicJointAPI at /World/envs/env_0/Nao/l_wrist/LFinger11 must have exactly 1 "referenceJoint" relationship defined.
   

   • Removing the mimic tags resolved the issue, but I’m open to better solutions.

2. Texture Error

   [Error] [omni.hydra] [UsdToMdl] Prim '/__Prototype_38/Looks/material_RThumb2UV/Shader' parameter 'diffuse_texture': References an asset that can not be found: './Props/materials/textureNAO.png'
   

   • The textures initially appeared correctly after the URDF conversion but now render as white. Any ideas why this happens?

3. Robot Stuck

   • Modifying the initial pose didn’t help; the robot seems stuck and doesn’t respond to gravity as expected.

Setup

• Isaac Sim : 4.2
• Ubuntu 22.04

Questions

1. Did I miss any critical steps or make mistakes in the process?
2. Are there better ways to handle mimic tags?
3. How can I resolve the issue of the robot being stuck in the ground or floating?
4. (Optional) Any suggestions to fix the texture issue?

Thanks in advance for your help!

---

URDF modified :
with mimic tags : nao_description_v1.zip
without : nao_description_v2.zip

Environment files : nao.zip

[RandomOakForest]

Thanks for posting this. There are several items to discuss here. To start, have you tried following this example and visualize the collision meshes?

I will move your post into our Discussions to better address this and follow up on your example.

[louislelay]

Thank you! So I followed the provided example and adjusted it to my case (including the additional steps for a mobile robot). Specifically, I set the Joint Drive Strength to 100,000 and the Joint Position Drive Damping to 5,000. The collision meshes appeared fine, and nothing seemed out of place visually.

After relaunching in the same environment using the zero agent, the texture issues were resolved—the colors displayed correctly. However, I’m still encountering the mimic tag problem. I tested it again with the modified URDF where I removed the mimic tags. Although this eliminated the error, the robot was still levitating slightly above the ground and remained stationary.

EDIT : I also wanted to mention that I tried importing the USD file directly into an empty Isaac Sim environment. After adding a ground plane and simply clicking "Play," the robot behaved as expected, falling naturally to the ground.

This makes me think the issue might lie in how the environment is being created using the script, but I’m not sure what could be causing it.

[louislelay]

Hi @RandomOakForest,

I'm still having trouble understanding why it's getting stuck like this. I tried importing a different robot (the Bolt robot) to see if following the same steps with another robot would make a difference, but I ran into the exact same issue. From what I gather, the robot seems to be stuck at a specific point, which corresponds to the position coordinates from the init_state. However, I don't see any joint restrictions in the Stage window of IsaacSim. I also attempted to set fix_root_link to false, but that didn't resolve the issue. If you have any suggestions, I would greatly appreciate it!

[louislelay]

Hi @RandomOakForest, I just wanted to let you know that I think I've finally resolved the issue with the robot getting stuck. I was experimenting with the Bolt robot and noticed that it seemed like there was a height that would reset the robot to its initial pose. Turns out, it was the termination_height variable, which was set higher than the robot’s height because it came from a modified environment based on the MuJoCo humanoid, which is much taller. I feel a bit silly for getting stuck on it for so long, but I’m glad I figured it out. Thanks again for your help!

[RandomOakForest]

Thank you for following up. We ran into some delay with the latest release and the holidays. Happy to hear you found what was causing the issue. Great job!

[louislelay]

For anyone wondering why the robot was getting stuck or if it feels like there's an invisible plane, make sure to check the termination_height variable – that was the issue I encountered.