Unable to successfully start two joints

[bryangd34]

I was able to bringup one zeroerr drive unit and successfully control it.

However, with two units I am unable to get the joints (neither one at a time, nor two units) to successfully enter OPERATIONAL state and be controllable.

Launching produces different results:
Sometimes:
...
[ros2_control_node-1] Slave: State 0x01.
[ros2_control_node-1] Slave: State 0x02.
[ros2_control_node-1] Domain: WC 3.
[ros2_control_node-1] Domain: State 2.
[ros2_control_node-1] STATE: Switch On with status word :4659

But the joint only enters into SAFEOP.

Other times:
...
[ros2_control_node-1] Slave: State 0x01.
[ros2_control_node-1] Slave: State 0x02.
[ros2_control_node-1] Domain: WC 3.
[ros2_control_node-1] Domain: State 2.
[ros2_control_node-1] STATE: Switch On with status word :4659
[ros2_control_node-1] STATE: Switch on Disabled with status word :4688
[ros2_control_node-1] Master AL states: 0x0A.
[ros2_control_node-1] Slave: State 0x08.
[ros2_control_node-1] Slave: operational.
[ros2_control_node-1] STATE: Ready to Switch On with status word :4657
[ros2_control_node-1] STATE: Switch On with status word :4659
[ros2_control_node-1] STATE: Fault with status word :4616
[ros2_control_node-1] STATE: Switch on Disabled with status word :4688
[ros2_control_node-1] STATE: Ready to Switch On with status word :4657
[ros2_control_node-1] STATE: Switch On with status word :563
[ros2_control_node-1] STATE: Fault with status word :4616
[ros2_control_node-1] STATE: Switch on Disabled with status word :4688
[ros2_control_node-1] STATE: Ready to Switch On with status word :4657
[ros2_control_node-1] STATE: Switch On with status word :4659
[ros2_control_node-1] STATE: Fault with status word :4616
[ros2_control_node-1] STATE: Switch on Disabled with status word :4688
[ros2_control_node-1] STATE: Ready to Switch On with status word :4657
[ros2_control_node-1] STATE: Switch On with status word :4659
[ros2_control_node-1] STATE: Fault with status word :4616
...
and so forth
In the case, the joint is actually returning the state of OP, but is still not controllable.

Please inform me of any potentials solutions you may have, or suggestions for things I could try out and give you feedback which may help debug the issue.

Currently, I am trying to just control one joint at a time, but leaving two joints connected. I had success controlling just one joint when only one was connected.

Thank you.

[ZeroErrControl]

I suggest moving the position of eRob to 0 via the upper computer before running this program, as the program assumes the initial position is set to 0. Please ensure this condition is met before executing the program. If you encounter any other issues, feel free to ask!

[bryangd34]

Okay thanks. I will try that. It would be nice to have an update to allow the joint to be started up from any position.

[ZeroErrControl]

Okay thanks. I will try that. It would be nice to have an update to allow the joint to be started up from any position.

That's a great suggestion! We will include this requirement in our project plan. Additionally, feel free to ask if you encounter any project-related issues.

[bryangd34]

I tried setting the eRob 0. It is maybe slightly more likely to be successfully brought up now, but still needs many retries to get it to enter OP and without consequent errors. It would be great to have a way to very consistently bring up the drives successfully into OP state.

[wdb-123]

I tried setting the eRob 0. It is maybe slightly more likely to be successfully brought up now, but still needs many retries to get it to enter OP and without consequent errors. It would be great to have a way to very consistently bring up the drives successfully into OP state.

This issue is related to the EtherCAT interface program in MoveIt. After testing, even though the configuration file has DC mode enabled, the interface program still forces eRob into SM mode, causing some slaves to randomly fail to enter the OP state.

Temporary Solution:
By entering the command ethercat state -p op ( represents the slave number), you can manually bring the slaves that failed to enter the OP state into the OP state. This allows all slaves to operate normally in CSP mode.

Long-term Solution:
The EtherCAT low-level interface program in ROS 2 MoveIt needs to be modified to enable eRob to correctly enter DC mode. Engineers are currently working on resolving this issue, and the code will be updated on GitHub as progress is made.

[bryangd34]

Thanks @wdb-123 .

I still struggle a lot with this issue:
...
[ros2_control_node-1] Slave: State 0x01.
[ros2_control_node-1] Slave: State 0x02.
[ros2_control_node-1] Domain: WC 3.
[ros2_control_node-1] Domain: State 2.
[ros2_control_node-1] STATE: Switch On with status word :4659
[ros2_control_node-1] STATE: Switch on Disabled with status word :4688
[ros2_control_node-1] Master AL states: 0x0A.
[ros2_control_node-1] Slave: State 0x08.
[ros2_control_node-1] Slave: operational.
[ros2_control_node-1] STATE: Ready to Switch On with status word :4657
[ros2_control_node-1] STATE: Switch On with status word :4659
[ros2_control_node-1] STATE: Fault with status word :4616
[ros2_control_node-1] STATE: Switch on Disabled with status word :4688
[ros2_control_node-1] STATE: Ready to Switch On with status word :4657
[ros2_control_node-1] STATE: Switch On with status word :563
[ros2_control_node-1] STATE: Fault with status word :4616
[ros2_control_node-1] STATE: Switch on Disabled with status word :4688
[ros2_control_node-1] STATE: Ready to Switch On with status word :4657
[ros2_control_node-1] STATE: Switch On with status word :4659
[ros2_control_node-1] STATE: Fault with status word :4616
[ros2_control_node-1] STATE: Switch on Disabled with status word :4688
[ros2_control_node-1] STATE: Ready to Switch On with status word :4657
[ros2_control_node-1] STATE: Switch On with status word :4659
[ros2_control_node-1] STATE: Fault with status word :4616
...

So even if it does not successfully enter OP state at the start, and I use ethercat states -p 0 OP , I do still get this condition above looping endlessly. The state is returned as OP, but I cannot control the drive. I assume as it is oscillating through error states.

[bryangd34]

The issue seems to be more about status word showing a fault, than about entering the OP state. It seems to often enter OP state, but there is a status word fault. It isn't really clear what the fault is though.

[bryangd34]

The looping through the faults is just a result of auto_fault_reset: true

[ZeroErrControl]

So is this problem solved？

[bryangd34]

@ZeroErrControl no, it is not solved.

Even one joint, can take as many as 5 tries to successfully bring up. So attempting to bring up two joints with one launch file, could take around 25 (5 x 5) attempts. Ultimately, I will be trying to bring up more joints at once, so this current situation is not acceptable.

[ZeroErrControl]

Even one joint, can take as many as 5 tries to successfully bring up. So attempting to bring up two joints with one launch file, could take around 25 (5 x 5) attempts. Ultimately, I will be trying to bring up more joints at once, so this current situation is not acceptable.

You can use the ethercat slave command to check which joints have not entered the OP state and manually enable those specific joints. We are currently working on a long-term solution, and if you could assist in addressing this issue, it would help expedite the resolution.

[bryangd34]

@ZeroErrControl
Please refer to my previous comment above

ethercat slaves will return the joint as being in OP state, but it will still loop that fault as shown in my comment.

[ZeroErrControl]

Thanks @wdb-123 .

I still struggle a lot with this issue: ... [ros2_control_node-1] Slave: State 0x01. [ros2_control_node-1] Slave: State 0x02. [ros2_control_node-1] Domain: WC 3. [ros2_control_node-1] Domain: State 2. [ros2_control_node-1] STATE: Switch On with status word :4659 [ros2_control_node-1] STATE: Switch on Disabled with status word :4688 [ros2_control_node-1] Master AL states: 0x0A. [ros2_control_node-1] Slave: State 0x08. [ros2_control_node-1] Slave: operational. [ros2_control_node-1] STATE: Ready to Switch On with status word :4657 [ros2_control_node-1] STATE: Switch On with status word :4659 [ros2_control_node-1] STATE: Fault with status word :4616 [ros2_control_node-1] STATE: Switch on Disabled with status word :4688 [ros2_control_node-1] STATE: Ready to Switch On with status word :4657 [ros2_control_node-1] STATE: Switch On with status word :563 [ros2_control_node-1] STATE: Fault with status word :4616 [ros2_control_node-1] STATE: Switch on Disabled with status word :4688 [ros2_control_node-1] STATE: Ready to Switch On with status word :4657 [ros2_control_node-1] STATE: Switch On with status word :4659 [ros2_control_node-1] STATE: Fault with status word :4616 [ros2_control_node-1] STATE: Switch on Disabled with status word :4688 [ros2_control_node-1] STATE: Ready to Switch On with status word :4657 [ros2_control_node-1] STATE: Switch On with status word :4659 [ros2_control_node-1] STATE: Fault with status word :4616 ...

So even if it does not successfully enter OP state at the start, and I use ethercat states -p 0 OP , I do still get this condition above looping endlessly. The state is returned as OP, but I cannot control the drive. I assume as it is oscillating through error states.

It is recommended to create a new question specifically for this issue so that more developers can see and address it. We will also attempt to reproduce the problem based on your description. Currently, we are focusing on resolving the issue of eRob failing to enter DC mode through the MoveIt interface. It is uncertain whether this problem will persist after optimizing the EtherCAT interface. Nevertheless, your careful observations and experiments have provided valuable insights for solving the issue.

[bryangd34]

Alright thanks. I have opened another issue here

[bryangd34]

Closing, see linked issue above.