[Enhancement] Some idling after executing the last layer in expert worker

[shawlleyw]

Need to figure out and fix it. One possible reason is copying tensor from GPU to CPU and sending it to sampler via zmq.

[hogura99]

At what cases?

step_attn=2, DP=1, step_exp=1, EP=2?

[shawlleyw]

Almost every case as I remember

[hogura99]

Will use context recorder in #50 to check the details.

[hogura99]

ZMQ send took a long time that also influences other kernels like NCCLRecv and gather_tokens_cuda. It should be caused by device2host memcpy.

[hogura99]

The overlapping of tensor copy and expert execution has been done. But the performance has not been improved a lot since the problem also exists in attn, as following.

Will be fixed later.