feat(port-hilserl): Proof of concept to improve episode recording UX using pygame instead of csv.imshow

[jackvial]

What this does

• Use pygame for rendering camera feed to screen instead of cv2.imshow. This should offer better performance, camera feed and robot movement should be smoother when recording episodes
• Use pygame event for handling keyboard inputs
• Encapsulate rendering and keyboard events in a ControlContext class that gets updated in the control_loop. This makes it simple to render helpful information on top of the camera feed e.g. current reward, control phase (warmup|record episode)

Demo

pygame_camera_rendering_demo.mp4

How To Test

pip install pygame

Teleoperate

python lerobot/scripts/control_robot.py teleoperate --robot-path lerobot/configs/robot/koch.yaml --fps 30

Teleoperate should look like this. No episodes or rewards shown at top.

Record With Rewards

python lerobot/scripts/control_robot.py record \
    --robot-path lerobot/configs/robot/koch.yaml \
    --fps 30 \
    --root outputs/pygame_rendering_test_0 \
    --repo-id your_hf_username/pygame_rendering_test_0 \
    --warmup-time-s 2 \
    --episode-time-s 10 \
    --reset-time-s 5 \
    --num-episodes 10 \
    --push-to-hub 1 \
    --assign-rewards 1 \
    --single-task test_description

Warmup phase will not show rewards or episodes at the top of the screen.
Record phase will show episodes and count at top of screen.

After recording one episode I'm getting this error about the shape of the actions column, not yet sure what is happening here, will have too look into

Exception has occurred: ArrowInvalid
Column 1 named action expected length 0 but got length 589

[jackvial]

Hey @michel-aractingi , thanks for the feedback. I've removed my scripts and configs.
I've added the commands I used for teleoperate and record along with screen shots.

After recording one episode I get an error of

Exception has occurred: ArrowInvalid
Column 1 named action expected length 0 but got length 589

so I have either broken something or I'm missing some changes, will try figure this out tomorrow.

Also, I think log_control_info could also be moved to ControlContext and be enabled/disabled with a boolean flag in ControlContextConfig. log_control_info produces a lot of output and can quickly bury other information printed to the terminal so would be nice if it were configurable

[jackvial]

@michel-aractingi I've created a new branch off main for this as it will probably be easier to coordinate that way and avoid merging conflicts. This is the branch I'm working on https://github.com/jackvial/lerobot/commits/pygame-for-rendering-and-event-handling/ . I have a bit more work to do on it but plan to open a PR in the next few days.

Let me know if it's ok to close this draft PR

[jackvial]

closing in favor of #601