The PPO balancer is a feedforward neural network policy trained by reinforcement learning with a sim-to-real pipeline. Like the MPC balancer and PID balancer, it balances Upkie with straight legs. Training uses the UpkieGroundVelocity gym environment and the PPO implementation from Stable Baselines3.
An overview video of the training pipeline is given in this video: Sim-to-real RL pipeline for Upkie wheeled bipeds.
conda env create -f environment.yaml
conda activate ppo_balancerThe PPO balancer uses pixi and pixi-pack to pack a standalone Python environment to run policies on your Upkie. First, create environment.tar and upload it by:
make pack_pixi_env
make uploadThen, unpack the remote environment:
$ ssh user@your-upkie
user@your-upkie:~$ cd ppo_balancer
user@your-upkie:ppo_balancer$ make unpack_pixi_envTo run the default policy:
make run_agentHere we assumed the spine is already up and running, for instance by running ./start_simulation.sh on your machine, or by starting a pi3hat spine on the robot.
To run a policy saved to a custom path, use for instance:
python ppo_balancer/run.py --policy ppo_balancer/training/2023-11-15/final.zipOnce the agent and Python environment have been uploaded with the instructions above, you can SSH into the robot and run the same target:
$ ssh user@your-upkie
user@your-upkie:~$ make run_agentThis will run the policy saved at the default path. To run a custom policy, save its ZIP file to the robot (save its operative config as well for your future reference) and pass it path as argument to run.py.
First, check that training progresses one rollout at a time:
make train_and_showOnce this works you can train for real, with more environments and no GUI:
make trainCheck out the time/fps plots in the command line or in TensorBoard to adjust the number of parallel environments:
make tensorboardYou should increase the number of environments from the default value (NB_TRAINING_ENVS in the Makefile) to "as much as you can as long as FPS keeps going up".