In this lab, we study how a quadruped robot, Pupper, can learn to walk using Reinforcement Learning.

Instead of manually programming every leg movement, we define a reward function. The robot receives higher scores for desired behaviours, such as moving forward at the requested speed, keeping its body stable, and producing smooth movements. At the same time, it receives penalties for undesired behaviours, such as high effort, sudden movements, or instability.

In real Reinforcement Learning systems, a neural policy is trained in a simulator using an algorithm such as PPO, which stands for Proximal Policy Optimization. In this lab, students will work with a simplified reward tuning sandbox in order to understand how a reward function is calculated and how changing the coefficients affects the final score.

If the physical Pupper robot is available, the second part of the lab allows students to test an already trained policy on the real robot.

The original repository contains the controller and deploy files for Pupper. For the reward tuning activity, each student team will create an additional folder named reward_lab.

This folder will contain a simplified reward tuning sandbox. It is not a full physics simulator and it does not train a real neural policy. Its purpose is to help students understand how a reward function is built, how different reward terms are combined, and how changing coefficients affects the final score.

From the root of the repository, run:

cd ~/lab_5_fall_2025
mkdir -p reward_lab/outputs

After this step, the repository should contain:

lab_5_fall_2025/
├── config.yaml
├── launch.py
├── estop_controller.cpp
├── parkour_policy.json
├── test_policy.json
├── rebuild_neural_controller.py
├── deploy.py
└── reward_lab/
    └── outputs/

The folder reward_lab will be used only for the reward tuning exercise. Students must not modify config.yaml or the .json policy files when working on the reward function.

For the reward tuning activity, use the starter archive provided on OCW:

• Download reward_lab.zip

The original repository contains the controller and deploy files for Pupper. The archive reward_lab.zip adds a simplified reward tuning sandbox. This sandbox is not a full physics simulator and it does not train a real neural policy. Its purpose is to help you understand how a reward function is built, how different reward terms are combined, and how changing coefficients affects the final score.

Download reward_lab.zip and extract it in the root folder of the repository:

cd ~/lab_5_fall_2025
unzip reward_lab.zip
ls reward_lab

After extraction, the repository should contain:

lab_5_fall_2025/
├── config.yaml
├── launch.py
├── estop_controller.cpp
├── parkour_policy.json
├── test_policy.json
├── rebuild_neural_controller.py
├── deploy.py
└── reward_lab/
    ├── reward_config.yaml
    ├── reward_functions.py
    ├── run_reward_experiment.py
    ├── README.md
    └── outputs/

The folder reward_lab is used only for the reward tuning exercise. Do not modify config.yaml or the .json policy files when working on the reward function.

Install the required Python package:

sudo apt install python3-yaml

Run the experiment:

python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml

List the generated output files:

ls reward_lab/outputs

The script prints:

• the average total reward;
• the average raw value of each reward component;
• the average weighted contribution of each component;
• the CSV and JSON files generated in reward_lab/outputs.

Important: effort and smoothness are calculated as positive values, but they usually have negative coefficients in the configuration file. This means they reduce the total reward.

Before continuing, each team must confirm that:

• the folder reward_lab exists;
• the file reward_config.yaml exists;
• the file reward_functions.py exists;
• the file run_reward_experiment.py exists;
• the command below runs without errors:

python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml

The script displays:

• the average total reward;
• the average contribution of each reward component;
• the files generated in reward_lab/outputs.

Example output:

Reward experiment completed
Average total reward: 2.1453

Average components:
tracking_lin_vel: 0.8741
tracking_ang_vel: 0.9632
effort: 0.5120
stability: 0.8018
smoothness: 0.0921 

Important: the effort and smoothness functions return positive values, but in the configuration file they usually have negative coefficients. This means they decrease the total reward.

List the main files in the repository:

ls

List the available policies:

ls     *.json

Open the configuration file:

cat config.yaml

Answer in the report:

• What .json files are available?
• Which policy is referenced in config.yaml?
• Why are the .json files not the place where we modify the reward?
• What is the difference between an already trained policy and the reward function used during training?

In this mission, the robot should receive a high reward for tracking the desired linear velocity.

Edit the file:

nano reward_lab/reward_config.yaml

Set:

target_linear_velocity: 1.0
target_angular_velocity: 0.0

experiment:
    steps: 200
    seed: 42
    profile: "baseline"

reward_weights:
    tracking_lin_vel: 3.0
    tracking_ang_vel: 0.5
    effort: 0.0
    stability: 0.5
    smoothness: 0.0 

Run:

python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml

Write down:

• the average total reward;
• the contribution of tracking_lin_vel;
• the risk of rewarding the robot almost only for speed.

Now we keep the target velocity, but we penalise effort.

Modify the configuration:

target_linear_velocity: 1.0
target_angular_velocity: 0.0

experiment:
    steps: 200
    seed: 42
    profile: "baseline"

reward_weights:
    tracking_lin_vel: 3.0
    tracking_ang_vel: 0.5
    effort: -0.5
    stability: 0.5
    smoothness: 0.0 

Run:

python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml

Compare the result with the previous mission.

Answer:

• Did the total score increase or decrease?
• Why?
• Why is it useful to penalise effort?
• What trade-off appears between speed and energy consumption?

Now we want a more stable behaviour, even if the robot becomes slightly slower.

Modify the configuration:

target_linear_velocity: 0.8
target_angular_velocity: 0.0

experiment:
steps: 200
seed: 42
profile: "stable"

reward_weights:
tracking_lin_vel: 1.5
tracking_ang_vel: 0.5
effort: -0.2
stability: 3.0
smoothness: -0.2 

Run:

python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml

Answer:

• Which term is the most important in this configuration?
• Why was the target velocity reduced?
• What is the effect of increasing the coefficient for stability?

In this mission, we penalise sudden movements.

Modify the configuration:

target_linear_velocity: 0.8
target_angular_velocity: 0.0

experiment:
steps: 200
seed: 42
profile: "stable"

reward_weights:
tracking_lin_vel: 1.5
tracking_ang_vel: 0.5
effort: -0.2
stability: 2.0
smoothness: -1.0 

Run:

python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml

Answer:

• How does the smoothness penalty influence the total score?
• Why are smooth movements important for a real robot?
• What could happen if this penalty is too large?

Each team must propose its own configuration.

Create a new file:

cp reward_lab/reward_config.yaml reward_lab/reward_config_team.yaml
nano reward_lab/reward_config_team.yaml

You may modify:

• the target linear velocity;
• the target angular velocity;
• the profile: baseline, fast, stable, or unstable;
• the reward coefficients.

Run:

python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config_team.yaml

The team must explain:

• what values were chosen;
• why those values were chosen;
• which term is considered the most important;
• what trade-off was accepted: speed, effort, stability, or smoothness;
• whether the result is better than the previous missions.

Choose the 3 most important configurations you tested and complete the table:

Answer:

• Which configuration achieved the highest score?
• Which configuration seems more suitable for a real robot?
• Is the configuration with the highest score always the best one? Why?

This part is done only if you have access to the physical Pupper robot.

The script rebuild_neural_controller.py must be executed on the robot's Raspberry Pi, not on Windows and not in the local folder used for reward tuning.

Connect to the Raspberry Pi:

ssh pi@IP_ROBOT

On the Raspberry Pi:

cd ~
git clone https://github.com/cs123-stanford/lab_5_fall_2025.git
cd lab_5_fall_2025

Check the required files:

ls config.yaml launch.py estop_controller.cpp parkour_policy.json test_policy.json

Run:

python3 rebuild_neural_controller.py

If you get the error Source file does not exist, check that the repository is located in:

/home/pi/lab_5_fall_2025

and that the files listed above exist.

This part is done only on the real robot.

On the Raspberry Pi:

cd ~/lab_5_fall_2025
python3 deploy.py

After loading the policy, test the robot in a safe space.

Observe:

• Does the robot start correctly?
• Is the policy activated?
• Is the walking behaviour stable?
• Are there sudden movements?
• Does the robot fall?
• Does the floor surface influence the behaviour?

Reward tuning:

• is done in reward_lab/;
• modifies reward_config.yaml;
• runs run_reward_experiment.py;
• helps us understand the reward function.

Deploy:

• is done on the Raspberry Pi;
• uses rebuild_neural_controller.py;
• uses already trained policies saved as .json files;
• does not modify the reward function.

Each team must submit:

• the file reward_config_team.yaml;
• at least 3 output files generated in reward_lab/outputs;
• the comparison table between configurations;
• the answers to the mission questions;
• if the real robot is available: a video of the policy running on Pupper;
• if the real robot is not available: conclusions about the reward configuration.

The report must have a maximum of 2-3 pages and include:

• the names of the team members;
• a description of the reward function;
• the tested configurations;
• the obtained scores;
• the comparison between configurations;
• the explanation of the team's own configuration;
• the difference between reward tuning and deploy;
• conclusions.

Answer briefly:

• What is the role of the reward function in Reinforcement Learning?
• Why can a large coefficient for speed produce instability?
• What is the effect of penalising effort?
• Why is a stability term useful?
• What does smoother movement mean?
• Why are .json files not the place where we modify the reward?
• Why must rebuild_neural_controller.py be executed on the Raspberry Pi?
• What was the best configuration tested by your team?

If the reward_lab folder is missing, create it using the instructions at the beginning of the lab.

If run_reward_experiment.py does not start, check that you are in the main repository folder:

pwd
ls
ls reward_lab

If you get a No such file or directory error, check the path to the configuration file.

If you get an error related to yaml, run:

pip install pyyaml

If you run rebuild_neural_controller.py on your laptop and get errors related to /home/pi, the script was executed in the wrong environment. This script is intended for the robot's Raspberry Pi.

If the physical robot is not available, the deploy part is optional.