In this lab, we study how a quadruped robot, Pupper, can learn to walk using Reinforcement Learning.

Instead of manually programming each leg movement, we define a reward function and train a neural policy in simulation. The robot receives higher rewards for behaviours we want, such as tracking a desired velocity, staying stable and moving efficiently. It receives penalties for behaviours we do not want, such as using too much effort, falling, tilting too much or producing jerky movements.

The original Stanford CS123 workflow uses large-scale parallel simulation for training. This means that many virtual Pupper robots are simulated at the same time. Each simulated robot collects experience, and the Reinforcement Learning algorithm uses that experience to improve the policy.

In this lab, students will follow this workflow:

• run or inspect the default policy;
• open the official training notebook or training environment;
• connect the experiment to Weights & Biases;
• modify reward terms;
• train a policy using parallel simulation;
• compare reward curves and generated videos;
• optionally deploy a trained policy on the physical Pupper robot.

By the end of the lab, students should be able to:

• explain the role of a reward function in Reinforcement Learning;
• describe why simulation is used before deploying on a real robot;
• identify the main reward terms used for locomotion;
• modify reward coefficients and explain their effect;
• run a training experiment using the official Stanford/Pupper workflow;
• read WandB curves and compare different policies;
• explain the sim-to-real gap;
• optionally deploy a trained policy on Pupper.

In Reinforcement Learning, an agent interacts with an environment. At each step, the agent observes the state of the environment, chooses an action and receives a reward.

In this lab:

• the agent is the neural policy controlling Pupper;
• the environment is the Pupper simulation;
• the actions are motor/controller commands;
• the reward measures how good the walking behaviour is;
• the policy is improved through training.

The algorithm used in the original workflow is PPO, which stands for Proximal Policy Optimization. We do not focus on the full mathematical derivation of PPO in this lab. The important idea is that PPO updates the neural policy so that actions leading to better rewards become more likely.

Training directly on the real robot would be slow, expensive and risky. During early training, the robot may fall, move aggressively, damage its servos or behave unpredictably.

Simulation solves this problem. In the simulator, thousands of trials can be run safely. If a virtual robot falls, the simulation simply resets. No hardware is damaged.

The Stanford workflow uses parallel simulation with MuJoCo/MJX and JAX. Instead of simulating one Pupper at a time, the system simulates many Pupper robots in parallel. This allows the policy to collect experience much faster.

The general idea is:

many simulated Pupper environments
        ↓
parallel experience collection
        ↓
PPO policy update
        ↓
improved walking behaviour

Pupper does not know what “good walking” means. We must define it through a reward function.

A reward function is usually built from several components. Each component encourages or discourages a specific behaviour.

Common reward terms for locomotion include:

• velocity tracking - reward for following the desired linear velocity;
• angular velocity tracking - reward for following the desired turning velocity;
• effort penalty - penalty for using too much torque or energy;
• stability - reward for keeping the body balanced;
• orientation penalty - penalty for excessive body tilt;
• smoothness - penalty for sudden changes in movement;
• height tracking - reward for maintaining an appropriate body height;
• fall penalty - penalty if the robot falls or becomes unstable.

A policy trained with a reward focused only on speed may become fast but unstable. A policy trained with too much effort penalty may become energy-efficient but too slow. A good reward function balances speed, stability, energy efficiency and smoothness.

The repository may contain files used for deploying a trained policy on the real robot, for example:

config.yaml
launch.py
estop_controller.cpp
parkour_policy.json
test_policy.json
rebuild_neural_controller.py
deploy.py

These files are used for controller setup and deployment. They are not necessarily the full training environment.

The reward function is not modified inside config.yaml and it is not modified inside .json policy files.

Important:

• .json files are already trained policies;
• config.yaml points the controller to a policy file;
• rebuild_neural_controller.py prepares the controller on the Raspberry Pi;
• deploy.py loads a trained policy on the robot;
• reward tuning is done in the official training notebook or training code.

If the full training notebook/environment is not available, students must ask the instructor for the official Lab 5 training material. The deploy repository alone is not enough for full Reinforcement Learning training.

Software:

• Python 3.9 or newer;
• Google Colab or a local Linux/WSL environment;
• MuJoCo/MJX;
• JAX;
• Weights & Biases;
• the official Lab 5 training notebook or training repository;
• the Lab 5 deploy repository, if the real robot is used.

Optional hardware:

• physical Pupper robot;
• Raspberry Pi on the robot;
• controller;
• stable power supply;
• safe testing area.

Clone the Lab 5 repository indicated by the instructor:

cd ~
git clone https://github.com/cs123-stanford/lab_5_fall_2025.git
cd lab_5_fall_2025

Check the files:

ls

If you only see deploy/controller files and policy .json files, this repository is mainly for deploying a trained policy. The actual reward tuning and training must be done in the official training notebook or training environment.

Open the official Lab 5 training notebook or training environment provided by the instructor.

The training environment is the place where you should find:

• reward configuration;
• reward terms;
• PPO configuration;
• simulation configuration;
• command sampling;
• termination conditions;
• training loop;
• video generation;
• WandB logging.

Do not try to modify the reward function in config.yaml from the deploy folder. That file is for loading a trained policy, not for training a new one.

Weights & Biases is used to monitor training experiments.

Create or use an existing WandB account. Then follow the instructions from the training notebook to log in.

Typical command:

wandb login

or, inside the notebook, paste your WandB API key when requested.

During training, WandB will record:

• total reward;
• reward components;
• policy performance;
• training videos;
• run name;
• training configuration.

Each team should save the WandB run link for the final report.

Before modifying the reward function, run or inspect the baseline policy.

The goal is to understand how the default policy behaves before training a new version.

Observe:

• Does Pupper walk forward?
• Is the movement stable?
• Does it fall?
• Does it move smoothly?
• Does it respond to velocity commands?
• What does the reward curve look like?

Deliverable for this mission:

• one screenshot or video of the baseline behaviour;
• 3-5 lines describing the baseline walking behaviour.

Find the reward section in the official training notebook or training code.

Identify the terms used for the reward function.

Look for terms such as:

tracking_lin_vel
tracking_ang_vel
effort
torque
stability
orientation
smoothness
height
termination

Write down:

• the name of each reward term;
• whether it is a reward or a penalty;
• what behaviour it encourages or discourages;
• the coefficient used for that term.

Complete the table:

In this mission, the goal is to encourage Pupper to follow a desired forward velocity.

Modify the reward configuration so that velocity tracking has a stronger influence.

Example idea:

tracking_lin_vel = higher value
tracking_ang_vel = moderate value
effort penalty = low or disabled

Use the exact variable names and configuration format from the official training notebook.

Run the training experiment.

While training, monitor:

• total reward;
• tracking reward;
• generated videos;
• stability of the policy;
• whether the robot moves too aggressively.

Deliverables:

• WandB run link;
• screenshot of reward curve;
• video or generated animation;
• short explanation of how increasing velocity tracking affected the behaviour.

A robot that moves fast but uses too much energy is not necessarily a good robot. In this mission, add or increase the effort penalty.

Example idea:

tracking_lin_vel = keep similar value
effort / torque penalty = increase penalty
smoothness = optional

Run the training again.

Compare this run with the velocity-focused run.

Observe:

• Did the total reward increase or decrease?
• Did the movement become slower?
• Did the movement become smoother?
• Did the robot become more stable?
• Was there a trade-off between speed and energy use?

Deliverables:

• WandB run link;
• reward curve comparison;
• short explanation of the trade-off between speed and effort.

In this mission, the goal is to obtain a more natural and stable gait.

Modify the reward configuration by adding or increasing terms related to:

• body stability;
• body orientation;
• smooth movement;
• fall prevention;
• height maintenance.

Example idea:

tracking_lin_vel = medium value
effort penalty = moderate value
stability / orientation = higher value
smoothness = moderate penalty

Run the training experiment.

Observe:

• Does the robot fall less often?
• Are the movements less aggressive?
• Does the robot maintain a better body posture?
• Is the walking behaviour more natural?
• Did the robot become slower?

Deliverables:

• WandB run link;
• training video or animation;
• short explanation of how stability and smoothness terms affected the gait.

Each team must design its own reward configuration.

The goal is to obtain the best balance between:

• forward walking;
• stability;
• energy efficiency;
• smoothness;
• command following.

Your team may modify:

• reward coefficients;
• target velocity range;
• command sampling;
• termination conditions, if allowed by the notebook;
• number of training steps, if time allows.

The team must explain:

• which reward terms were changed;
• why those terms were changed;
• what behaviour was expected;
• what behaviour was obtained;
• what trade-off was accepted.

Deliverables:

• final configuration;
• WandB run link;
• final video or generated animation;
• explanation of the design choices.

Compare at least three policies:

• baseline policy;
• velocity-focused policy;
• effort/stability-focused policy;
• team policy.

Complete the table:

Answer:

• Which policy achieved the highest reward?
• Which policy looked most stable?
• Which policy looked most natural?
• Is the policy with the highest reward always the best one?
• What would you change if you had more training time?

A policy trained in simulation may not work perfectly on the real robot. This difference is called the sim-to-real gap.

Possible causes:

• friction is different in the real world;
• servos do not behave exactly like the simulated motors;
• the robot mass may not match the simulated model;
• the floor surface may change the behaviour;
• sensors may be noisy;
• the battery level may affect the robot;
• the controller may have delays.

Answer in the report:

• Why is sim-to-real transfer difficult?
• Which reward terms could help make the policy more robust?
• How could domain randomization help?

If the training environment supports it, enable or modify domain randomization.

Examples of randomized parameters:

• robot mass;
• friction;
• motor strength;
• sensor noise;
• terrain;
• control delay.

Train again and compare the policy with the previous version.

Observe:

• Is the policy more robust?
• Does it work better on varied terrain?
• Did the training become harder?
• Did the reward curve become noisier?

Deliverable:

• short explanation of what was randomized;
• comparison with a non-randomized policy.

This part is done only if the physical Pupper robot is available.

The script rebuild_neural_controller.py must be executed on the robot's Raspberry Pi, not on Windows and not in the training notebook.

Connect to the Raspberry Pi:

ssh pi@IP_ROBOT

Clone the deploy repository on the Raspberry Pi:

cd ~
git clone https://github.com/cs123-stanford/lab_5_fall_2025.git
cd lab_5_fall_2025

Check the required files:

ls config.yaml launch.py estop_controller.cpp parkour_policy.json test_policy.json

Run:

python3 rebuild_neural_controller.py

If you get the error Source file does not exist, check that:

• you are on the Raspberry Pi;
• the repository is located in /home/pi/lab_5_fall_2025;
• the required files exist;
• the Pupper ROS2 workspace exists.

After training a policy, deploy it only if the physical robot is available and the instructor confirms that the robot is ready for testing.

On the Raspberry Pi:

cd ~/lab_5_fall_2025
python3 deploy.py

Select the trained policy or WandB run indicated by the instructor.

Test the robot in a safe space.

Observe:

• Does the policy load correctly?
• Does the robot stand safely?
• Does the robot walk?
• Does it fall?
• Are the movements smooth or aggressive?
• Does the behaviour match the simulation?

Deliverables:

• short video of the real robot;
• short comparison between simulation and real behaviour.

Each team must submit:

• the reward table from Mission 2;
• WandB links for the training runs;
• screenshots of the reward curves;
• generated videos or animations;
• the final reward configuration;
• the policy comparison table;
• answers to the report questions;
• optional: real robot video, if deployed.

The report should have 2-3 pages and include:

• team members;
• baseline policy observations;
• reward terms identified;
• experiments performed;
• reward curve screenshots;
• comparison between policies;
• final reward configuration;
• sim-to-real discussion;
• conclusions.

Answer briefly:

• What is the role of the reward function in Reinforcement Learning?
• Why is Pupper trained in simulation before being tested on the real robot?
• What is the advantage of running many simulated environments in parallel?
• What happened when velocity tracking was emphasized?
• What happened when effort was penalized?
• Why are stability and smoothness important?
• What is the sim-to-real gap?
• How could domain randomization help?
• Which policy was the best and why?

If the training notebook is not available, ask the instructor for the official Lab 5 training material. The deploy repository alone is not enough for full RL training.

If WandB does not work, check your API key and internet connection.

If training is very slow, check whether GPU/accelerator support is enabled.

If MuJoCo, MJX or JAX fails to import, check the environment setup instructions from the official training notebook.

If the generated videos do not appear, check the notebook output folder and WandB media tab.

If rebuild_neural_controller.py gives errors involving /home/pi, it is probably being run in the wrong environment. It must be run on the Raspberry Pi of the robot.

If the policy works in simulation but not on the real robot, discuss the sim-to-real gap instead of assuming that the training failed.