In this lab, you train a quadruped robot, Pupper, to stand and walk using Reinforcement Learning.

You do not manually program each leg movement. Instead, you define what “good movement” means through a reward function. The training algorithm runs many simulated Pupper robots in parallel, tests different behaviours and gradually improves the neural policy.

The main idea is:

• many simulated Pupper robots run in parallel;
• each simulated robot collects experience;
• the reward function evaluates the behaviour;
• PPO updates the neural policy;
• the policy gradually learns better movement.

This lab uses a Stanford-style Reinforcement Learning workflow based on Legged Gym / Isaac Gym. The training runs on the UPB GPU cluster through SLURM.

There are two different parts of the Pupper workflow:

• training a policy in simulation;
• deploying an already trained policy on the real robot.

These are not the same thing.

The deploy repository may contain files such as:

• config.yaml;
• launch.py;
• estop_controller.cpp;
• deploy.py;
• rebuild_neural_controller.py;
• parkour_policy.json;
• test_policy.json.

Those files are used for loading and deploying trained policies. They are not the full Reinforcement Learning training environment.

Do not modify the reward function in config.yaml. That file points the controller to an already trained policy.

For this lab, you need the training environment. The reward functions are edited in the Legged Gym training code, usually in files related to the Pupper environment.

In this lab, you will:

• connect to the UPB HPC cluster;
• submit a GPU job through SLURM;
• run or inspect the Pupper training environment;
• locate the reward functions;
• train a baseline policy;
• modify reward terms;
• run a new training experiment;
• compare the results;
• explain the trade-off between speed, stability and effort.

Download the Lab 5 HPC pack from OCW:

lab5_hpc_pack.zip

After extracting it, you should have this folder:

Use SSH to connect to the FEP.

Replace user.name with your faculty account.

Important: do not run the training directly on the FEP. The FEP is only the access node. Training must be submitted as a SLURM job.

After connecting, check the available partitions.

Look for GPU partitions such as:

• dgxa100;
• dgxh100;
• ucsx;
• ml.

For this lab, use one GPU per job.

If an A100 partition is available, prefer it for the old Stanford-style Legged Gym workflow. H100 is more powerful, but older simulator stacks can be more sensitive to software compatibility.

Check the available software modules.

If libraries/cuda-12.3 is not available, use module avail cuda and choose the CUDA module available on the cluster.

Create a working folder on the FEP.

Copy or upload the files from lab5_hpc_pack.zip into ~/pupper_lab5.

After copying the files, check that they are there.

You should see:

• README.txt;
• lab5_gpu_check.slurm;
• setup_leggedgym_env.sh;
• lab5_train_leggedgym.slurm;
• lab5_play_leggedgym.slurm;
• lab5_hpc_ocw_section.txt.

Before installing or running the training environment, test that you can submit a GPU job.

After you submit the job, SLURM prints something similar to:

Submitted batch job 123456

The number 123456 is an example job ID. Your job ID will be different.

When the job finishes, inspect the logs.

Replace 123456 with your real job ID.

The output should show information from:

• nvidia-smi;
• python3 –version;
• nvcc –version.

If this job fails, do not continue to the training part. Fix the GPU job first.

The training environment is based on Legged Gym.

The setup script creates a Python virtual environment and clones the required repositories.

You should get folders similar to:

• ~/pupper_lab5/leggedgym;
• ~/pupper_lab5/rsl_rl;
• ~/pupper_lab5/.venv;
• ~/pupper_lab5/logs.

Important: this script does not automatically install Isaac Gym. The old Stanford-style workflow may require Isaac Gym Preview and specific Python / PyTorch / CUDA versions.

If the instructor provides a prepared environment, container or module, use that instead of manually installing everything.

The simulator is not lab_5_fall_2025.

For this lab, the simulator is the Legged Gym training environment. This environment runs many simulated robots in parallel on the GPU.

The important argument is:

–num_envs=2000

This means that the training process uses many parallel simulated Pupper environments. Each simulated Pupper collects experience, and the PPO algorithm uses that experience to update the policy.

A simplified view:

• Pupper 1: observation → action → reward;
• Pupper 2: observation → action → reward;
• Pupper 3: observation → action → reward;
• …
• Pupper 2000: observation → action → reward;
• the collected data is combined;
• the policy is updated.

This is why a GPU is needed.

Go to the Legged Gym repository and search for the Pupper files.

You are looking for files similar to:

• legged_gym/envs/pupper/pupper.py;
• legged_gym/envs/pupper/pupper_config.py.

The exact path may differ depending on the repository version.

A reward function tells the robot what behaviour is good.

For example, a walking robot can receive:

• positive reward for moving forward;
• negative reward for using too much torque;
• positive reward for staying balanced;
• negative reward for falling;
• negative reward for sudden, unstable movements.

Common reward terms include:

• forward velocity;
• base height;
• torques;
• orientation;
• stability;
• smoothness;
• termination / fall penalty.

A reward function is not just about making the robot fast. If you reward only speed, the robot may move aggressively, fall or use too much energy.

A good policy balances:

• speed;
• stability;
• energy efficiency;
• smoothness;
• command following.

Open the training SLURM file.

Look for these variables:

For a first test, keep the default values.

Submit the training job.

After you submit the job, SLURM prints something similar to:

Submitted batch job 123457

The number 123457 is an example. Your job ID will be different.

Use your real job ID to check the logs.

Replace 123457 with your real job ID.

After the job finishes, check the training logs.

Save the output files. You need them for the report.

Before changing the reward function, run a very short experiment to make sure the pipeline works.

This run is not meant to produce a good policy. It is only a debugging run.

Check:

• Did the job start?
• Was the GPU detected?
• Did Python load the environment?
• Did training begin?
• Were logs created?

If the debug run fails, fix the environment before changing the reward function.

Open the Pupper environment file.

If the file path is different, use the result from find . -iname ” *pupper *”.

Identify at least three reward functions.

Complete this table in your report:

Write short explanations. Do not copy code without explaining it.

In this experiment, focus on forward movement.

Find the reward term related to forward velocity.

Possible names include:

• _reward_forward_velocity;
• tracking_lin_vel;
• lin_vel_x;
• forward_velocity.

Increase the importance of forward movement.

Do not delete the other reward terms. You still need the robot to stay stable.

Run a new job.

After the job finishes, save:

• the job ID;
• the SLURM output file;
• the modified reward code;
• the training logs;
• any generated curve or video.

Answer:

• Did the robot become faster?
• Did it become less stable?
• Did the reward increase?
• Was the movement natural or aggressive?

In this experiment, focus on reducing effort.

Find the reward term related to torque or effort.

Possible names include:

• _reward_torques;
• torque_penalty;
• energy;
• effort.

Increase the penalty for using too much torque.

Run a new job.

Compare this result with the velocity-focused run.

Answer:

• Did the movement become smoother?
• Did the robot become slower?
• Did the reward improve or decrease?
• What trade-off did you observe?

In this experiment, focus on stability.

Find reward terms related to:

• base height;
• orientation;
• falling;
• body tilt;
• smoothness.

Increase the importance of stability.

Run a new job.

Answer:

• Did the robot fall less often?
• Did the movement become slower?
• Did the robot keep a better posture?
• Was this policy better than the velocity-focused one?

Create your own reward configuration.

Your goal is to balance:

• forward walking;
• stability;
• effort;
• smoothness.

Run a final experiment.

Save the final reward configuration.

Explain:

• what you changed;
• why you changed it;
• what you expected;
• what actually happened.

Use the play script only after you have a trained policy or checkpoint.

After the job finishes, check the output.

Replace 123458 with your real job ID.

If no video is generated, use the training logs and SLURM output for your report.

Compare at least three runs:

• baseline or debug run;
• velocity-focused run;
• effort or stability-focused run;
• final run.

Complete the table:

Then answer:

• Which run had the best reward?
• Which run looked most stable?
• Which run looked most natural?
• Did the best numerical reward also produce the best movement?
• Which reward term had the strongest visible effect?

A policy trained in simulation may not work perfectly on the real robot. This difference is called the sim-to-real gap.

Possible causes:

• simulated motors are not identical to real motors;
• friction is different in the real world;
• the floor surface may be different;
• the battery level affects the robot;
• sensors are noisy;
• real hardware has delays;
• the simulated mass may not perfectly match the real robot.

Answer:

• Why is simulation useful before testing on hardware?
• Why can a policy fail when moved from simulation to the real robot?
• What could make the policy more robust?
• How could domain randomization help?

Do this only if the instructor confirms that the physical Pupper robot is ready.

The deploy repository is used only after the policy has been trained.

On the Raspberry Pi of the robot, the deploy repository may contain:

• config.yaml;
• launch.py;
• deploy.py;
• rebuild_neural_controller.py;
• policy.json.

Run deployment scripts only on the robot's Raspberry Pi, not on the FEP and not on your laptop.

If you see paths such as /home/pi/…, that script is meant for the Raspberry Pi.

Do not run robot deployment scripts on the HPC cluster.

Use this section when you work on the HPC cluster.

The numbers 123456 and 123457 are examples. Replace them with your real job ID.

After submitting a job with sbatch, SLURM prints something like:

Submitted batch job 123457

In that case, 123457 is the job ID.

Use this sequence for a normal run:

Use these commands for different experiments:

Run these commands inside the Legged Gym repository.

Do not run the training command directly on the FEP.

Do not run this directly in the terminal:

python legged_gym/scripts/train.py –task=pupper_flat –num_envs=2000 –max_iterations=300 –headless

Instead, run:

sbatch lab5_train_leggedgym.slurm

This sends the training job to a GPU compute node through SLURM.

Submit one archive containing:

team_name_lab5.zip

Inside the archive, include:

• report.pdf;
• slurm_outputs/;
• reward_code/;
• screenshots/;
• logs/;
• videos_optional/.

Your report must include:

• your team name;
• the GPU partition used;
• the job IDs;
• the baseline command;
• the reward functions you identified;
• the reward changes you made;
• the commands used for each run;
• screenshots or logs showing training;
• the comparison table;
• answers about the sim-to-real gap;
• final conclusions.

Answer these questions in your report:

• What is the role of the reward function in Reinforcement Learning?
• Why do we train Pupper in simulation before testing on the real robot?
• What does –num_envs=2000 mean?
• Why is GPU acceleration useful for this experiment?
• What happened when you emphasized forward velocity?
• What happened when you penalized effort?
• What happened when you emphasized stability?
• Which reward configuration worked best?
• Was the best numerical reward also the best behaviour?
• What is the sim-to-real gap?

Check your jobs with:

squeue -u $USER

The cluster may be busy. Wait or reduce the requested time.

Check available CUDA modules with:

module avail cuda

Choose an available CUDA module and modify the SLURM files.

You may not be on the correct cluster environment. Reconnect to the FEP and try again.

Check both files:

• cat logs/pupper_train_123457.out;
• cat logs/pupper_train_123457.err.

Replace 123457 with your real job ID.

The error file usually contains the useful information.

The Python environment is not ready. Activate the correct environment or install the required dependencies.

The training script checks this automatically.

If the output says False for CUDA availability, check:

• the CUDA module;
• the PyTorch version;
• whether the job really received a GPU;
• whether you used sbatch, not direct execution on FEP.

Search for it with:

• cd ~/pupper_lab5/leggedgym;
• find . -iname ” *pupper *”;
• grep -R “_reward” legged_gym/envs.

Use fewer iterations for testing:

MAX_ITERATIONS=50 sbatch lab5_train_leggedgym.slurm

Do not use long runs until the short run works.

That is normal for short training runs. The goal is to understand reward shaping and compare experiments, not necessarily to produce a perfect robot gait in one lab session.

Discuss the sim-to-real gap. Do not assume the training failed only because real-world deployment is imperfect.

In this lab, you used a Stanford-style Reinforcement Learning workflow to train Pupper in simulation. You submitted GPU jobs through SLURM, inspected reward functions, changed reward terms and compared policies.

The key idea is that robot learning is not only about running a training script. The quality of the learned behaviour depends strongly on how the reward function is designed.

A fast robot is not always a good robot. A stable robot is not always fast. A useful policy balances speed, effort, stability and smoothness.