Show page

Differences

This shows you the differences between two versions of the page.

--- rasb:lab:08 [2026/06/18 10:29]
vlad.radulescu2901 [Part 1 - Connect to the UPB HPC cluster]
+++ rasb:lab:08 [2026/07/08 19:29] (current)
vlad.radulescu2901 [Manual Install]
@@ Line 1: / Line 1: @@
 ====== Lab 8: Reinforcement Learning for Robotics ======
-===== Lab idea =====
+===== 1. Lab idea =====
-In this lab, you train a quadruped robot, Pupper, to stand and walk using Reinforcement Learning.
+In this lab, you will train a quadruped robot called Pupper in simulation using Reinforcement Learning.
-You do not manually program each leg movement. Instead, you define what “good movement” means through a reward function. The training algorithm runs many simulated Pupper robots in parallel, tests different behaviours and gradually improves the neural policy.
+The robot will be trained inside NVIDIA Isaac Gym, a GPU-based physics simulator that can run many environments in parallel. Instead of training one robot at a time, we can train hundreds or thousands of simulated robots at the same time.
-The main idea is:
+The learning algorithm used in this lab is PPO - Proximal Policy Optimization.
-    * many simulated Pupper robots run in parallel;
+The main goal of the lab is not to install Isaac Gym. The simulator and training pipeline are already prepared. Your task is to complete the reward functions used by the Pupper robot.
-    * each simulated robot collects experience;
-    * the reward function evaluates the behaviour;
-    * PPO updates the neural policy;
-    * the policy gradually learns better movement.
-This lab uses a Stanford-style Reinforcement Learning workflow based on Legged Gym / Isaac Gym. The training runs on the UPB GPU cluster through SLURM.
+Initially, the reward functions return zero, so the robot has no useful learning signal. You will implement reward terms that encourage the robot to:
-===== What you need to understand first =====
+    * move forward;
+    * keep a stable body height;
+    * avoid using unnecessarily large motor torques.
-There are two different parts of the Pupper workflow:
+At the end of the lab, you will compare the training results before and after modifying the reward function.
-    * training a policy in simulation;
+===== 2. Learning objectives =====
-    * deploying an already trained policy on the real robot.
-These are not the same thing.
+After this lab, you should be able to:
-The deploy repository may contain files such as:
+    * explain why reinforcement learning needs a reward function;
+    * understand why many simulated environments are used in parallel;
+    * run a training job on the HPC cluster using SLURM;
+    * identify and modify reward functions in a Legged Gym environment;
+    * compare multiple training runs using logs;
+    * explain how reward shaping influences the behavior learned by a robot;
+    * understand the basic idea of transferring a trained policy from simulation to the real Pupper robot.
-    * ''config.yaml'';
+===== 3. Background =====
-    * ''launch.py'';
-    * ''estop_controller.cpp'';
-    * ''deploy.py'';
-    * ''rebuild_neural_controller.py'';
-    * ''parkour_policy.json'';
-    * ''test_policy.json''.
-Those files are used for loading and deploying trained policies. They are not the full Reinforcement Learning training environment.
+A reinforcement learning agent learns by interacting with an environment.
-Do not modify the reward function in ''config.yaml''. That file points the controller to an already trained policy.
+For a robot, the environment contains:
-For this lab, you need the training environment. The reward functions are edited in the Legged Gym training code, usually in files related to the Pupper environment.
+    * the robot body;
+    * the physics simulation;
+    * gravity;
+    * contacts with the ground;
+    * joint positions and velocities;
+    * actions applied to the motors.
-===== What you will do =====
+At every step, the agent receives an observation and outputs an action. The simulator applies the action and returns a reward.
-In this lab, you will:
+The reward tells the agent whether its behavior is good or bad.
-    * connect to the UPB HPC cluster;
+For example:
-    * submit a GPU job through SLURM;
-    * run or inspect the Pupper training environment;
-    * locate the reward functions;
-    * train a baseline policy;
-    * modify reward terms;
-    * run a new training experiment;
-    * compare the results;
-    * explain the trade-off between speed, stability and effort.
-===== Required files =====
+    * if the robot moves forward, it should receive a positive reward;
+    * if it falls, it should receive a penalty;
+    * if it uses too much torque, it should receive a penalty;
+    * if it keeps a stable body height, it should receive a better score.
-Download the Lab 5 HPC pack from OCW:
+A bad reward function can make the robot learn nothing. A good reward function can make the robot learn useful locomotion.
-''lab5_hpc_pack.zip''
+===== 4. Important note about this lab =====
-After extracting it, you should have this folder:
+The HPC environment, container, Isaac Gym, PyTorch and Legged Gym are already prepared for you.
-^ File ^ Purpose ^
+You should not try to reinstall Isaac Gym manually during the lab.
-| ''README.txt'' | Short explanation of the pack |
-| ''lab5_gpu_check.slurm'' | Checks whether your GPU job works |
-| ''setup_leggedgym_env.sh'' | Prepares the project folder and clones the repositories |
-| ''lab5_train_leggedgym.slurm'' | Submits a Pupper training job |
-| ''lab5_play_leggedgym.slurm'' | Tests or visualizes a trained policy |
-| ''lab5_hpc_ocw_section.txt'' | Extra OCW text section, if needed |
-===== Part 1 - Connect to the UPB HPC cluster =====
+The important part of this lab is inside the file:
-Use SSH to connect to the FEP.
+<code>
+~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
+</code>
-^ Step ^ Command ^
+The initial version contains TODO functions similar to this:
-| Connect to the FEP | ''ssh -X -o ServerAliveInterval=100 [user.name@fep.grid.pub.ro]'' |
-Replace ''user.name'' with your faculty account.
+<code python>
+def _reward_base_height(self):
+    return 0.0
-Important: do not run the training directly on the FEP. The FEP is only the access node. Training must be submitted as a SLURM job.
+def _reward_forward_velocity(self):
+return 0
-===== Part 2 - Check the available GPU partitions =====
+def _reward_torques(self):
+return 0 </code>
-After connecting, check the available partitions.
+As long as these functions return zero, the robot has no meaningful learning signal.
-^ What you want to do ^ Command ^
+===== 5. Files used in this lab =====
-| Check available partitions | ''sinfo -o "%10P %55N %8c %10m %20G"'' |
-| Check all partition information | ''sinfo'' |
-| Check jobs on the A100 partition | ''squeue -p dgxa100'' |
-| Check jobs on the H100 partition | ''squeue -p dgxh100'' |
-Look for GPU partitions such as:
+Download the starter archive from OCW:
-    * ''dgxa100'';
+{{ :rasb:lab:lab8_hpc_pack.zip | Download Lab 8 HPC starter pack }}
-    * ''dgxh100'';
-    * ''ucsx'';
-    * ''ml''.
-For this lab, use one GPU per job.
+The starter pack contains the SLURM scripts needed for running the training jobs.
-If an A100 partition is available, prefer it for the old Stanford-style Legged Gym workflow. H100 is more powerful, but older simulator stacks can be more sensitive to software compatibility.
+Expected working directory:
-===== Part 3 - Check available modules =====
+<code>
+~/pupper_lab8
+</code>
-Check the available software modules.
+Expected repository structure:
-^ What you want to do ^ Command ^
+<code>
-| Show all available modules | ''module avail'' |
+~/pupper_lab8/
-| Search for CUDA modules | ''module avail cuda'' |
+├── isaacgym/
-| Show currently loaded modules | ''module list'' |
+├── leggedgym/
-| Clear loaded modules | ''module purge'' |
+├── rsl_rl/
-| Load CUDA 12.3 | ''module load libraries/cuda-12.3'' |
+├── pytorch_isaacgym.sif
-| Check loaded modules again | ''module list'' |
+├── pyuser_isaac/
+├── conda_tools/
+├── local_include/
+├── torch_extensions/
+├── logs/
+├── lab8_gpu_check.slurm
+├── lab8_import_check.slurm
+├── lab8_train_isaacgym.slurm
+└── lab8_reward_check.sh
+</code>
-If ''libraries/cuda-12.3'' is not available, use ''module avail cuda'' and choose the CUDA module available on the cluster.
+The archive contains only the small helper scripts. It does not contain the large simulator files, the container image or the full repositories.
-===== Part 4 - Create your project folder =====
+===== 6. Connect to the HPC cluster =====
-Create a working folder on the FEP.
+Connect to the frontend node:
-^ Step ^ Command ^
+<code bash>
-| Create the project folder | ''mkdir -p ~/pupper_lab5'' |
+ssh your_username@fep.grid.pub.ro
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+</code>
-| Create the logs folder | ''mkdir -p logs'' |
-| Check the folder content | ''ls'' |
-Copy or upload the files from ''lab5_hpc_pack.zip'' into ''~/pupper_lab5''.
+Go to the lab directory:
-After copying the files, check that they are there.
+<code bash>
+cd ~/pupper_lab8
+</code>
-^ What you want to check ^ Command ^
+Create the logs directory if it does not already exist:
-| List files | ''ls'' |
-| Make setup script executable | ''chmod +x setup_leggedgym_env.sh'' |
-| Make GPU check file executable | ''chmod +x lab5_gpu_check.slurm'' |
-| Make training file executable | ''chmod +x lab5_train_leggedgym.slurm'' |
-| Make play file executable | ''chmod +x lab5_play_leggedgym.slurm'' |
-You should see:
+<code bash>
+mkdir -p logs
+</code>
-    * ''README.txt'';
+===== 7. Check that the GPU is available =====
-    * ''lab5_gpu_check.slurm'';
-    * ''setup_leggedgym_env.sh'';
-    * ''lab5_train_leggedgym.slurm'';
-    * ''lab5_play_leggedgym.slurm'';
-    * ''lab5_hpc_ocw_section.txt''.
-===== Part 5 - Test that your GPU job works =====
+Before running Isaac Gym, check that your SLURM job can access the GPU.
-Before installing or running the training environment, test that you can submit a GPU job.
+Submit the GPU check job:
-^ Step ^ Command ^
+<code bash>
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+cd ~/pupper_lab8
-| Submit the GPU check job | ''sbatch lab5_gpu_check.slurm'' |
+sbatch lab8_gpu_check.slurm
-| Check your jobs | ''squeue -u $USER'' |
+</code>
-| List the log files | ''ls logs'' |
-After you submit the job, SLURM prints something similar to:
+Check the queue:
-''Submitted batch job 123456''
+<code bash>
+squeue -u $USER
+</code>
-The number ''123456'' is an example job ID. Your job ID will be different.
+After the job finishes, inspect the output:
-When the job finishes, inspect the logs.
+<code bash>
+ls logs
+cat logs/gpu_check_<JOB_ID>.out
+cat logs/gpu_check_<JOB_ID>.err
+</code>
-^ What you want to do ^ Example command ^
+Replace `<JOB_ID>` with the job id printed by `sbatch`.
-| Open the GPU check output | ''cat logs/gpu_check_123456.out'' |
-| Open the GPU check error file | ''cat logs/gpu_check_123456.err'' |
-Replace ''123456'' with your real job ID.
+A successful run should show an NVIDIA GPU through `nvidia-smi`.
-The output should show information from:
+===== 8. Check that Isaac Gym and Legged Gym work =====
-    * ''nvidia-smi'';
+Before changing the reward function, check that the simulator imports correctly.
-    * ''python3 --version'';
-    * ''nvcc --version''.
-If this job fails, do not continue to the training part. Fix the GPU job first.
+Submit the import check job:
-===== Part 6 - Prepare the training environment =====
+<code bash>
+cd ~/pupper_lab8
+sbatch lab8_import_check.slurm
+</code>
-The training environment is based on Legged Gym.
+After the job finishes:
-^ Step ^ Command ^
+<code bash>
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+cat logs/import_check_<JOB_ID>.out
-| Run the setup script | ''./setup_leggedgym_env.sh'' |
+cat logs/import_check_<JOB_ID>.err
-| Check the project folder | ''ls ~/pupper_lab5'' |
+</code>
-The setup script creates a Python virtual environment and clones the required repositories.
+A successful import check should contain:
-You should get folders similar to:
+<code>
+isaacgym import: OK
+gymtorch import: OK
+rsl_rl import: OK
+legged_gym import: OK
+CUDA available: True
+</code>
-    * ''~/pupper_lab5/leggedgym'';
+If this step fails, do not continue to the reward task. Ask the instructor or lab assistant for help.
-    * ''~/pupper_lab5/rsl_rl'';
-    * ''~/pupper_lab5/.venv'';
-    * ''~/pupper_lab5/logs''.
-Important: this script does not automatically install Isaac Gym. The old Stanford-style workflow may require Isaac Gym Preview and specific Python / PyTorch / CUDA versions.
+===== 9. Run a small debug training job =====
-If the instructor provides a prepared environment, container or module, use that instead of manually installing everything.
+Now check that the Pupper training task can start.
-===== Part 7 - Understand the simulator =====
+Submit a small debug job:
-The simulator is not ''lab_5_fall_2025''.
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
+</code>
-For this lab, the simulator is the Legged Gym training environment. This environment runs many simulated robots in parallel on the GPU.
+Check the queue:
-The important argument is:
+<code bash>
+squeue -u $USER
+</code>
-''--num_envs=2000''
+After the job finishes, check its output:
-This means that the training process uses many parallel simulated Pupper environments. Each simulated Pupper collects experience, and the PPO algorithm uses that experience to update the policy.
+<code bash>
+cat logs/pupper_train_<JOB_ID>.out
+cat logs/pupper_train_<JOB_ID>.err
+</code>
-A simplified view:
+A successful run should contain messages similar to:
-    * Pupper 1: observation -> action -> reward;
+<code>
-    * Pupper 2: observation -> action -> reward;
+Physics Engine: PhysX
-    * Pupper 3: observation -> action -> reward;
+Physics Device: cuda:0
-    * ...
+GPU Pipeline: enabled
-    * Pupper 2000: observation -> action -> reward;
+Learning iteration 0/10
-    * the collected data is combined;
+Learning iteration 9/10
-    * the policy is updated.
+Training finished.
+</code>
-This is why a GPU is needed.
+This means that Isaac Gym, PyTorch, CUDA and the Pupper environment are working.
-===== Part 8 - Locate the Pupper environment =====
+===== 10. Run the initial baseline =====
-Go to the Legged Gym repository and search for the Pupper files.
+Now run a slightly larger baseline:
-^ What you want to do ^ Command ^
+<code bash>
-| Go to the repository | ''cd ~/pupper_lab5/leggedgym'' |
+cd ~/pupper_lab8
-| List files | ''ls'' |
+RUN_NAME="baseline_zero_reward_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
-| Search for Pupper files | ''find . -iname "    *pupper    *"'' |
+</code>
-| Search for reward functions | ''grep -R "_reward" legged_gym/envs'' |
-| Search for velocity-related code | ''grep -R "velocity" legged_gym/envs'' |
-| Search for torque-related code | ''grep -R "torque" legged_gym/envs'' |
-| Search for height-related code | ''grep -R "height" legged_gym/envs'' |
-You are looking for files similar to:
+After the job finishes, inspect the reward values:
-    * ''legged_gym/envs/pupper/pupper.py'';
+<code bash>
-    * ''legged_gym/envs/pupper/pupper_config.py''.
+cat logs/pupper_train_<JOB_ID>.out | grep -E "Learning iteration|Mean reward|rew_forward_velocity|episode length" | tail -80
+</code>
-The exact path may differ depending on the repository version.
+You should observe that the reward is not useful. In the initial version, the relevant reward functions return zero.
-===== Part 9 - Understand reward functions =====
+Example:
-A reward function tells the robot what behaviour is good.
+<code>
+Mean reward: 0.00
+Mean episode rew_forward_velocity: 0.0000
+</code>
-For example, a walking robot can receive:
+This is expected before solving the lab.
-    * positive reward for moving forward;
+===== 11. Inspect the reward functions =====
-    * negative reward for using too much torque;
-    * positive reward for staying balanced;
-    * negative reward for falling;
-    * negative reward for sudden, unstable movements.
-Common reward terms include:
+Open the Pupper environment file:
-    * forward velocity;
+<code bash>
-    * base height;
+cd ~/pupper_lab8/leggedgym
-    * torques;
+nano legged_gym/envs/pupper/pupper.py
-    * orientation;
+</code>
-    * stability;
-    * smoothness;
-    * termination / fall penalty.
-A reward function is not just about making the robot fast. If you reward only speed, the robot may move aggressively, fall or use too much energy.
+Find the following functions:
-A good policy balances:
+<code python>
+def _reward_base_height(self):
+    return 0.0
-    * speed;
+def _reward_forward_velocity(self):
-    * stability;
+return 0
-    * energy efficiency;
-    * smoothness;
-    * command following.
-===== Part 10 - Inspect the baseline training script =====
+def _reward_torques(self):
+return 0 </code>
-Open the training SLURM file.
+These functions are the main TODOs of this lab.
-^ What you want to do ^ Command ^
+You can also use the helper script:
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Open the training file | ''nano lab5_train_leggedgym.slurm'' |
-Look for these variables:
+<code bash>
+cd ~/pupper_lab8
+./lab8_reward_check.sh
+</code>
-^ Variable ^ Meaning ^
+This script prints the reward functions and checks whether there are still `return 0` statements inside `pupper.py`.
-| ''TASK="${TASK:-pupper_flat}"'' | The training task. The default task is ''pupper_flat''. |
-| ''NUM_ENVS="${NUM_ENVS:-2000}"'' | The number of parallel simulated environments. |
-| ''MAX_ITERATIONS="${MAX_ITERATIONS:-300}"'' | The number of training iterations. |
-| ''RUN_NAME="${RUN_NAME:-team_${SLURM_JOB_ID}_pupper_flat}"'' | The experiment name. |
-For a first test, keep the default values.
+===== 12. Inspect the reward configuration =====
-===== Part 11 - Run a short baseline training job =====
+Open the configuration file:
-Submit the training job.
+<code bash>
+cd ~/pupper_lab8/leggedgym
+nano legged_gym/envs/pupper/pupper_config.py
+</code>
-^ Step ^ Command ^
+Look for the reward section. It should contain values similar to:
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Submit the training job | ''sbatch lab5_train_leggedgym.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-After you submit the job, SLURM prints something similar to:
+<code python>
+class rewards:
+    forward_velocity_clip = 1.0
-''Submitted batch job 123457''
+```
+class scales:
+    forward_velocity = 3.0
+```
-The number ''123457'' is an example. Your job ID will be different.
+</code>
-Use your real job ID to check the logs.
+The exact values may differ depending on the starter code version.
-^ What you want to do ^ Example command ^
+The important idea is that the config file contains the coefficients used to scale the reward terms.
-| Watch the output live | ''tail -f logs/pupper_train_123457.out'' |
-| Open the output file | ''cat logs/pupper_train_123457.out'' |
-| Open the error file | ''cat logs/pupper_train_123457.err'' |
-| Stop watching a live log | ''CTRL + C'' |
-| Cancel the job if needed | ''scancel 123457'' |
-Replace ''123457'' with your real job ID.
+For example:
-After the job finishes, check the training logs.
+    * a positive scale means the term increases the reward;
+    * a negative scale means the term becomes a penalty;
+    * a zero scale disables the term.
-^ What you want to do ^ Command ^
+===== 13. Task 1 - Implement forward velocity reward =====
-| Go to the Legged Gym repository | ''cd ~/pupper_lab5/leggedgym'' |
-| Find generated log files | ''find logs -maxdepth 3 -type f | head'' |
-| Find saved models | ''find . -type f -name "    *.pt"'' |
-| Find videos | ''find . -type f -name "    *.mp4"'' |
-| Find images | ''find . -type f -name "    *.png"'' |
-Save the output files. You need them for the report.
+The robot should receive a positive reward when it moves forward.
-===== Part 12 - Run a very short debugging experiment =====
+In Legged Gym, the forward velocity of the robot base is usually stored in:
-Before changing the reward function, run a very short experiment to make sure the pipeline works.
+<code python>
+self.base_lin_vel[:, 0]
+</code>
-^ What you want to do ^ Command ^
+This is the x-axis linear velocity of the robot base.
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Run a debug experiment | ''MAX_ITERATIONS=50 RUN_NAME="debug_${USER}" sbatch lab5_train_leggedgym.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-| List logs | ''ls logs'' |
-This run is not meant to produce a good policy. It is only a debugging run.
+Replace the initial function:
-Check:
+<code python>
+def _reward_forward_velocity(self):
+    return 0
+</code>
-    * Did the job start?
+with:
-    * Was the GPU detected?
-    * Did Python load the environment?
-    * Did training begin?
-    * Were logs created?
-If the debug run fails, fix the environment before changing the reward function.
+<code python>
+def _reward_forward_velocity(self):
+    return torch.clip(
+        self.base_lin_vel[:, 0],
+        min=0.0,
+        max=self.cfg.rewards.forward_velocity_clip
+    )
+</code>
-===== Part 13 - Mission 1: Identify the reward terms =====
+This reward gives positive values only when the robot moves forward. Negative velocity is clipped to zero, so moving backwards is not rewarded.
-Open the Pupper environment file.
+If `torch` is not imported at the top of the file, add:
-^ What you want to do ^ Command ^
+<code python>
-| Go to the Legged Gym repository | ''cd ~/pupper_lab5/leggedgym'' |
+import torch
-| Search for Pupper files | ''find . -iname "    *pupper    *"'' |
+</code>
-| Search for reward functions | ''grep -R "_reward" legged_gym/envs'' |
-| Open a file with nano | ''nano legged_gym/envs/pupper/pupper.py'' |
-If the file path is different, use the result from ''find . -iname "    *pupper    *"''.
+===== 14. Task 2 - Implement torque penalty =====
-Identify at least three reward functions.
+A robot should not learn to move by using extremely large motor torques. Large torques are inefficient and can produce unstable behavior.
-Complete this table in your report:
+The torque values are stored in:
-^ Reward term ^ Reward or penalty? ^ What behaviour does it encourage or discourage? ^
+<code python>
-| forward velocity | | |
+self.torques
-| torque / effort | | |
+</code>
-| base height | | |
-| stability / orientation | | |
-| smoothness | | |
-Write short explanations. Do not copy code without explaining it.
+Replace:
-===== Part 14 - Mission 2: Velocity-focused policy =====
+<code python>
+def _reward_torques(self):
+    return 0
+</code>
-In this experiment, focus on forward movement.
+with:
-Find the reward term related to forward velocity.
+<code python>
+def _reward_torques(self):
+    return torch.sum(torch.square(self.torques), dim=1)
+</code>
-Possible names include:
+This function returns a positive value representing how much torque the robot uses.
-    * ''_reward_forward_velocity'';
+Important: this function returns a positive value, but it becomes a penalty if the scale in the config file is negative.
-    * ''tracking_lin_vel'';
-    * ''lin_vel_x'';
-    * ''forward_velocity''.
-Increase the importance of forward movement.
+For example:
-Do not delete the other reward terms. You still need the robot to stay stable.
+<code python>
+torques = -0.0002
+</code>
-Run a new job.
+means that large torques reduce the final reward.
-^ What you want to do ^ Command ^
+===== 15. Task 3 - Implement base height penalty =====
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Run the velocity experiment | ''RUN_NAME="velocity_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-| List logs | ''ls logs'' |
-After the job finishes, save:
+The robot should keep its body at a reasonable height. If the body is too low or too high, the behavior is probably unstable.
-    * the job ID;
+The base height is stored in:
-    * the SLURM output file;
-    * the modified reward code;
-    * the training logs;
-    * any generated curve or video.
-Answer:
+<code python>
+self.root_states[:, 2]
+</code>
-    * Did the robot become faster?
+Replace:
-    * Did it become less stable?
-    * Did the reward increase?
-    * Was the movement natural or aggressive?
-===== Part 15 - Mission 3: Effort penalty =====
+<code python>
+def _reward_base_height(self):
+    return 0.0
+</code>
-In this experiment, focus on reducing effort.
+with:
-Find the reward term related to torque or effort.
+<code python>
+def _reward_base_height(self):
+    base_height = self.root_states[:, 2]
+    return torch.square(base_height - self.cfg.rewards.base_height_target)
+</code>
-Possible names include:
+This function returns the squared error between the current base height and the target base height.
-    * ''_reward_torques'';
+Again, this becomes a penalty if its scale in the config file is negative.
-    * ''torque_penalty'';
-    * ''energy'';
-    * ''effort''.
-Increase the penalty for using too much torque.
+===== 16. Check your code =====
-Run a new job.
+After editing `pupper.py`, check the relevant lines:
-^ What you want to do ^ Command ^
+<code bash>
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+cd ~/pupper_lab8/leggedgym
-| Run the effort experiment | ''RUN_NAME="effort_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
+nl -ba legged_gym/envs/pupper/pupper.py | sed -n '80,120p'
-| Check your jobs | ''squeue -u $USER'' |
+</code>
-| List logs | ''ls logs'' |
-Compare this result with the velocity-focused run.
+You should see the implemented reward functions, not `return 0`.
-Answer:
+You can also run:
-    * Did the movement become smoother?
+<code bash>
-    * Did the robot become slower?
+cd ~/pupper_lab8
-    * Did the reward improve or decrease?
+./lab8_reward_check.sh
-    * What trade-off did you observe?
+</code>
-===== Part 16 - Mission 4: Stability-focused policy =====
+If the script still shows `return 0` inside the reward functions, your implementation is not complete.
-In this experiment, focus on stability.
+===== 17. Train again after implementing the rewards =====
-Find reward terms related to:
+Run the training job again:
-    * base height;
+<code bash>
-    * orientation;
+cd ~/pupper_lab8
-    * falling;
+RUN_NAME="reward_fixed_512_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
-    * body tilt;
+</code>
-    * smoothness.
-Increase the importance of stability.
+Check the job:
-Run a new job.
+<code bash>
+squeue -u $USER
+</code>
+After it finishes:
+<code bash>
+cat logs/pupper_train_<JOB_ID>.out | grep -E "Learning iteration|Mean reward|rew_forward_velocity|episode length" | tail -100
+cat logs/pupper_train_<JOB_ID>.err
+</code>
+Compare the new output with the initial baseline.
+You should focus on:
+    * `Mean reward`;
+    * `Mean episode rew_forward_velocity`;
+    * `Mean episode length`;
+    * total timesteps;
+    * whether the job completed successfully.
+===== 18. Larger training runs =====
+After the small run works, you can try larger experiments.
+Run with 1000 environments:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_1000_${USER}" NUM_ENVS=1000 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
+If that works, try 2000 environments:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_2000_${USER}" NUM_ENVS=2000 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
+For a longer training run:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_long_${USER}" NUM_ENVS=2000 MAX_ITERATIONS=300 sbatch lab8_train_isaacgym.slurm
+</code>
+Do not start with the largest experiment. First check that the short run works.
+===== 19. Useful SLURM commands =====
 ^ What you want to do ^ Command ^
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+| Submit GPU check | ''sbatch lab8_gpu_check.slurm'' |
-| Run the stability experiment | ''RUN_NAME="stability_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
+| Submit import check | ''sbatch lab8_import_check.slurm'' |
+| Submit training | ''sbatch lab8_train_isaacgym.slurm'' |
+| Submit debug training | ''RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm'' |
 | Check your jobs | ''squeue -u $USER'' |
-| List logs | ''ls logs'' |
+| Check finished job status | ''sacct -j <JOB_ID> --format=JobID,JobName,Partition,State,Elapsed,ExitCode,MaxRSS,ReqMem'' |
+| Show output log | ''cat logs/pupper_train_<JOB_ID>.out'' |
+| Show error log | ''cat logs/pupper_train_<JOB_ID>.err'' |
+| Show only reward lines | ''cat logs/pupper_train_<JOB_ID>.out | grep -E "Mean reward|rew_forward_velocity|episode length"'' |
+| Inspect reward functions | ''./lab8_reward_check.sh'' |
-Answer:
+===== 20. Common problems =====
-    * Did the robot fall less often?
+==== Problem 1 - The reward stays zero ====
-    * Did the movement become slower?
-    * Did the robot keep a better posture?
-    * Was this policy better than the velocity-focused one?
-===== Part 17 - Mission 5: Your final reward configuration =====
+If the output still shows:
-Create your own reward configuration.
+<code>
+Mean reward: 0.00
+Mean episode rew_forward_velocity: 0.0000
+</code>
-Your goal is to balance:
+check that you actually modified:
-    * forward walking;
+<code>
-    * stability;
+~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
-    * effort;
+</code>
-    * smoothness.
-Run a final experiment.
+and that the reward functions no longer return zero.
-^ Situation ^ Command ^
+Use:
-| Normal final run | ''RUN_NAME="final_${USER}" MAX_ITERATIONS=500 sbatch lab5_train_leggedgym.slurm'' |
-| Shorter final run if the queue is busy | ''RUN_NAME="final_${USER}" MAX_ITERATIONS=200 sbatch lab5_train_leggedgym.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-| List logs | ''ls logs'' |
-Save the final reward configuration.
+<code bash>
+cd ~/pupper_lab8
+./lab8_reward_check.sh
+</code>
-Explain:
+or:
-    * what you changed;
+<code bash>
-    * why you changed it;
+grep -R "return 0" ~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
-    * what you expected;
+</code>
-    * what actually happened.
-===== Part 18 - Test or visualize a trained policy =====
+==== Problem 2 - The job is killed with OOM ====
-Use the play script only after you have a trained policy or checkpoint.
+If the job fails with:
-^ Step ^ Command ^
+<code>
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+Detected 1 oom_kill event
-| Submit the play job | ''sbatch lab5_play_leggedgym.slurm'' |
+Some of the step tasks have been OOM Killed
-| Check your jobs | ''squeue -u $USER'' |
+</code>
-| List logs | ''ls logs'' |
-After the job finishes, check the output.
+then the job used too much RAM.
-^ What you want to do ^ Example command ^
+Use fewer environments:
-| Open the play output | ''cat logs/pupper_play_123458.out'' |
-| Open the play error file | ''cat logs/pupper_play_123458.err'' |
-| Find generated files | ''cd ~/pupper_lab5/leggedgym && find logs -type f | head -20'' |
-| Search for videos | ''cd ~/pupper_lab5/leggedgym && find . -type f -name "    *.mp4"'' |
-Replace ''123458'' with your real job ID.
+<code bash>
+NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-If no video is generated, use the training logs and SLURM output for your report.
+or:
-===== Part 19 - Compare your experiments =====
+<code bash>
+NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
+</code>
-Compare at least three runs:
+If needed, the SLURM memory limit can be increased by the instructor in the training script:
-    * baseline or debug run;
+<code bash>
-    * velocity-focused run;
+#SBATCH --mem=128G
-    * effort or stability-focused run;
+</code>
-    * final run.
-Complete the table:
+==== Problem 3 - Task not registered ====
-^ Run name ^ Main reward change ^ Iterations ^ Result observed ^ Main problem ^
+If you see:
-| baseline/debug | none | | | |
-| velocity | increased forward movement reward | | | |
-| effort | increased effort penalty | | | |
-| stability | increased stability terms | | | |
-| final | custom configuration | | | |
-Then answer:
+<code>
+ValueError: Task with name: ... was not registered
+</code>
-    * Which run had the best reward?
+check that the task name is:
-    * Which run looked most stable?
-    * Which run looked most natural?
-    * Did the best numerical reward also produce the best movement?
-    * Which reward term had the strongest visible effect?
-===== Part 20 - Understand the sim-to-real gap =====
+<code>
+pupper_flat
+</code>
-A policy trained in simulation may not work perfectly on the real robot. This difference is called the sim-to-real gap.
+You can inspect registered tasks with:
-Possible causes:
+<code bash>
+cd ~/pupper_lab8/leggedgym
+grep -R "task_registry.register" -n legged_gym/envs
+</code>
-    * simulated motors are not identical to real motors;
+==== Problem 4 - You edited the wrong task ====
-    * friction is different in the real world;
-    * the floor surface may be different;
-    * the battery level affects the robot;
-    * sensors are noisy;
-    * real hardware has delays;
-    * the simulated mass may not perfectly match the real robot.
-Answer:
+This lab uses:
-    * Why is simulation useful before testing on hardware?
+<code>
-    * Why can a policy fail when moved from simulation to the real robot?
+pupper_flat
-    * What could make the policy more robust?
+</code>
-    * How could domain randomization help?
-===== Part 21 - Optional: Deploy on real Pupper =====
+Do not confuse it with:
-Do this only if the instructor confirms that the physical Pupper robot is ready.
+<code>
+pupper_standup
+</code>
-The deploy repository is used only after the policy has been trained.
+The standup task may have different reward scales, including zero forward velocity reward.
-On the Raspberry Pi of the robot, the deploy repository may contain:
+==== Problem 5 - The output stops after gymtorch ====
-    * ''config.yaml'';
+If the output stops around:
-    * ''launch.py'';
-    * ''deploy.py'';
-    * ''rebuild_neural_controller.py'';
-    * ''policy.json''.
-Run deployment scripts only on the robot's Raspberry Pi, not on the FEP and not on your laptop.
+<code>
+Building extension module gymtorch...
+ninja: no work to do.
+</code>
-If you see paths such as ''/home/pi/...'', that script is meant for the Raspberry Pi.
+check the error file:
-Do not run robot deployment scripts on the HPC cluster.
+<code bash>
+cat logs/pupper_train_<JOB_ID>.err
+</code>
-===== Useful SLURM commands =====
+If the job completed successfully, use:
-Use this section when you work on the HPC cluster.
+<code bash>
+sacct -j <JOB_ID> --format=JobID,JobName,Partition,State,Elapsed,ExitCode,MaxRSS,ReqMem
+</code>
-^ What you want to do ^ Command ^
+===== 21. Questions =====
-| Submit the GPU check job | ''sbatch lab5_gpu_check.slurm'' |
-| Submit the training job | ''sbatch lab5_train_leggedgym.slurm'' |
-| Submit the play / test job | ''sbatch lab5_play_leggedgym.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-| Check jobs on the A100 partition | ''squeue -p dgxa100'' |
-| Check jobs on the H100 partition | ''squeue -p dgxh100'' |
-| Cancel a job | ''scancel 123457'' |
-| Check information about a finished job | ''sacct -j 123457'' |
-| Check detailed job information | ''sacct -j 123457 --format=JobID,JobName,Partition,State,Elapsed,MaxRSS,AllocGRES'' |
-| List log files | ''ls logs'' |
-| Open GPU check output | ''cat logs/gpu_check_123456.out'' |
-| Open GPU check error file | ''cat logs/gpu_check_123456.err'' |
-| Open training output | ''cat logs/pupper_train_123457.out'' |
-| Open training error file | ''cat logs/pupper_train_123457.err'' |
-| Watch a running training log | ''tail -f logs/pupper_train_123457.out'' |
-| Show the last lines of a training log | ''tail logs/pupper_train_123457.out'' |
-| Show the last 50 lines of a training log | ''tail -n 50 logs/pupper_train_123457.out'' |
-| Check available partitions | ''sinfo'' |
-| Check partitions in compact format | ''sinfo -o "%10P %55N %8c %10m %20G"'' |
-| Check available modules | ''module avail'' |
-| Search for CUDA modules | ''module avail cuda'' |
-| Clear loaded modules | ''module purge'' |
-| Load CUDA | ''module load libraries/cuda-12.3'' |
-| Check loaded modules | ''module list'' |
-| Check GPU information | ''nvidia-smi'' |
-| Search for Pupper files | ''find . -iname "    *pupper    *"'' |
-| Search for reward functions | ''grep -R "_reward" legged_gym/envs'' |
-| Find generated files in logs | ''find logs -type f | head -20'' |
-| Search for saved models | ''find . -type f -name "    *.pt"'' |
-| Search for videos | ''find . -type f -name "    *.mp4"'' |
-| Search for images | ''find . -type f -name "    *.png"'' |
-The numbers ''123456'' and ''123457'' are examples. Replace them with your real job ID.
+Answer the following questions in your report:
-After submitting a job with ''sbatch'', SLURM prints something like:
+    * Why did the initial training run produce zero reward?
+    * Why is forward velocity a useful reward term for locomotion?
+    * Why should large torques be penalized?
+    * Why can body height be used as a stability-related reward term?
+    * What is the purpose of training many environments in parallel?
+    * What changed after you implemented the reward functions?
+    * Did the robot learn better behavior after more iterations? Explain using the log values.
+    * Why might a policy that works in simulation behave differently on the real robot?
-''Submitted batch job 123457''
+===== 22. Deliverables =====
-In that case, ''123457'' is the job ID.
+Submit a short report containing:
-===== Example workflow =====
+    * your implemented reward functions;
+    * a screenshot or copied log section from the baseline run;
+    * a screenshot or copied log section from the improved reward run;
+    * a small comparison table;
+    * short answers to the lab questions.
-Use this sequence for a normal run:
+Example comparison table:
-^ Step ^ Command ^
+^ Run name ^ NUM_ENVS ^ MAX_ITERATIONS ^ Mean reward ^ rew_forward_velocity ^ Observation ^
-| 1. Go to the project folder | ''cd ~/pupper_lab5'' |
+| baseline_zero_reward | 512 | 50 | 0.00 | 0.0000 | No useful learning signal |
-| 2. Submit GPU check | ''sbatch lab5_gpu_check.slurm'' |
+| reward_fixed_512 | 512 | 50 | ... | ... | Reward functions implemented |
-| 3. Check queue | ''squeue -u $USER'' |
+| reward_fixed_1000 | 1000 | 50 | ... | ... | More parallel environments |
-| 4. List logs | ''ls logs'' |
-| 5. Open GPU output | ''cat logs/gpu_check_123456.out'' |
-| 6. Submit training | ''sbatch lab5_train_leggedgym.slurm'' |
-| 7. Check queue again | ''squeue -u $USER'' |
-| 8. Watch training output | ''tail -f logs/pupper_train_123457.out'' |
-| 9. Open final output | ''cat logs/pupper_train_123457.out'' |
-| 10. Open error output | ''cat logs/pupper_train_123457.err'' |
-===== Named training runs =====
+===== 23. What to remember =====
-Use these commands for different experiments:
+The most important idea in this lab is that reinforcement learning does not magically learn the behavior we want.
-^ Experiment ^ Command ^
+The agent learns what the reward function encourages.
-| Short debug run | ''MAX_ITERATIONS=50 RUN_NAME="debug_${USER}" sbatch lab5_train_leggedgym.slurm'' |
-| Velocity-focused run | ''RUN_NAME="velocity_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
-| Effort-focused run | ''RUN_NAME="effort_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
-| Stability-focused run | ''RUN_NAME="stability_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
-| Final run | ''RUN_NAME="final_${USER}" MAX_ITERATIONS=500 sbatch lab5_train_leggedgym.slurm'' |
-| Shorter final run if the queue is busy | ''RUN_NAME="final_${USER}" MAX_ITERATIONS=200 sbatch lab5_train_leggedgym.slurm'' |
-===== Useful repository commands =====
+If the reward function is zero, the robot has no reason to improve.
-Run these commands inside the Legged Gym repository.
+If the reward function encourages forward movement but also penalizes unstable or inefficient behavior, the robot has a better chance of learning useful locomotion.
-^ What you want to do ^ Command ^
+===== 24. Optional final step - Upload the trained policy to the real Pupper robot =====
-| Go to the repository | ''cd ~/pupper_lab5/leggedgym'' |
-| List files | ''ls'' |
-| Find Pupper files | ''find . -iname "    *pupper    *"'' |
-| Find reward functions | ''grep -R "_reward" legged_gym/envs'' |
-| Find Python files containing torque | ''grep -R "torque" legged_gym/envs'' |
-| Find Python files containing velocity | ''grep -R "velocity" legged_gym/envs'' |
-| Find Python files containing height | ''grep -R "height" legged_gym/envs'' |
-| Find logs | ''find logs -type f | head -20'' |
-| Find saved models | ''find . -type f -name "    *.pt"'' |
-| Find videos | ''find . -type f -name "    *.mp4"'' |
-===== Important note about running training =====
+After training a policy in simulation, the next step is to test it on the real Pupper robot.
-Do not run the training command directly on the FEP.
+This step should only be done under instructor supervision.
-Do not run this directly in the terminal:
+Before uploading anything to the real robot, make sure that:
-''python legged_gym/scripts/train.py --task=pupper_flat --num_envs=2000 --max_iterations=300 --headless''
+    * the robot battery is charged;
+    * the robot is placed on the floor in a safe open area;
+    * the emergency stop is available;
+    * the policy was tested in simulation;
+    * the correct configuration file is used on the robot;
+    * the instructor or lab assistant is present.
-Instead, run:
+==== 24.1 Find the trained policy ====
-''sbatch lab5_train_leggedgym.slurm''
+After training, the policy is saved inside the Legged Gym logs directory.
-This sends the training job to a GPU compute node through SLURM.
+Use:
-===== What to submit =====
+<code bash>
+cd ~/pupper_lab8/leggedgym
+find logs -name "    *.pt" | tail -20
+</code>
-Submit one archive containing:
+Look for a file similar to:
-''team_name_lab5.zip''
+<code>
+model_300.pt
+model_1500.pt
+</code>
-Inside the archive, include:
+The exact name depends on the number of training iterations.
-    * ''report.pdf'';
+==== 24.2 Copy the policy to a deployment folder ====
-    * ''slurm_outputs/'';
-    * ''reward_code/'';
-    * ''screenshots/'';
-    * ''logs/'';
-    * ''videos_optional/''.
-Your report must include:
+Go to the deployment repository or folder provided by the instructor.
-    * your team name;
+Example:
-    * the GPU partition used;
-    * the job IDs;
-    * the baseline command;
-    * the reward functions you identified;
-    * the reward changes you made;
-    * the commands used for each run;
-    * screenshots or logs showing training;
-    * the comparison table;
-    * answers about the sim-to-real gap;
-    * final conclusions.
-===== Report questions =====
+<code bash>
+cd ~/pupper_lab8
+mkdir -p deploy_policy
+</code>
-Answer these questions in your report:
+Copy the trained model:
-    * What is the role of the reward function in Reinforcement Learning?
+<code bash>
-    * Why do we train Pupper in simulation before testing on the real robot?
+cp ~/pupper_lab8/leggedgym/logs/<experiment_folder>/model_<iteration>.pt ~/pupper_lab8/deploy_policy/
-    * What does ''--num_envs=2000'' mean?
+</code>
-    * Why is GPU acceleration useful for this experiment?
-    * What happened when you emphasized forward velocity?
+Replace `<experiment_folder>` and `<iteration>` with the real names from your training output.
-    * What happened when you penalized effort?
-    * What happened when you emphasized stability?
+==== 24.3 Convert or rebuild the neural controller ====
-    * Which reward configuration worked best?
-    * Was the best numerical reward also the best behaviour?
+Some Pupper deployment code does not use the raw `.pt` file directly. It may require rebuilding or exporting the neural controller.
+If the deployment folder contains a script such as:
+<code>
+rebuild_neural_controller.py
+</code>
+run it according to the instructor’s instructions.
+Example:
+<code bash>
+cd ~/pupper_lab8/deploy
+python rebuild_neural_controller.py
+</code>
+The exact command may differ depending on the deployment package used in the lab.
+==== 24.4 Upload the controller to Pupper ====
+Connect to the Pupper robot using SSH.
+Example:
+<code bash>
+ssh pi@pupper.local
+</code>
+or, if the robot has a fixed IP address:
+<code bash>
+ssh pi@<PUPPER_IP_ADDRESS>
+</code>
+From your local or HPC environment, copy the generated controller or policy files to the robot:
+<code bash>
+scp -r ~/pupper_lab8/deploy_policy/    * pi@<PUPPER_IP_ADDRESS>:~/pupper_deploy/policies/
+</code>
+Replace `<PUPPER_IP_ADDRESS>` with the real IP address of the robot.
+==== 24.5 Run the policy on the robot ====
+On the Pupper robot:
+<code bash>
+cd ~/pupper_deploy
+python launch.py
+</code>
+or use the command provided by the instructor for the specific robot setup.
+Observe the robot carefully.
+Stop the program immediately if:
+    * the robot moves violently;
+    * the joints oscillate strongly;
+    * the robot falls repeatedly;
+    * the motors overheat;
+    * the emergency stop is needed.
+==== 24.6 Reflection question ====
+Compare the behavior in simulation with the behavior on the real robot.
+Answer:
+    * Did the robot behave the same in simulation and reality?
+    * What differences did you observe?
+    * Why can a policy trained in simulation fail on a real robot?
     * What is the sim-to-real gap?
+    * How could domain randomization help?
-===== Common problems =====
+===== 25. Instructor notes =====
-==== The job stays in the queue ====
+This section is for the instructor or lab assistant.
-Check your jobs with:
+The tested working stack on the HPC cluster was:
-''squeue -u $USER''
+    * SLURM job on the `dgxa100` partition;
+    * NVIDIA A100-SXM4-80GB GPU;
+    * Apptainer with `--nv`;
+    * PyTorch 1.10.0 CUDA 11.3 container;
+    * Isaac Gym Preview 4;
+    * Python 3.7;
+    * Legged Gym;
+    * `pupper_flat` task.
-The cluster may be busy. Wait or reduce the requested time.
+The final import check must confirm:
-==== The CUDA module does not exist ====
+<code>
+isaacgym import: OK
+gymtorch import: OK
+rsl_rl import: OK
+legged_gym import: OK
+CUDA available: True
+</code>
-Check available CUDA modules with:
+The final training script must export the same environment variables used during the successful import check, especially:
-''module avail cuda''
+<code bash>
+export PYTHONUSERBASE=$LAB_DIR/pyuser_isaac
+export TOOL_PREFIX=$LAB_DIR/conda_tools
+export PATH=$TOOL_PREFIX/bin:$PYTHONUSERBASE/bin:$PATH
+export PYTHONPATH=$PYTHONUSERBASE/lib/python3.7/site-packages:${PYTHONPATH:-}
+export LD_LIBRARY_PATH=$TOOL_PREFIX/lib:/opt/conda/lib:${LD_LIBRARY_PATH:-}
+export LD_PRELOAD=$TOOL_PREFIX/lib/libstdc++.so.6:$TOOL_PREFIX/lib/libgcc_s.so.1
+export CPATH=$LAB_DIR/local_include:$TOOL_PREFIX/include:${CPATH:-}
+export CC=$TOOL_PREFIX/bin/x86_64-conda-linux-gnu-gcc
+export CXX=$TOOL_PREFIX/bin/x86_64-conda-linux-gnu-c++
+export TORCH_EXTENSIONS_DIR=$LAB_DIR/torch_extensions
+export MAX_JOBS=1
+</code>
-Choose an available CUDA module and modify the SLURM files.
+The final `lab8_train_isaacgym.slurm` should pass the task variables into Apptainer using:
-==== The script fails with “command not found: module” ====
+<code bash>
+export APPTAINERENV_LAB_DIR="$LAB_DIR"
+export APPTAINERENV_TASK="$TASK"
+export APPTAINERENV_NUM_ENVS="$NUM_ENVS"
+export APPTAINERENV_MAX_ITERATIONS="$MAX_ITERATIONS"
+export APPTAINERENV_RUN_NAME="$RUN_NAME"
+</code>
-You may not be on the correct cluster environment. Reconnect to the FEP and try again.
+A safe starting point for students is:
-==== The job fails immediately ====
+<code bash>
+RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
+</code>
-Check both files:
+A tested larger run is:
-    * ''cat logs/pupper_train_123457.out'';
+<code bash>
-    * ''cat logs/pupper_train_123457.err''.
+RUN_NAME="baseline_zero_reward_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-Replace ''123457'' with your real job ID.
+If `NUM_ENVS=2000` causes an OOM kill, reduce the number of environments or increase the requested memory in the SLURM script.
-The error file usually contains the useful information.
-==== Python cannot import torch ====
-The Python environment is not ready. Activate the correct environment or install the required dependencies.
+=====Manual Install=====
+https://developer.nvidia.com/isaac-gym/download
-==== CUDA is not available in Python ====
+<code>
+#!/bin/bash
+#SBATCH --job-name=lab8_setup
+#SBATCH --partition=dgxa100
+#SBATCH --gres=gpu:1
+#SBATCH --cpus-per-task=8
+#SBATCH --mem=32G
+#SBATCH --time=02:00:00
+#SBATCH --output=lab8_setup_%j.out
+#SBATCH --error=lab8_setup_%j.err
-The training script checks this automatically.
+set -e
-If the output says ''False'' for CUDA availability, check:
+LAB_DIR="$HOME/pupper_lab8"
+mkdir -p "$LAB_DIR"
+cd "$LAB_DIR"
-    * the CUDA module;
+echo "LAB_DIR=$LAB_DIR"
-    * the PyTorch version;
-    * whether the job really received a GPU;
-    * whether you used ''sbatch'', not direct execution on FEP.
-==== The reward file cannot be found ====
+echo "1. Check Apptainer image"
+if [ ! -f pytorch_isaacgym.sif ]; then
+    echo "pytorch_isaacgym.sif not found."
+    echo "Pulling base PyTorch image..."
+    apptainer pull pytorch_isaacgym.sif docker://pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
+else
+    echo "pytorch_isaacgym.sif already exists."
+fi
-Search for it with:
+echo "2. Check Isaac Gym package"
+if [ ! -f IsaacGym_Preview_4_Package.tar.gz ]; then
+    echo "ERROR: IsaacGym_Preview_4_Package.tar.gz is missing."
+    echo "The instructor must upload it manually to $LAB_DIR."
+    exit 1
+fi
-    * ''cd ~/pupper_lab5/leggedgym'';
+if [ ! -d isaacgym ]; then
-    * ''find . -iname "    *pupper    *"'';
+    echo "Extracting Isaac Gym..."
-    * ''grep -R "_reward" legged_gym/envs''.
+    tar -xzf IsaacGym_Preview_4_Package.tar.gz
+else
+    echo "isaacgym folder already exists."
+fi
-==== Training is too slow ====
+echo "3. Clone rsl_rl if missing"
+if [ ! -d rsl_rl ]; then
+    git clone https://github.com/leggedrobotics/rsl_rl.git
+else
+    echo "rsl_rl already exists."
+fi
-Use fewer iterations for testing:
+echo "4. Clone leggedgym if missing"
+if [ ! -d leggedgym ]; then
+    git clone https://github.com/cs123-stanford/leggedgym.git
+else
+    echo "leggedgym already exists."
+fi
-''MAX_ITERATIONS=50 sbatch lab5_train_leggedgym.slurm''
+echo "5. Install Python packages inside Apptainer user base"
-Do not use long runs until the short run works.
+export PYTHONUSERBASE="$LAB_DIR/pyuser_isaac"
+export PATH="$PYTHONUSERBASE/bin:$PATH"
+export PYTHONPATH="$PYTHONUSERBASE/lib/python3.7/site-packages:${PYTHONPATH:-}"
-==== The policy does not walk well ====
+apptainer exec --nv pytorch_isaacgym.sif bash -lc "
+set -e
+cd $LAB_DIR
-That is normal for short training runs. The goal is to understand reward shaping and compare experiments, not necessarily to produce a perfect robot gait in one lab session.
+export PYTHONUSERBASE=$PYTHONUSERBASE
+export PATH=$PATH
+export PYTHONPATH=$PYTHONPATH
-==== The policy works in simulation but not on the real robot ====
+python -m pip install --user --upgrade pip
-Discuss the sim-to-real gap. Do not assume the training failed only because real-world deployment is imperfect.
+python -m pip install --user -e isaacgym/python
+python -m pip install --user -e rsl_rl
+python -m pip install --user -e leggedgym
-===== Final conclusion =====
+python - <<'PY'
+import isaacgym
+import torch
+import rsl_rl
+import legged_gym
-In this lab, you used a Stanford-style Reinforcement Learning workflow to train Pupper in simulation. You submitted GPU jobs through SLURM, inspected reward functions, changed reward terms and compared policies.
+print('isaacgym OK')
+print('torch', torch.__version__)
+print('cuda available', torch.cuda.is_available())
+print('rsl_rl OK')
+print('legged_gym OK')
+PY
+"
-The key idea is that robot learning is not only about running a training script. The quality of the learned behaviour depends strongly on how the reward function is designed.
+echo "Lab 8 setup finished successfully."
-A fast robot is not always a good robot. A stable robot is not always fast. A useful policy balances speed, effort, stability and smoothness.
+</code>

Labs

Resources

rasb/lab/08.1781767742.txt.gz · Last modified: 2026/06/18 10:29 by vlad.radulescu2901

Show page Old revisions

Media Manager Back to top