Show page

Differences

This shows you the differences between two versions of the page.

--- rasb:lab:08 [2026/06/18 10:35]
vlad.radulescu2901 [Required files]
+++ rasb:lab:08 [2026/07/08 19:29] (current)
vlad.radulescu2901 [Manual Install]
@@ Line 1: / Line 1: @@
 ====== Lab 8: Reinforcement Learning for Robotics ======
-===== Lab idea =====
+===== 1. Lab idea =====
-In this lab, you train a quadruped robot, Pupper, to stand and walk using Reinforcement Learning.
+In this lab, you will train a quadruped robot called Pupper in simulation using Reinforcement Learning.
-You do not manually program each leg movement. Instead, you define what “good movement” means through a reward function. The training algorithm runs many simulated Pupper robots in parallel, tests different behaviours and gradually improves the neural policy.
+The robot will be trained inside NVIDIA Isaac Gym, a GPU-based physics simulator that can run many environments in parallel. Instead of training one robot at a time, we can train hundreds or thousands of simulated robots at the same time.
-The main idea is:
+The learning algorithm used in this lab is PPO - Proximal Policy Optimization.
-    * many simulated Pupper robots run in parallel;
+The main goal of the lab is not to install Isaac Gym. The simulator and training pipeline are already prepared. Your task is to complete the reward functions used by the Pupper robot.
-    * each simulated robot collects experience;
-    * the reward function evaluates the behaviour;
-    * PPO updates the neural policy;
-    * the policy gradually learns better movement.
-This lab uses a Stanford-style Reinforcement Learning workflow based on Legged Gym / Isaac Gym. The training runs on the UPB GPU cluster through SLURM.
+Initially, the reward functions return zero, so the robot has no useful learning signal. You will implement reward terms that encourage the robot to:
-===== What you need to understand first =====
+    * move forward;
+    * keep a stable body height;
+    * avoid using unnecessarily large motor torques.
-There are two different parts of the Pupper workflow:
+At the end of the lab, you will compare the training results before and after modifying the reward function.
-    * training a policy in simulation;
+===== 2. Learning objectives =====
-    * deploying an already trained policy on the real robot.
-These are not the same thing.
+After this lab, you should be able to:
-The deploy repository may contain files such as:
+    * explain why reinforcement learning needs a reward function;
+    * understand why many simulated environments are used in parallel;
+    * run a training job on the HPC cluster using SLURM;
+    * identify and modify reward functions in a Legged Gym environment;
+    * compare multiple training runs using logs;
+    * explain how reward shaping influences the behavior learned by a robot;
+    * understand the basic idea of transferring a trained policy from simulation to the real Pupper robot.
-    * ''config.yaml'';
+===== 3. Background =====
-    * ''launch.py'';
-    * ''estop_controller.cpp'';
-    * ''deploy.py'';
-    * ''rebuild_neural_controller.py'';
-    * ''parkour_policy.json'';
-    * ''test_policy.json''.
-Those files are used for loading and deploying trained policies. They are not the full Reinforcement Learning training environment.
+A reinforcement learning agent learns by interacting with an environment.
-Do not modify the reward function in ''config.yaml''. That file points the controller to an already trained policy.
+For a robot, the environment contains:
-For this lab, you need the training environment. The reward functions are edited in the Legged Gym training code, usually in files related to the Pupper environment.
+    * the robot body;
+    * the physics simulation;
+    * gravity;
+    * contacts with the ground;
+    * joint positions and velocities;
+    * actions applied to the motors.
-===== What you will do =====
+At every step, the agent receives an observation and outputs an action. The simulator applies the action and returns a reward.
-In this lab, you will:
+The reward tells the agent whether its behavior is good or bad.
-    * connect to the UPB HPC cluster;
+For example:
-    * submit a GPU job through SLURM;
-    * run or inspect the Pupper training environment;
-    * locate the reward functions;
-    * train a baseline policy;
-    * modify reward terms;
-    * run a new training experiment;
-    * compare the results;
-    * explain the trade-off between speed, stability and effort.
-===== Required files =====
+    * if the robot moves forward, it should receive a positive reward;
+    * if it falls, it should receive a penalty;
+    * if it uses too much torque, it should receive a penalty;
+    * if it keeps a stable body height, it should receive a better score.
-Download the Lab 8 HPC pack:
+A bad reward function can make the robot learn nothing. A good reward function can make the robot learn useful locomotion.
-{{ :rasb:lab:lab8_hpc_pack.zip | Download lab8_hpc_pack.zip }}
+===== 4. Important note about this lab =====
-After downloading it, upload or copy the archive to the FEP and extract it inside your Lab 5 project folder.
+The HPC environment, container, Isaac Gym, PyTorch and Legged Gym are already prepared for you.
-Use the following commands on the FEP:
+You should not try to reinstall Isaac Gym manually during the lab.
-^ Step ^ Command ^
+The important part of this lab is inside the file:
-| Create the project folder | ''mkdir -p ~/pupper_lab8'' |
-| Go to the project folder | ''cd ~/pupper_lab8'' |
-| Upload or copy the archive here | ''ls'' |
-| Extract the archive | ''unzip lab8_hpc_pack.zip'' |
-| Enter the extracted folder | ''cd lab5_hpc_pack'' |
-| List the files | ''ls'' |
-After extracting it, you should see:
+<code>
+~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
+</code>
-^ File ^ Purpose ^
+The initial version contains TODO functions similar to this:
-| ''README.txt'' | Short explanation of the pack |
-| ''lab5_gpu_check.slurm'' | Checks whether your GPU job works |
-| ''setup_leggedgym_env.sh'' | Prepares the project folder and clones the repositories |
-| ''lab5_train_leggedgym.slurm'' | Submits a Pupper training job |
-| ''lab5_play_leggedgym.slurm'' | Tests or visualizes a trained policy |
-| ''lab5_hpc_ocw_section.txt'' | Extra OCW text section, if needed |
-===== Part 1 - Connect to the UPB HPC cluster =====
+<code python>
+def _reward_base_height(self):
+    return 0.0
-Use SSH to connect to the FEP.
+def _reward_forward_velocity(self):
+return 0
-^ Step ^ Command ^
+def _reward_torques(self):
-| Connect to the FEP | ''ssh -X -o ServerAliveInterval=100 [user.name@fep.grid.pub.ro]'' |
+return 0 </code>
-Replace ''user.name'' with your faculty account.
+As long as these functions return zero, the robot has no meaningful learning signal.
-Important: do not run the training directly on the FEP. The FEP is only the access node. Training must be submitted as a SLURM job.
+===== 5. Files used in this lab =====
-===== Part 2 - Check the available GPU partitions =====
+Download the starter archive from OCW:
-After connecting, check the available partitions.
+{{ :rasb:lab:lab8_hpc_pack.zip | Download Lab 8 HPC starter pack }}
-^ What you want to do ^ Command ^
+The starter pack contains the SLURM scripts needed for running the training jobs.
-| Check available partitions | ''sinfo -o "%10P %55N %8c %10m %20G"'' |
-| Check all partition information | ''sinfo'' |
-| Check jobs on the A100 partition | ''squeue -p dgxa100'' |
-| Check jobs on the H100 partition | ''squeue -p dgxh100'' |
-Look for GPU partitions such as:
+Expected working directory:
-    * ''dgxa100'';
+<code>
-    * ''dgxh100'';
+~/pupper_lab8
-    * ''ucsx'';
+</code>
-    * ''ml''.
-For this lab, use one GPU per job.
+Expected repository structure:
-If an A100 partition is available, prefer it for the old Stanford-style Legged Gym workflow. H100 is more powerful, but older simulator stacks can be more sensitive to software compatibility.
+<code>
+~/pupper_lab8/
+├── isaacgym/
+├── leggedgym/
+├── rsl_rl/
+├── pytorch_isaacgym.sif
+├── pyuser_isaac/
+├── conda_tools/
+├── local_include/
+├── torch_extensions/
+├── logs/
+├── lab8_gpu_check.slurm
+├── lab8_import_check.slurm
+├── lab8_train_isaacgym.slurm
+└── lab8_reward_check.sh
+</code>
-===== Part 3 - Check available modules =====
+The archive contains only the small helper scripts. It does not contain the large simulator files, the container image or the full repositories.
-Check the available software modules.
+===== 6. Connect to the HPC cluster =====
-^ What you want to do ^ Command ^
+Connect to the frontend node:
-| Show all available modules | ''module avail'' |
-| Search for CUDA modules | ''module avail cuda'' |
-| Show currently loaded modules | ''module list'' |
-| Clear loaded modules | ''module purge'' |
-| Load CUDA 12.3 | ''module load libraries/cuda-12.3'' |
-| Check loaded modules again | ''module list'' |
-If ''libraries/cuda-12.3'' is not available, use ''module avail cuda'' and choose the CUDA module available on the cluster.
+<code bash>
+ssh your_username@fep.grid.pub.ro
+</code>
-===== Part 4 - Create your project folder =====
+Go to the lab directory:
-Create a working folder on the FEP.
+<code bash>
+cd ~/pupper_lab8
+</code>
-^ Step ^ Command ^
+Create the logs directory if it does not already exist:
-| Create the project folder | ''mkdir -p ~/pupper_lab5'' |
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Create the logs folder | ''mkdir -p logs'' |
-| Check the folder content | ''ls'' |
-Copy or upload the files from ''lab5_hpc_pack.zip'' into ''~/pupper_lab5''.
+<code bash>
+mkdir -p logs
+</code>
-After copying the files, check that they are there.
+===== 7. Check that the GPU is available =====
-^ What you want to check ^ Command ^
+Before running Isaac Gym, check that your SLURM job can access the GPU.
-| List files | ''ls'' |
-| Make setup script executable | ''chmod +x setup_leggedgym_env.sh'' |
-| Make GPU check file executable | ''chmod +x lab5_gpu_check.slurm'' |
-| Make training file executable | ''chmod +x lab5_train_leggedgym.slurm'' |
-| Make play file executable | ''chmod +x lab5_play_leggedgym.slurm'' |
-You should see:
+Submit the GPU check job:
-    * ''README.txt'';
+<code bash>
-    * ''lab5_gpu_check.slurm'';
+cd ~/pupper_lab8
-    * ''setup_leggedgym_env.sh'';
+sbatch lab8_gpu_check.slurm
-    * ''lab5_train_leggedgym.slurm'';
+</code>
-    * ''lab5_play_leggedgym.slurm'';
-    * ''lab5_hpc_ocw_section.txt''.
-===== Part 5 - Test that your GPU job works =====
+Check the queue:
-Before installing or running the training environment, test that you can submit a GPU job.
+<code bash>
+squeue -u $USER
+</code>
-^ Step ^ Command ^
+After the job finishes, inspect the output:
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Submit the GPU check job | ''sbatch lab5_gpu_check.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-| List the log files | ''ls logs'' |
-After you submit the job, SLURM prints something similar to:
+<code bash>
+ls logs
+cat logs/gpu_check_<JOB_ID>.out
+cat logs/gpu_check_<JOB_ID>.err
+</code>
-''Submitted batch job 123456''
+Replace `<JOB_ID>` with the job id printed by `sbatch`.
-The number ''123456'' is an example job ID. Your job ID will be different.
+A successful run should show an NVIDIA GPU through `nvidia-smi`.
-When the job finishes, inspect the logs.
+===== 8. Check that Isaac Gym and Legged Gym work =====
-^ What you want to do ^ Example command ^
+Before changing the reward function, check that the simulator imports correctly.
-| Open the GPU check output | ''cat logs/gpu_check_123456.out'' |
-| Open the GPU check error file | ''cat logs/gpu_check_123456.err'' |
-Replace ''123456'' with your real job ID.
+Submit the import check job:
-The output should show information from:
+<code bash>
+cd ~/pupper_lab8
+sbatch lab8_import_check.slurm
+</code>
-    * ''nvidia-smi'';
+After the job finishes:
-    * ''python3 --version'';
-    * ''nvcc --version''.
-If this job fails, do not continue to the training part. Fix the GPU job first.
+<code bash>
+cat logs/import_check_<JOB_ID>.out
+cat logs/import_check_<JOB_ID>.err
+</code>
-===== Part 6 - Prepare the training environment =====
+A successful import check should contain:
-The training environment is based on Legged Gym.
+<code>
+isaacgym import: OK
+gymtorch import: OK
+rsl_rl import: OK
+legged_gym import: OK
+CUDA available: True
+</code>
-^ Step ^ Command ^
+If this step fails, do not continue to the reward task. Ask the instructor or lab assistant for help.
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Run the setup script | ''./setup_leggedgym_env.sh'' |
-| Check the project folder | ''ls ~/pupper_lab5'' |
-The setup script creates a Python virtual environment and clones the required repositories.
+===== 9. Run a small debug training job =====
-You should get folders similar to:
+Now check that the Pupper training task can start.
-    * ''~/pupper_lab5/leggedgym'';
+Submit a small debug job:
-    * ''~/pupper_lab5/rsl_rl'';
-    * ''~/pupper_lab5/.venv'';
-    * ''~/pupper_lab5/logs''.
-Important: this script does not automatically install Isaac Gym. The old Stanford-style workflow may require Isaac Gym Preview and specific Python / PyTorch / CUDA versions.
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
+</code>
-If the instructor provides a prepared environment, container or module, use that instead of manually installing everything.
+Check the queue:
-===== Part 7 - Understand the simulator =====
+<code bash>
+squeue -u $USER
+</code>
-The simulator is not ''lab_5_fall_2025''.
+After the job finishes, check its output:
-For this lab, the simulator is the Legged Gym training environment. This environment runs many simulated robots in parallel on the GPU.
+<code bash>
+cat logs/pupper_train_<JOB_ID>.out
+cat logs/pupper_train_<JOB_ID>.err
+</code>
-The important argument is:
+A successful run should contain messages similar to:
-''--num_envs=2000''
+<code>
+Physics Engine: PhysX
+Physics Device: cuda:0
+GPU Pipeline: enabled
+Learning iteration 0/10
+Learning iteration 9/10
+Training finished.
+</code>
-This means that the training process uses many parallel simulated Pupper environments. Each simulated Pupper collects experience, and the PPO algorithm uses that experience to update the policy.
+This means that Isaac Gym, PyTorch, CUDA and the Pupper environment are working.
-A simplified view:
+===== 10. Run the initial baseline =====
-    * Pupper 1: observation -> action -> reward;
+Now run a slightly larger baseline:
-    * Pupper 2: observation -> action -> reward;
-    * Pupper 3: observation -> action -> reward;
-    * ...
-    * Pupper 2000: observation -> action -> reward;
-    * the collected data is combined;
-    * the policy is updated.
-This is why a GPU is needed.
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="baseline_zero_reward_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-===== Part 8 - Locate the Pupper environment =====
+After the job finishes, inspect the reward values:
-Go to the Legged Gym repository and search for the Pupper files.
+<code bash>
+cat logs/pupper_train_<JOB_ID>.out | grep -E "Learning iteration|Mean reward|rew_forward_velocity|episode length" | tail -80
+</code>
-^ What you want to do ^ Command ^
+You should observe that the reward is not useful. In the initial version, the relevant reward functions return zero.
-| Go to the repository | ''cd ~/pupper_lab5/leggedgym'' |
-| List files | ''ls'' |
-| Search for Pupper files | ''find . -iname "    *pupper    *"'' |
-| Search for reward functions | ''grep -R "_reward" legged_gym/envs'' |
-| Search for velocity-related code | ''grep -R "velocity" legged_gym/envs'' |
-| Search for torque-related code | ''grep -R "torque" legged_gym/envs'' |
-| Search for height-related code | ''grep -R "height" legged_gym/envs'' |
-You are looking for files similar to:
+Example:
-    * ''legged_gym/envs/pupper/pupper.py'';
+<code>
-    * ''legged_gym/envs/pupper/pupper_config.py''.
+Mean reward: 0.00
+Mean episode rew_forward_velocity: 0.0000
+</code>
-The exact path may differ depending on the repository version.
+This is expected before solving the lab.
-===== Part 9 - Understand reward functions =====
+===== 11. Inspect the reward functions =====
-A reward function tells the robot what behaviour is good.
+Open the Pupper environment file:
-For example, a walking robot can receive:
+<code bash>
+cd ~/pupper_lab8/leggedgym
+nano legged_gym/envs/pupper/pupper.py
+</code>
-    * positive reward for moving forward;
+Find the following functions:
-    * negative reward for using too much torque;
-    * positive reward for staying balanced;
-    * negative reward for falling;
-    * negative reward for sudden, unstable movements.
-Common reward terms include:
+<code python>
+def _reward_base_height(self):
+    return 0.0
-    * forward velocity;
+def _reward_forward_velocity(self):
-    * base height;
+return 0
-    * torques;
-    * orientation;
-    * stability;
-    * smoothness;
-    * termination / fall penalty.
-A reward function is not just about making the robot fast. If you reward only speed, the robot may move aggressively, fall or use too much energy.
+def _reward_torques(self):
+return 0 </code>
-A good policy balances:
+These functions are the main TODOs of this lab.
-    * speed;
+You can also use the helper script:
-    * stability;
-    * energy efficiency;
-    * smoothness;
-    * command following.
-===== Part 10 - Inspect the baseline training script =====
+<code bash>
+cd ~/pupper_lab8
+./lab8_reward_check.sh
+</code>
-Open the training SLURM file.
+This script prints the reward functions and checks whether there are still `return 0` statements inside `pupper.py`.
-^ What you want to do ^ Command ^
+===== 12. Inspect the reward configuration =====
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Open the training file | ''nano lab5_train_leggedgym.slurm'' |
-Look for these variables:
+Open the configuration file:
-^ Variable ^ Meaning ^
+<code bash>
-| ''TASK="${TASK:-pupper_flat}"'' | The training task. The default task is ''pupper_flat''. |
+cd ~/pupper_lab8/leggedgym
-| ''NUM_ENVS="${NUM_ENVS:-2000}"'' | The number of parallel simulated environments. |
+nano legged_gym/envs/pupper/pupper_config.py
-| ''MAX_ITERATIONS="${MAX_ITERATIONS:-300}"'' | The number of training iterations. |
+</code>
-| ''RUN_NAME="${RUN_NAME:-team_${SLURM_JOB_ID}_pupper_flat}"'' | The experiment name. |
-For a first test, keep the default values.
+Look for the reward section. It should contain values similar to:
-===== Part 11 - Run a short baseline training job =====
+<code python>
+class rewards:
+    forward_velocity_clip = 1.0
-Submit the training job.
+```
+class scales:
+    forward_velocity = 3.0
+```
-^ Step ^ Command ^
+</code>
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Submit the training job | ''sbatch lab5_train_leggedgym.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-After you submit the job, SLURM prints something similar to:
+The exact values may differ depending on the starter code version.
-''Submitted batch job 123457''
+The important idea is that the config file contains the coefficients used to scale the reward terms.
-The number ''123457'' is an example. Your job ID will be different.
+For example:
-Use your real job ID to check the logs.
+    * a positive scale means the term increases the reward;
+    * a negative scale means the term becomes a penalty;
+    * a zero scale disables the term.
-^ What you want to do ^ Example command ^
+===== 13. Task 1 - Implement forward velocity reward =====
-| Watch the output live | ''tail -f logs/pupper_train_123457.out'' |
-| Open the output file | ''cat logs/pupper_train_123457.out'' |
-| Open the error file | ''cat logs/pupper_train_123457.err'' |
-| Stop watching a live log | ''CTRL + C'' |
-| Cancel the job if needed | ''scancel 123457'' |
-Replace ''123457'' with your real job ID.
+The robot should receive a positive reward when it moves forward.
-After the job finishes, check the training logs.
+In Legged Gym, the forward velocity of the robot base is usually stored in:
-^ What you want to do ^ Command ^
+<code python>
-| Go to the Legged Gym repository | ''cd ~/pupper_lab5/leggedgym'' |
+self.base_lin_vel[:, 0]
-| Find generated log files | ''find logs -maxdepth 3 -type f | head'' |
+</code>
-| Find saved models | ''find . -type f -name "    *.pt"'' |
-| Find videos | ''find . -type f -name "    *.mp4"'' |
-| Find images | ''find . -type f -name "    *.png"'' |
-Save the output files. You need them for the report.
+This is the x-axis linear velocity of the robot base.
-===== Part 12 - Run a very short debugging experiment =====
+Replace the initial function:
-Before changing the reward function, run a very short experiment to make sure the pipeline works.
+<code python>
+def _reward_forward_velocity(self):
+    return 0
+</code>
-^ What you want to do ^ Command ^
+with:
-| Go to the project folder | ''cd ~/pupper_lab5'' |
-| Run a debug experiment | ''MAX_ITERATIONS=50 RUN_NAME="debug_${USER}" sbatch lab5_train_leggedgym.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-| List logs | ''ls logs'' |
-This run is not meant to produce a good policy. It is only a debugging run.
+<code python>
+def _reward_forward_velocity(self):
+    return torch.clip(
+        self.base_lin_vel[:, 0],
+        min=0.0,
+        max=self.cfg.rewards.forward_velocity_clip
+    )
+</code>
-Check:
+This reward gives positive values only when the robot moves forward. Negative velocity is clipped to zero, so moving backwards is not rewarded.
-    * Did the job start?
+If `torch` is not imported at the top of the file, add:
-    * Was the GPU detected?
-    * Did Python load the environment?
-    * Did training begin?
-    * Were logs created?
-If the debug run fails, fix the environment before changing the reward function.
+<code python>
+import torch
+</code>
-===== Part 13 - Mission 1: Identify the reward terms =====
+===== 14. Task 2 - Implement torque penalty =====
-Open the Pupper environment file.
+A robot should not learn to move by using extremely large motor torques. Large torques are inefficient and can produce unstable behavior.
-^ What you want to do ^ Command ^
+The torque values are stored in:
-| Go to the Legged Gym repository | ''cd ~/pupper_lab5/leggedgym'' |
-| Search for Pupper files | ''find . -iname "    *pupper    *"'' |
-| Search for reward functions | ''grep -R "_reward" legged_gym/envs'' |
-| Open a file with nano | ''nano legged_gym/envs/pupper/pupper.py'' |
-If the file path is different, use the result from ''find . -iname "    *pupper    *"''.
+<code python>
+self.torques
+</code>
-Identify at least three reward functions.
+Replace:
-Complete this table in your report:
+<code python>
+def _reward_torques(self):
+    return 0
+</code>
-^ Reward term ^ Reward or penalty? ^ What behaviour does it encourage or discourage? ^
+with:
-| forward velocity | | |
-| torque / effort | | |
-| base height | | |
-| stability / orientation | | |
-| smoothness | | |
-Write short explanations. Do not copy code without explaining it.
+<code python>
+def _reward_torques(self):
+    return torch.sum(torch.square(self.torques), dim=1)
+</code>
-===== Part 14 - Mission 2: Velocity-focused policy =====
+This function returns a positive value representing how much torque the robot uses.
-In this experiment, focus on forward movement.
+Important: this function returns a positive value, but it becomes a penalty if the scale in the config file is negative.
-Find the reward term related to forward velocity.
+For example:
-Possible names include:
+<code python>
+torques = -0.0002
+</code>
-    * ''_reward_forward_velocity'';
+means that large torques reduce the final reward.
-    * ''tracking_lin_vel'';
-    * ''lin_vel_x'';
-    * ''forward_velocity''.
-Increase the importance of forward movement.
+===== 15. Task 3 - Implement base height penalty =====
-Do not delete the other reward terms. You still need the robot to stay stable.
+The robot should keep its body at a reasonable height. If the body is too low or too high, the behavior is probably unstable.
-Run a new job.
+The base height is stored in:
-^ What you want to do ^ Command ^
+<code python>
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+self.root_states[:, 2]
-| Run the velocity experiment | ''RUN_NAME="velocity_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
+</code>
-| Check your jobs | ''squeue -u $USER'' |
-| List logs | ''ls logs'' |
-After the job finishes, save:
+Replace:
-    * the job ID;
+<code python>
-    * the SLURM output file;
+def _reward_base_height(self):
-    * the modified reward code;
+    return 0.0
-    * the training logs;
+</code>
-    * any generated curve or video.
-Answer:
+with:
-    * Did the robot become faster?
+<code python>
-    * Did it become less stable?
+def _reward_base_height(self):
-    * Did the reward increase?
+    base_height = self.root_states[:, 2]
-    * Was the movement natural or aggressive?
+    return torch.square(base_height - self.cfg.rewards.base_height_target)
+</code>
-===== Part 15 - Mission 3: Effort penalty =====
+This function returns the squared error between the current base height and the target base height.
-In this experiment, focus on reducing effort.
+Again, this becomes a penalty if its scale in the config file is negative.
-Find the reward term related to torque or effort.
+===== 16. Check your code =====
-Possible names include:
+After editing `pupper.py`, check the relevant lines:
-    * ''_reward_torques'';
+<code bash>
-    * ''torque_penalty'';
+cd ~/pupper_lab8/leggedgym
-    * ''energy'';
+nl -ba legged_gym/envs/pupper/pupper.py | sed -n '80,120p'
-    * ''effort''.
+</code>
-Increase the penalty for using too much torque.
+You should see the implemented reward functions, not `return 0`.
-Run a new job.
+You can also run:
-^ What you want to do ^ Command ^
+<code bash>
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+cd ~/pupper_lab8
-| Run the effort experiment | ''RUN_NAME="effort_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
+./lab8_reward_check.sh
-| Check your jobs | ''squeue -u $USER'' |
+</code>
-| List logs | ''ls logs'' |
-Compare this result with the velocity-focused run.
+If the script still shows `return 0` inside the reward functions, your implementation is not complete.
-Answer:
+===== 17. Train again after implementing the rewards =====
-    * Did the movement become smoother?
+Run the training job again:
-    * Did the robot become slower?
-    * Did the reward improve or decrease?
-    * What trade-off did you observe?
-===== Part 16 - Mission 4: Stability-focused policy =====
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_512_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-In this experiment, focus on stability.
+Check the job:
-Find reward terms related to:
+<code bash>
+squeue -u $USER
+</code>
-    * base height;
+After it finishes:
-    * orientation;
-    * falling;
-    * body tilt;
-    * smoothness.
-Increase the importance of stability.
+<code bash>
+cat logs/pupper_train_<JOB_ID>.out | grep -E "Learning iteration|Mean reward|rew_forward_velocity|episode length" | tail -100
+cat logs/pupper_train_<JOB_ID>.err
+</code>
-Run a new job.
+Compare the new output with the initial baseline.
+You should focus on:
+    * `Mean reward`;
+    * `Mean episode rew_forward_velocity`;
+    * `Mean episode length`;
+    * total timesteps;
+    * whether the job completed successfully.
+===== 18. Larger training runs =====
+After the small run works, you can try larger experiments.
+Run with 1000 environments:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_1000_${USER}" NUM_ENVS=1000 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
+If that works, try 2000 environments:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_2000_${USER}" NUM_ENVS=2000 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
+For a longer training run:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_long_${USER}" NUM_ENVS=2000 MAX_ITERATIONS=300 sbatch lab8_train_isaacgym.slurm
+</code>
+Do not start with the largest experiment. First check that the short run works.
+===== 19. Useful SLURM commands =====
 ^ What you want to do ^ Command ^
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+| Submit GPU check | ''sbatch lab8_gpu_check.slurm'' |
-| Run the stability experiment | ''RUN_NAME="stability_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
+| Submit import check | ''sbatch lab8_import_check.slurm'' |
+| Submit training | ''sbatch lab8_train_isaacgym.slurm'' |
+| Submit debug training | ''RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm'' |
 | Check your jobs | ''squeue -u $USER'' |
-| List logs | ''ls logs'' |
+| Check finished job status | ''sacct -j <JOB_ID> --format=JobID,JobName,Partition,State,Elapsed,ExitCode,MaxRSS,ReqMem'' |
+| Show output log | ''cat logs/pupper_train_<JOB_ID>.out'' |
+| Show error log | ''cat logs/pupper_train_<JOB_ID>.err'' |
+| Show only reward lines | ''cat logs/pupper_train_<JOB_ID>.out | grep -E "Mean reward|rew_forward_velocity|episode length"'' |
+| Inspect reward functions | ''./lab8_reward_check.sh'' |
-Answer:
+===== 20. Common problems =====
-    * Did the robot fall less often?
+==== Problem 1 - The reward stays zero ====
-    * Did the movement become slower?
-    * Did the robot keep a better posture?
-    * Was this policy better than the velocity-focused one?
-===== Part 17 - Mission 5: Your final reward configuration =====
+If the output still shows:
-Create your own reward configuration.
+<code>
+Mean reward: 0.00
+Mean episode rew_forward_velocity: 0.0000
+</code>
-Your goal is to balance:
+check that you actually modified:
-    * forward walking;
+<code>
-    * stability;
+~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
-    * effort;
+</code>
-    * smoothness.
-Run a final experiment.
+and that the reward functions no longer return zero.
-^ Situation ^ Command ^
+Use:
-| Normal final run | ''RUN_NAME="final_${USER}" MAX_ITERATIONS=500 sbatch lab5_train_leggedgym.slurm'' |
-| Shorter final run if the queue is busy | ''RUN_NAME="final_${USER}" MAX_ITERATIONS=200 sbatch lab5_train_leggedgym.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-| List logs | ''ls logs'' |
-Save the final reward configuration.
+<code bash>
+cd ~/pupper_lab8
+./lab8_reward_check.sh
+</code>
-Explain:
+or:
-    * what you changed;
+<code bash>
-    * why you changed it;
+grep -R "return 0" ~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
-    * what you expected;
+</code>
-    * what actually happened.
-===== Part 18 - Test or visualize a trained policy =====
+==== Problem 2 - The job is killed with OOM ====
-Use the play script only after you have a trained policy or checkpoint.
+If the job fails with:
-^ Step ^ Command ^
+<code>
-| Go to the project folder | ''cd ~/pupper_lab5'' |
+Detected 1 oom_kill event
-| Submit the play job | ''sbatch lab5_play_leggedgym.slurm'' |
+Some of the step tasks have been OOM Killed
-| Check your jobs | ''squeue -u $USER'' |
+</code>
-| List logs | ''ls logs'' |
-After the job finishes, check the output.
+then the job used too much RAM.
-^ What you want to do ^ Example command ^
+Use fewer environments:
-| Open the play output | ''cat logs/pupper_play_123458.out'' |
-| Open the play error file | ''cat logs/pupper_play_123458.err'' |
-| Find generated files | ''cd ~/pupper_lab5/leggedgym && find logs -type f | head -20'' |
-| Search for videos | ''cd ~/pupper_lab5/leggedgym && find . -type f -name "    *.mp4"'' |
-Replace ''123458'' with your real job ID.
+<code bash>
+NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-If no video is generated, use the training logs and SLURM output for your report.
+or:
-===== Part 19 - Compare your experiments =====
+<code bash>
+NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
+</code>
-Compare at least three runs:
+If needed, the SLURM memory limit can be increased by the instructor in the training script:
-    * baseline or debug run;
+<code bash>
-    * velocity-focused run;
+#SBATCH --mem=128G
-    * effort or stability-focused run;
+</code>
-    * final run.
-Complete the table:
+==== Problem 3 - Task not registered ====
-^ Run name ^ Main reward change ^ Iterations ^ Result observed ^ Main problem ^
+If you see:
-| baseline/debug | none | | | |
-| velocity | increased forward movement reward | | | |
-| effort | increased effort penalty | | | |
-| stability | increased stability terms | | | |
-| final | custom configuration | | | |
-Then answer:
+<code>
+ValueError: Task with name: ... was not registered
+</code>
-    * Which run had the best reward?
+check that the task name is:
-    * Which run looked most stable?
-    * Which run looked most natural?
-    * Did the best numerical reward also produce the best movement?
-    * Which reward term had the strongest visible effect?
-===== Part 20 - Understand the sim-to-real gap =====
+<code>
+pupper_flat
+</code>
-A policy trained in simulation may not work perfectly on the real robot. This difference is called the sim-to-real gap.
+You can inspect registered tasks with:
-Possible causes:
+<code bash>
+cd ~/pupper_lab8/leggedgym
+grep -R "task_registry.register" -n legged_gym/envs
+</code>
-    * simulated motors are not identical to real motors;
+==== Problem 4 - You edited the wrong task ====
-    * friction is different in the real world;
-    * the floor surface may be different;
-    * the battery level affects the robot;
-    * sensors are noisy;
-    * real hardware has delays;
-    * the simulated mass may not perfectly match the real robot.
-Answer:
+This lab uses:
-    * Why is simulation useful before testing on hardware?
+<code>
-    * Why can a policy fail when moved from simulation to the real robot?
+pupper_flat
-    * What could make the policy more robust?
+</code>
-    * How could domain randomization help?
-===== Part 21 - Optional: Deploy on real Pupper =====
+Do not confuse it with:
-Do this only if the instructor confirms that the physical Pupper robot is ready.
+<code>
+pupper_standup
+</code>
-The deploy repository is used only after the policy has been trained.
+The standup task may have different reward scales, including zero forward velocity reward.
-On the Raspberry Pi of the robot, the deploy repository may contain:
+==== Problem 5 - The output stops after gymtorch ====
-    * ''config.yaml'';
+If the output stops around:
-    * ''launch.py'';
-    * ''deploy.py'';
-    * ''rebuild_neural_controller.py'';
-    * ''policy.json''.
-Run deployment scripts only on the robot's Raspberry Pi, not on the FEP and not on your laptop.
+<code>
+Building extension module gymtorch...
+ninja: no work to do.
+</code>
-If you see paths such as ''/home/pi/...'', that script is meant for the Raspberry Pi.
+check the error file:
-Do not run robot deployment scripts on the HPC cluster.
+<code bash>
+cat logs/pupper_train_<JOB_ID>.err
+</code>
-===== Useful SLURM commands =====
+If the job completed successfully, use:
-Use this section when you work on the HPC cluster.
+<code bash>
+sacct -j <JOB_ID> --format=JobID,JobName,Partition,State,Elapsed,ExitCode,MaxRSS,ReqMem
+</code>
-^ What you want to do ^ Command ^
+===== 21. Questions =====
-| Submit the GPU check job | ''sbatch lab5_gpu_check.slurm'' |
-| Submit the training job | ''sbatch lab5_train_leggedgym.slurm'' |
-| Submit the play / test job | ''sbatch lab5_play_leggedgym.slurm'' |
-| Check your jobs | ''squeue -u $USER'' |
-| Check jobs on the A100 partition | ''squeue -p dgxa100'' |
-| Check jobs on the H100 partition | ''squeue -p dgxh100'' |
-| Cancel a job | ''scancel 123457'' |
-| Check information about a finished job | ''sacct -j 123457'' |
-| Check detailed job information | ''sacct -j 123457 --format=JobID,JobName,Partition,State,Elapsed,MaxRSS,AllocGRES'' |
-| List log files | ''ls logs'' |
-| Open GPU check output | ''cat logs/gpu_check_123456.out'' |
-| Open GPU check error file | ''cat logs/gpu_check_123456.err'' |
-| Open training output | ''cat logs/pupper_train_123457.out'' |
-| Open training error file | ''cat logs/pupper_train_123457.err'' |
-| Watch a running training log | ''tail -f logs/pupper_train_123457.out'' |
-| Show the last lines of a training log | ''tail logs/pupper_train_123457.out'' |
-| Show the last 50 lines of a training log | ''tail -n 50 logs/pupper_train_123457.out'' |
-| Check available partitions | ''sinfo'' |
-| Check partitions in compact format | ''sinfo -o "%10P %55N %8c %10m %20G"'' |
-| Check available modules | ''module avail'' |
-| Search for CUDA modules | ''module avail cuda'' |
-| Clear loaded modules | ''module purge'' |
-| Load CUDA | ''module load libraries/cuda-12.3'' |
-| Check loaded modules | ''module list'' |
-| Check GPU information | ''nvidia-smi'' |
-| Search for Pupper files | ''find . -iname "    *pupper    *"'' |
-| Search for reward functions | ''grep -R "_reward" legged_gym/envs'' |
-| Find generated files in logs | ''find logs -type f | head -20'' |
-| Search for saved models | ''find . -type f -name "    *.pt"'' |
-| Search for videos | ''find . -type f -name "    *.mp4"'' |
-| Search for images | ''find . -type f -name "    *.png"'' |
-The numbers ''123456'' and ''123457'' are examples. Replace them with your real job ID.
+Answer the following questions in your report:
-After submitting a job with ''sbatch'', SLURM prints something like:
+    * Why did the initial training run produce zero reward?
+    * Why is forward velocity a useful reward term for locomotion?
+    * Why should large torques be penalized?
+    * Why can body height be used as a stability-related reward term?
+    * What is the purpose of training many environments in parallel?
+    * What changed after you implemented the reward functions?
+    * Did the robot learn better behavior after more iterations? Explain using the log values.
+    * Why might a policy that works in simulation behave differently on the real robot?
-''Submitted batch job 123457''
+===== 22. Deliverables =====
-In that case, ''123457'' is the job ID.
+Submit a short report containing:
-===== Example workflow =====
+    * your implemented reward functions;
+    * a screenshot or copied log section from the baseline run;
+    * a screenshot or copied log section from the improved reward run;
+    * a small comparison table;
+    * short answers to the lab questions.
-Use this sequence for a normal run:
+Example comparison table:
-^ Step ^ Command ^
+^ Run name ^ NUM_ENVS ^ MAX_ITERATIONS ^ Mean reward ^ rew_forward_velocity ^ Observation ^
-| 1. Go to the project folder | ''cd ~/pupper_lab5'' |
+| baseline_zero_reward | 512 | 50 | 0.00 | 0.0000 | No useful learning signal |
-| 2. Submit GPU check | ''sbatch lab5_gpu_check.slurm'' |
+| reward_fixed_512 | 512 | 50 | ... | ... | Reward functions implemented |
-| 3. Check queue | ''squeue -u $USER'' |
+| reward_fixed_1000 | 1000 | 50 | ... | ... | More parallel environments |
-| 4. List logs | ''ls logs'' |
-| 5. Open GPU output | ''cat logs/gpu_check_123456.out'' |
-| 6. Submit training | ''sbatch lab5_train_leggedgym.slurm'' |
-| 7. Check queue again | ''squeue -u $USER'' |
-| 8. Watch training output | ''tail -f logs/pupper_train_123457.out'' |
-| 9. Open final output | ''cat logs/pupper_train_123457.out'' |
-| 10. Open error output | ''cat logs/pupper_train_123457.err'' |
-===== Named training runs =====
+===== 23. What to remember =====
-Use these commands for different experiments:
+The most important idea in this lab is that reinforcement learning does not magically learn the behavior we want.
-^ Experiment ^ Command ^
+The agent learns what the reward function encourages.
-| Short debug run | ''MAX_ITERATIONS=50 RUN_NAME="debug_${USER}" sbatch lab5_train_leggedgym.slurm'' |
-| Velocity-focused run | ''RUN_NAME="velocity_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
-| Effort-focused run | ''RUN_NAME="effort_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
-| Stability-focused run | ''RUN_NAME="stability_${USER}" MAX_ITERATIONS=300 sbatch lab5_train_leggedgym.slurm'' |
-| Final run | ''RUN_NAME="final_${USER}" MAX_ITERATIONS=500 sbatch lab5_train_leggedgym.slurm'' |
-| Shorter final run if the queue is busy | ''RUN_NAME="final_${USER}" MAX_ITERATIONS=200 sbatch lab5_train_leggedgym.slurm'' |
-===== Useful repository commands =====
+If the reward function is zero, the robot has no reason to improve.
-Run these commands inside the Legged Gym repository.
+If the reward function encourages forward movement but also penalizes unstable or inefficient behavior, the robot has a better chance of learning useful locomotion.
-^ What you want to do ^ Command ^
+===== 24. Optional final step - Upload the trained policy to the real Pupper robot =====
-| Go to the repository | ''cd ~/pupper_lab5/leggedgym'' |
-| List files | ''ls'' |
-| Find Pupper files | ''find . -iname "    *pupper    *"'' |
-| Find reward functions | ''grep -R "_reward" legged_gym/envs'' |
-| Find Python files containing torque | ''grep -R "torque" legged_gym/envs'' |
-| Find Python files containing velocity | ''grep -R "velocity" legged_gym/envs'' |
-| Find Python files containing height | ''grep -R "height" legged_gym/envs'' |
-| Find logs | ''find logs -type f | head -20'' |
-| Find saved models | ''find . -type f -name "    *.pt"'' |
-| Find videos | ''find . -type f -name "    *.mp4"'' |
-===== Important note about running training =====
+After training a policy in simulation, the next step is to test it on the real Pupper robot.
-Do not run the training command directly on the FEP.
+This step should only be done under instructor supervision.
-Do not run this directly in the terminal:
+Before uploading anything to the real robot, make sure that:
-''python legged_gym/scripts/train.py --task=pupper_flat --num_envs=2000 --max_iterations=300 --headless''
+    * the robot battery is charged;
+    * the robot is placed on the floor in a safe open area;
+    * the emergency stop is available;
+    * the policy was tested in simulation;
+    * the correct configuration file is used on the robot;
+    * the instructor or lab assistant is present.
-Instead, run:
+==== 24.1 Find the trained policy ====
-''sbatch lab5_train_leggedgym.slurm''
+After training, the policy is saved inside the Legged Gym logs directory.
-This sends the training job to a GPU compute node through SLURM.
+Use:
-===== What to submit =====
+<code bash>
+cd ~/pupper_lab8/leggedgym
+find logs -name "    *.pt" | tail -20
+</code>
-Submit one archive containing:
+Look for a file similar to:
-''team_name_lab5.zip''
+<code>
+model_300.pt
+model_1500.pt
+</code>
-Inside the archive, include:
+The exact name depends on the number of training iterations.
-    * ''report.pdf'';
+==== 24.2 Copy the policy to a deployment folder ====
-    * ''slurm_outputs/'';
-    * ''reward_code/'';
-    * ''screenshots/'';
-    * ''logs/'';
-    * ''videos_optional/''.
-Your report must include:
+Go to the deployment repository or folder provided by the instructor.
-    * your team name;
+Example:
-    * the GPU partition used;
-    * the job IDs;
-    * the baseline command;
-    * the reward functions you identified;
-    * the reward changes you made;
-    * the commands used for each run;
-    * screenshots or logs showing training;
-    * the comparison table;
-    * answers about the sim-to-real gap;
-    * final conclusions.
-===== Report questions =====
+<code bash>
+cd ~/pupper_lab8
+mkdir -p deploy_policy
+</code>
-Answer these questions in your report:
+Copy the trained model:
-    * What is the role of the reward function in Reinforcement Learning?
+<code bash>
-    * Why do we train Pupper in simulation before testing on the real robot?
+cp ~/pupper_lab8/leggedgym/logs/<experiment_folder>/model_<iteration>.pt ~/pupper_lab8/deploy_policy/
-    * What does ''--num_envs=2000'' mean?
+</code>
-    * Why is GPU acceleration useful for this experiment?
-    * What happened when you emphasized forward velocity?
+Replace `<experiment_folder>` and `<iteration>` with the real names from your training output.
-    * What happened when you penalized effort?
-    * What happened when you emphasized stability?
+==== 24.3 Convert or rebuild the neural controller ====
-    * Which reward configuration worked best?
-    * Was the best numerical reward also the best behaviour?
+Some Pupper deployment code does not use the raw `.pt` file directly. It may require rebuilding or exporting the neural controller.
+If the deployment folder contains a script such as:
+<code>
+rebuild_neural_controller.py
+</code>
+run it according to the instructor’s instructions.
+Example:
+<code bash>
+cd ~/pupper_lab8/deploy
+python rebuild_neural_controller.py
+</code>
+The exact command may differ depending on the deployment package used in the lab.
+==== 24.4 Upload the controller to Pupper ====
+Connect to the Pupper robot using SSH.
+Example:
+<code bash>
+ssh pi@pupper.local
+</code>
+or, if the robot has a fixed IP address:
+<code bash>
+ssh pi@<PUPPER_IP_ADDRESS>
+</code>
+From your local or HPC environment, copy the generated controller or policy files to the robot:
+<code bash>
+scp -r ~/pupper_lab8/deploy_policy/    * pi@<PUPPER_IP_ADDRESS>:~/pupper_deploy/policies/
+</code>
+Replace `<PUPPER_IP_ADDRESS>` with the real IP address of the robot.
+==== 24.5 Run the policy on the robot ====
+On the Pupper robot:
+<code bash>
+cd ~/pupper_deploy
+python launch.py
+</code>
+or use the command provided by the instructor for the specific robot setup.
+Observe the robot carefully.
+Stop the program immediately if:
+    * the robot moves violently;
+    * the joints oscillate strongly;
+    * the robot falls repeatedly;
+    * the motors overheat;
+    * the emergency stop is needed.
+==== 24.6 Reflection question ====
+Compare the behavior in simulation with the behavior on the real robot.
+Answer:
+    * Did the robot behave the same in simulation and reality?
+    * What differences did you observe?
+    * Why can a policy trained in simulation fail on a real robot?
     * What is the sim-to-real gap?
+    * How could domain randomization help?
-===== Common problems =====
+===== 25. Instructor notes =====
-==== The job stays in the queue ====
+This section is for the instructor or lab assistant.
-Check your jobs with:
+The tested working stack on the HPC cluster was:
-''squeue -u $USER''
+    * SLURM job on the `dgxa100` partition;
+    * NVIDIA A100-SXM4-80GB GPU;
+    * Apptainer with `--nv`;
+    * PyTorch 1.10.0 CUDA 11.3 container;
+    * Isaac Gym Preview 4;
+    * Python 3.7;
+    * Legged Gym;
+    * `pupper_flat` task.
-The cluster may be busy. Wait or reduce the requested time.
+The final import check must confirm:
-==== The CUDA module does not exist ====
+<code>
+isaacgym import: OK
+gymtorch import: OK
+rsl_rl import: OK
+legged_gym import: OK
+CUDA available: True
+</code>
-Check available CUDA modules with:
+The final training script must export the same environment variables used during the successful import check, especially:
-''module avail cuda''
+<code bash>
+export PYTHONUSERBASE=$LAB_DIR/pyuser_isaac
+export TOOL_PREFIX=$LAB_DIR/conda_tools
+export PATH=$TOOL_PREFIX/bin:$PYTHONUSERBASE/bin:$PATH
+export PYTHONPATH=$PYTHONUSERBASE/lib/python3.7/site-packages:${PYTHONPATH:-}
+export LD_LIBRARY_PATH=$TOOL_PREFIX/lib:/opt/conda/lib:${LD_LIBRARY_PATH:-}
+export LD_PRELOAD=$TOOL_PREFIX/lib/libstdc++.so.6:$TOOL_PREFIX/lib/libgcc_s.so.1
+export CPATH=$LAB_DIR/local_include:$TOOL_PREFIX/include:${CPATH:-}
+export CC=$TOOL_PREFIX/bin/x86_64-conda-linux-gnu-gcc
+export CXX=$TOOL_PREFIX/bin/x86_64-conda-linux-gnu-c++
+export TORCH_EXTENSIONS_DIR=$LAB_DIR/torch_extensions
+export MAX_JOBS=1
+</code>
-Choose an available CUDA module and modify the SLURM files.
+The final `lab8_train_isaacgym.slurm` should pass the task variables into Apptainer using:
-==== The script fails with “command not found: module” ====
+<code bash>
+export APPTAINERENV_LAB_DIR="$LAB_DIR"
+export APPTAINERENV_TASK="$TASK"
+export APPTAINERENV_NUM_ENVS="$NUM_ENVS"
+export APPTAINERENV_MAX_ITERATIONS="$MAX_ITERATIONS"
+export APPTAINERENV_RUN_NAME="$RUN_NAME"
+</code>
-You may not be on the correct cluster environment. Reconnect to the FEP and try again.
+A safe starting point for students is:
-==== The job fails immediately ====
+<code bash>
+RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
+</code>
-Check both files:
+A tested larger run is:
-    * ''cat logs/pupper_train_123457.out'';
+<code bash>
-    * ''cat logs/pupper_train_123457.err''.
+RUN_NAME="baseline_zero_reward_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-Replace ''123457'' with your real job ID.
+If `NUM_ENVS=2000` causes an OOM kill, reduce the number of environments or increase the requested memory in the SLURM script.
-The error file usually contains the useful information.
-==== Python cannot import torch ====
-The Python environment is not ready. Activate the correct environment or install the required dependencies.
+=====Manual Install=====
+https://developer.nvidia.com/isaac-gym/download
-==== CUDA is not available in Python ====
+<code>
+#!/bin/bash
+#SBATCH --job-name=lab8_setup
+#SBATCH --partition=dgxa100
+#SBATCH --gres=gpu:1
+#SBATCH --cpus-per-task=8
+#SBATCH --mem=32G
+#SBATCH --time=02:00:00
+#SBATCH --output=lab8_setup_%j.out
+#SBATCH --error=lab8_setup_%j.err
-The training script checks this automatically.
+set -e
-If the output says ''False'' for CUDA availability, check:
+LAB_DIR="$HOME/pupper_lab8"
+mkdir -p "$LAB_DIR"
+cd "$LAB_DIR"
-    * the CUDA module;
+echo "LAB_DIR=$LAB_DIR"
-    * the PyTorch version;
-    * whether the job really received a GPU;
-    * whether you used ''sbatch'', not direct execution on FEP.
-==== The reward file cannot be found ====
+echo "1. Check Apptainer image"
+if [ ! -f pytorch_isaacgym.sif ]; then
+    echo "pytorch_isaacgym.sif not found."
+    echo "Pulling base PyTorch image..."
+    apptainer pull pytorch_isaacgym.sif docker://pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
+else
+    echo "pytorch_isaacgym.sif already exists."
+fi
-Search for it with:
+echo "2. Check Isaac Gym package"
+if [ ! -f IsaacGym_Preview_4_Package.tar.gz ]; then
+    echo "ERROR: IsaacGym_Preview_4_Package.tar.gz is missing."
+    echo "The instructor must upload it manually to $LAB_DIR."
+    exit 1
+fi
-    * ''cd ~/pupper_lab5/leggedgym'';
+if [ ! -d isaacgym ]; then
-    * ''find . -iname "    *pupper    *"'';
+    echo "Extracting Isaac Gym..."
-    * ''grep -R "_reward" legged_gym/envs''.
+    tar -xzf IsaacGym_Preview_4_Package.tar.gz
+else
+    echo "isaacgym folder already exists."
+fi
-==== Training is too slow ====
+echo "3. Clone rsl_rl if missing"
+if [ ! -d rsl_rl ]; then
+    git clone https://github.com/leggedrobotics/rsl_rl.git
+else
+    echo "rsl_rl already exists."
+fi
-Use fewer iterations for testing:
+echo "4. Clone leggedgym if missing"
+if [ ! -d leggedgym ]; then
+    git clone https://github.com/cs123-stanford/leggedgym.git
+else
+    echo "leggedgym already exists."
+fi
-''MAX_ITERATIONS=50 sbatch lab5_train_leggedgym.slurm''
+echo "5. Install Python packages inside Apptainer user base"
-Do not use long runs until the short run works.
+export PYTHONUSERBASE="$LAB_DIR/pyuser_isaac"
+export PATH="$PYTHONUSERBASE/bin:$PATH"
+export PYTHONPATH="$PYTHONUSERBASE/lib/python3.7/site-packages:${PYTHONPATH:-}"
-==== The policy does not walk well ====
+apptainer exec --nv pytorch_isaacgym.sif bash -lc "
+set -e
+cd $LAB_DIR
-That is normal for short training runs. The goal is to understand reward shaping and compare experiments, not necessarily to produce a perfect robot gait in one lab session.
+export PYTHONUSERBASE=$PYTHONUSERBASE
+export PATH=$PATH
+export PYTHONPATH=$PYTHONPATH
-==== The policy works in simulation but not on the real robot ====
+python -m pip install --user --upgrade pip
-Discuss the sim-to-real gap. Do not assume the training failed only because real-world deployment is imperfect.
+python -m pip install --user -e isaacgym/python
+python -m pip install --user -e rsl_rl
+python -m pip install --user -e leggedgym
-===== Final conclusion =====
+python - <<'PY'
+import isaacgym
+import torch
+import rsl_rl
+import legged_gym
-In this lab, you used a Stanford-style Reinforcement Learning workflow to train Pupper in simulation. You submitted GPU jobs through SLURM, inspected reward functions, changed reward terms and compared policies.
+print('isaacgym OK')
+print('torch', torch.__version__)
+print('cuda available', torch.cuda.is_available())
+print('rsl_rl OK')
+print('legged_gym OK')
+PY
+"
-The key idea is that robot learning is not only about running a training script. The quality of the learned behaviour depends strongly on how the reward function is designed.
+echo "Lab 8 setup finished successfully."
-A fast robot is not always a good robot. A stable robot is not always fast. A useful policy balances speed, effort, stability and smoothness.
+</code>

Labs

Resources

rasb/lab/08.1781768155.txt.gz · Last modified: 2026/06/18 10:35 by vlad.radulescu2901

Show page Old revisions

Media Manager Back to top