Show page

Differences

This shows you the differences between two versions of the page.

--- rasb:lab:08 [2026/06/15 15:55]
vlad.radulescu2901 [Mission 5 - Smooth movements]
+++ rasb:lab:08 [2026/07/08 19:29] (current)
vlad.radulescu2901 [Manual Install]
@@ Line 1: / Line 1: @@
 ====== Lab 8: Reinforcement Learning for Robotics ======
-===== The story of the lab: from reward to walking =====
+===== 1. Lab idea =====
-In this lab, we study how a quadruped robot, Pupper, can learn to walk using Reinforcement Learning.
+In this lab, you will train a quadruped robot called Pupper in simulation using Reinforcement Learning.
-Instead of manually programming every leg movement, we define a reward function. The robot receives higher scores for desired behaviours, such as moving forward at the requested speed, keeping its body stable, and producing smooth movements. At the same time, it receives penalties for undesired behaviours, such as high effort, sudden movements, or instability.
+The robot will be trained inside NVIDIA Isaac Gym, a GPU-based physics simulator that can run many environments in parallel. Instead of training one robot at a time, we can train hundreds or thousands of simulated robots at the same time.
-In real Reinforcement Learning systems, a neural policy is trained in a simulator using an algorithm such as PPO, which stands for Proximal Policy Optimization. In this lab, students will work with a simplified reward tuning sandbox in order to understand how a reward function is calculated and how changing the coefficients affects the final score.
+The learning algorithm used in this lab is PPO - Proximal Policy Optimization.
-If the physical Pupper robot is available, the second part of the lab allows students to test an already trained policy on the real robot.
+The main goal of the lab is not to install Isaac Gym. The simulator and training pipeline are already prepared. Your task is to complete the reward functions used by the Pupper robot.
-===== Creating the reward_lab starter folder =====
+Initially, the reward functions return zero, so the robot has no useful learning signal. You will implement reward terms that encourage the robot to:
-The original repository contains the controller and deploy files for Pupper. For the reward tuning activity, each student team will create an additional folder named ''reward_lab''.
+    * move forward;
+    * keep a stable body height;
+    * avoid using unnecessarily large motor torques.
-This folder will contain a simplified reward tuning sandbox. It is not a full physics simulator and it does not train a real neural policy. Its purpose is to help students understand how a reward function is built, how different reward terms are combined, and how changing coefficients affects the final score.
+At the end of the lab, you will compare the training results before and after modifying the reward function.
-From the root of the repository, run:
+===== 2. Learning objectives =====
-<code bash>
+After this lab, you should be able to:
-cd ~/lab_5_fall_2025
-mkdir -p reward_lab/outputs
+    * explain why reinforcement learning needs a reward function;
+    * understand why many simulated environments are used in parallel;
+    * run a training job on the HPC cluster using SLURM;
+    * identify and modify reward functions in a Legged Gym environment;
+    * compare multiple training runs using logs;
+    * explain how reward shaping influences the behavior learned by a robot;
+    * understand the basic idea of transferring a trained policy from simulation to the real Pupper robot.
+===== 3. Background =====
+A reinforcement learning agent learns by interacting with an environment.
+For a robot, the environment contains:
+    * the robot body;
+    * the physics simulation;
+    * gravity;
+    * contacts with the ground;
+    * joint positions and velocities;
+    * actions applied to the motors.
+At every step, the agent receives an observation and outputs an action. The simulator applies the action and returns a reward.
+The reward tells the agent whether its behavior is good or bad.
+For example:
+    * if the robot moves forward, it should receive a positive reward;
+    * if it falls, it should receive a penalty;
+    * if it uses too much torque, it should receive a penalty;
+    * if it keeps a stable body height, it should receive a better score.
+A bad reward function can make the robot learn nothing. A good reward function can make the robot learn useful locomotion.
+===== 4. Important note about this lab =====
+The HPC environment, container, Isaac Gym, PyTorch and Legged Gym are already prepared for you.
+You should not try to reinstall Isaac Gym manually during the lab.
+The important part of this lab is inside the file:
+<code>
+~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
 </code>
-After this step, the repository should contain:
+The initial version contains TODO functions similar to this:
+<code python>
+def _reward_base_height(self):
+    return 0.0
+def _reward_forward_velocity(self):
+return 0
+def _reward_torques(self):
+return 0 </code>
+As long as these functions return zero, the robot has no meaningful learning signal.
+===== 5. Files used in this lab =====
+Download the starter archive from OCW:
+{{ :rasb:lab:lab8_hpc_pack.zip | Download Lab 8 HPC starter pack }}
+The starter pack contains the SLURM scripts needed for running the training jobs.
+Expected working directory:
 <code>
-lab_5_fall_2025/
+~/pupper_lab8
-├── config.yaml
-├── launch.py
-├── estop_controller.cpp
-├── parkour_policy.json
-├── test_policy.json
-├── rebuild_neural_controller.py
-├── deploy.py
-└── reward_lab/
-    └── outputs/
 </code>
-The folder ''reward_lab'' will be used only for the reward tuning exercise. Students must not modify ''config.yaml'' or the ''.json'' policy files when working on the reward function.
+Expected repository structure:
+<code>
+~/pupper_lab8/
+├── isaacgym/
+├── leggedgym/
+├── rsl_rl/
+├── pytorch_isaacgym.sif
+├── pyuser_isaac/
+├── conda_tools/
+├── local_include/
+├── torch_extensions/
+├── logs/
+├── lab8_gpu_check.slurm
+├── lab8_import_check.slurm
+├── lab8_train_isaacgym.slurm
+└── lab8_reward_check.sh
+</code>
-===== Downloading the reward_lab starter folder =====
+The archive contains only the small helper scripts. It does not contain the large simulator files, the container image or the full repositories.
-For the reward tuning activity, use the starter archive provided on OCW:
+===== 6. Connect to the HPC cluster =====
-  * {{:rasb:lab:reward_lab.zip|Download reward_lab.zip}}
+Connect to the frontend node:
-The original repository contains the controller and deploy files for Pupper. The archive ''reward_lab.zip'' adds a simplified reward tuning sandbox. This sandbox is not a full physics simulator and it does not train a real neural policy. Its purpose is to help you understand how a reward function is built, how different reward terms are combined, and how changing coefficients affects the final score.
+<code bash>
+ssh your_username@fep.grid.pub.ro
+</code>
-Download ''reward_lab.zip'' and extract it in the root folder of the repository:
+Go to the lab directory:
 <code bash>
-cd ~/lab_5_fall_2025
+cd ~/pupper_lab8
-unzip reward_lab.zip
-ls reward_lab
 </code>
-After extraction, the repository should contain:
+Create the logs directory if it does not already exist:
-<code>
+<code bash>
-lab_5_fall_2025/
+mkdir -p logs
-├── config.yaml
-├── launch.py
-├── estop_controller.cpp
-├── parkour_policy.json
-├── test_policy.json
-├── rebuild_neural_controller.py
-├── deploy.py
-└── reward_lab/
-    ├── reward_config.yaml
-    ├── reward_functions.py
-    ├── run_reward_experiment.py
-    ├── README.md
-    └── outputs/
 </code>
-The folder ''reward_lab'' is used only for the reward tuning exercise. Do not modify ''config.yaml'' or the ''.json'' policy files when working on the reward function.
+===== 7. Check that the GPU is available =====
+Before running Isaac Gym, check that your SLURM job can access the GPU.
+Submit the GPU check job:
-===== Running the reward experiment =====
+<code bash>
+cd ~/pupper_lab8
+sbatch lab8_gpu_check.slurm
+</code>
-Install the required Python package:
+Check the queue:
 <code bash>
-sudo apt install python3-yaml
+squeue -u $USER
 </code>
-Run the experiment:
+After the job finishes, inspect the output:
 <code bash>
-python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml
+ls logs
+cat logs/gpu_check_<JOB_ID>.out
+cat logs/gpu_check_<JOB_ID>.err
 </code>
-List the generated output files:
+Replace `<JOB_ID>` with the job id printed by `sbatch`.
+A successful run should show an NVIDIA GPU through `nvidia-smi`.
+===== 8. Check that Isaac Gym and Legged Gym work =====
+Before changing the reward function, check that the simulator imports correctly.
+Submit the import check job:
 <code bash>
-ls reward_lab/outputs
+cd ~/pupper_lab8
+sbatch lab8_import_check.slurm
 </code>
-The script prints:
+After the job finishes:
-    * the average total reward;
+<code bash>
-    * the average raw value of each reward component;
+cat logs/import_check_<JOB_ID>.out
-    * the average weighted contribution of each component;
+cat logs/import_check_<JOB_ID>.err
-    * the CSV and JSON files generated in ''reward_lab/outputs''.
+</code>
-Important: ''effort'' and ''smoothness'' are calculated as positive values, but they usually have negative coefficients in the configuration file. This means they reduce the total reward.
+A successful import check should contain:
-===== Student checkpoint =====
+<code>
+isaacgym import: OK
+gymtorch import: OK
+rsl_rl import: OK
+legged_gym import: OK
+CUDA available: True
+</code>
-Before continuing, each team must confirm that:
+If this step fails, do not continue to the reward task. Ask the instructor or lab assistant for help.
-    * the folder ''reward_lab'' exists;
+===== 9. Run a small debug training job =====
-    * the file ''reward_config.yaml'' exists;
-    * the file ''reward_functions.py'' exists;
+Now check that the Pupper training task can start.
-    * the file ''run_reward_experiment.py'' exists;
-    * the command below runs without errors:
+Submit a small debug job:
 <code bash>
-python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml
+cd ~/pupper_lab8
+RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
 </code>
+Check the queue:
-===== How to read the result =====
+<code bash>
+squeue -u $USER
+</code>
-The script displays:
+After the job finishes, check its output:
-    * the average total reward;
+<code bash>
-    * the average contribution of each reward component;
+cat logs/pupper_train_<JOB_ID>.out
-    * the files generated in ''reward_lab/outputs''.
+cat logs/pupper_train_<JOB_ID>.err
+</code>
-Example output:
+A successful run should contain messages similar to:
 <code>
-Reward experiment completed
+Physics Engine: PhysX
-Average total reward: 2.1453
+Physics Device: cuda:0
+GPU Pipeline: enabled
+Learning iteration 0/10
+Learning iteration 9/10
+Training finished.
+</code>
-Average components:
+This means that Isaac Gym, PyTorch, CUDA and the Pupper environment are working.
-tracking_lin_vel: 0.8741
-tracking_ang_vel: 0.9632
-effort: 0.5120
-stability: 0.8018
-smoothness: 0.0921 </code>
-Important: the ''effort'' and ''smoothness'' functions return positive values, but in the configuration file they usually have negative coefficients. This means they decrease the total reward.
+===== 10. Run the initial baseline =====
-===== Mission 1 - Understanding already trained policies =====
+Now run a slightly larger baseline:
-List the main files in the repository:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="baseline_zero_reward_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
+After the job finishes, inspect the reward values:
 <code bash>
-ls
+cat logs/pupper_train_<JOB_ID>.out | grep -E "Learning iteration|Mean reward|rew_forward_velocity|episode length" | tail -80
 </code>
-List the available policies:
+You should observe that the reward is not useful. In the initial version, the relevant reward functions return zero.
+Example:
+<code>
+Mean reward: 0.00
+Mean episode rew_forward_velocity: 0.0000
+</code>
+This is expected before solving the lab.
+===== 11. Inspect the reward functions =====
+Open the Pupper environment file:
 <code bash>
-ls     *.json
+cd ~/pupper_lab8/leggedgym
+nano legged_gym/envs/pupper/pupper.py
 </code>
+Find the following functions:
+<code python>
+def _reward_base_height(self):
+    return 0.0
+def _reward_forward_velocity(self):
+return 0
+def _reward_torques(self):
+return 0 </code>
+These functions are the main TODOs of this lab.
+You can also use the helper script:
+<code bash>
+cd ~/pupper_lab8
+./lab8_reward_check.sh
+</code>
+This script prints the reward functions and checks whether there are still `return 0` statements inside `pupper.py`.
+===== 12. Inspect the reward configuration =====
 Open the configuration file:
 <code bash>
-cat config.yaml
+cd ~/pupper_lab8/leggedgym
+nano legged_gym/envs/pupper/pupper_config.py
 </code>
-Answer in the report:
+Look for the reward section. It should contain values similar to:
-    * What ''.json'' files are available?
+<code python>
-    * Which policy is referenced in ''config.yaml''?
+class rewards:
-    * Why are the ''.json'' files not the place where we modify the reward?
+    forward_velocity_clip = 1.0
-    * What is the difference between an already trained policy and the reward function used during training?
-===== Mission 2 - Forward velocity =====
+```
+class scales:
+    forward_velocity = 3.0
+```
-In this mission, the robot should receive a high reward for tracking the desired linear velocity.
+</code>
-Edit the file:
+The exact values may differ depending on the starter code version.
-<code bash>
+The important idea is that the config file contains the coefficients used to scale the reward terms.
-nano reward_lab/reward_config.yaml
+For example:
+    * a positive scale means the term increases the reward;
+    * a negative scale means the term becomes a penalty;
+    * a zero scale disables the term.
+===== 13. Task 1 - Implement forward velocity reward =====
+The robot should receive a positive reward when it moves forward.
+In Legged Gym, the forward velocity of the robot base is usually stored in:
+<code python>
+self.base_lin_vel[:, 0]
 </code>
-Set:
+This is the x-axis linear velocity of the robot base.
-<code yaml>
+Replace the initial function:
-target_linear_velocity: 1.0
-target_angular_velocity: 0.0
-experiment:
+<code python>
-    steps: 200
+def _reward_forward_velocity(self):
-    seed: 42
+    return 0
-    profile: "baseline"
+</code>
-reward_weights:
+with:
-    tracking_lin_vel: 3.0
-    tracking_ang_vel: 0.5
-    effort: 0.0
-    stability: 0.5
-    smoothness: 0.0 </code>
-Run:
+<code python>
+def _reward_forward_velocity(self):
+    return torch.clip(
+        self.base_lin_vel[:, 0],
+        min=0.0,
+        max=self.cfg.rewards.forward_velocity_clip
+    )
+</code>
+This reward gives positive values only when the robot moves forward. Negative velocity is clipped to zero, so moving backwards is not rewarded.
+If `torch` is not imported at the top of the file, add:
+<code python>
+import torch
+</code>
+===== 14. Task 2 - Implement torque penalty =====
+A robot should not learn to move by using extremely large motor torques. Large torques are inefficient and can produce unstable behavior.
+The torque values are stored in:
+<code python>
+self.torques
+</code>
+Replace:
+<code python>
+def _reward_torques(self):
+    return 0
+</code>
+with:
+<code python>
+def _reward_torques(self):
+    return torch.sum(torch.square(self.torques), dim=1)
+</code>
+This function returns a positive value representing how much torque the robot uses.
+Important: this function returns a positive value, but it becomes a penalty if the scale in the config file is negative.
+For example:
+<code python>
+torques = -0.0002
+</code>
+means that large torques reduce the final reward.
+===== 15. Task 3 - Implement base height penalty =====
+The robot should keep its body at a reasonable height. If the body is too low or too high, the behavior is probably unstable.
+The base height is stored in:
+<code python>
+self.root_states[:, 2]
+</code>
+Replace:
+<code python>
+def _reward_base_height(self):
+    return 0.0
+</code>
+with:
+<code python>
+def _reward_base_height(self):
+    base_height = self.root_states[:, 2]
+    return torch.square(base_height - self.cfg.rewards.base_height_target)
+</code>
+This function returns the squared error between the current base height and the target base height.
+Again, this becomes a penalty if its scale in the config file is negative.
+===== 16. Check your code =====
+After editing `pupper.py`, check the relevant lines:
 <code bash>
-python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml
+cd ~/pupper_lab8/leggedgym
+nl -ba legged_gym/envs/pupper/pupper.py | sed -n '80,120p'
 </code>
-Write down:
+You should see the implemented reward functions, not `return 0`.
-    * the average total reward;
+You can also run:
-    * the contribution of ''tracking_lin_vel'';
-    * the risk of rewarding the robot almost only for speed.
-===== Mission 3 - Penalising effort =====
+<code bash>
+cd ~/pupper_lab8
+./lab8_reward_check.sh
+</code>
-Now we keep the target velocity, but we penalise effort.
+If the script still shows `return 0` inside the reward functions, your implementation is not complete.
-Modify the configuration:
+===== 17. Train again after implementing the rewards =====
-<code yaml>
+Run the training job again:
-target_linear_velocity: 1.0
-target_angular_velocity: 0.0
-experiment:
+<code bash>
-    steps: 200
+cd ~/pupper_lab8
-    seed: 42
+RUN_NAME="reward_fixed_512_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
-    profile: "baseline"
+</code>
-reward_weights:
+Check the job:
-    tracking_lin_vel: 3.0
-    tracking_ang_vel: 0.5
-    effort: -0.5
-    stability: 0.5
-    smoothness: 0.0 </code>
-Run:
+<code bash>
+squeue -u $USER
+</code>
+After it finishes:
 <code bash>
-python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml
+cat logs/pupper_train_<JOB_ID>.out | grep -E "Learning iteration|Mean reward|rew_forward_velocity|episode length" | tail -100
+cat logs/pupper_train_<JOB_ID>.err
 </code>
-Compare the result with the previous mission.
+Compare the new output with the initial baseline.
-Answer:
+You should focus on:
-    * Did the total score increase or decrease?
+    * `Mean reward`;
-    * Why?
+    * `Mean episode rew_forward_velocity`;
-    * Why is it useful to penalise effort?
+    * `Mean episode length`;
-    * What trade-off appears between speed and energy consumption?
+    * total timesteps;
+    * whether the job completed successfully.
-===== Mission 4 - Stability =====
+===== 18. Larger training runs =====
-Now we want a more stable behaviour, even if the robot becomes slightly slower.
+After the small run works, you can try larger experiments.
-Modify the configuration:
+Run with 1000 environments:
-<code yaml>
+<code bash>
-target_linear_velocity: 0.8
+cd ~/pupper_lab8
-target_angular_velocity: 0.0
+RUN_NAME="reward_fixed_1000_${USER}" NUM_ENVS=1000 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-experiment:
+If that works, try 2000 environments:
-    steps: 200
-    seed: 42
-    profile: "stable"
-reward_weights:
+<code bash>
-    tracking_lin_vel: 1.5
+cd ~/pupper_lab8
-    tracking_ang_vel: 0.5
+RUN_NAME="reward_fixed_2000_${USER}" NUM_ENVS=2000 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
-    effort: -0.2
+</code>
-    stability: 3.0
-    smoothness: -0.2 </code>
-Run:
+For a longer training run:
 <code bash>
-python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_long_${USER}" NUM_ENVS=2000 MAX_ITERATIONS=300 sbatch lab8_train_isaacgym.slurm
 </code>
-Answer:
+Do not start with the largest experiment. First check that the short run works.
-    * Which term is the most important in this configuration?
+===== 19. Useful SLURM commands =====
-    * Why was the target velocity reduced?
-    * What is the effect of increasing the coefficient for ''stability''?
-===== Mission 5 - Smooth movements =====
+^ What you want to do ^ Command ^
+| Submit GPU check | ''sbatch lab8_gpu_check.slurm'' |
+| Submit import check | ''sbatch lab8_import_check.slurm'' |
+| Submit training | ''sbatch lab8_train_isaacgym.slurm'' |
+| Submit debug training | ''RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm'' |
+| Check your jobs | ''squeue -u $USER'' |
+| Check finished job status | ''sacct -j <JOB_ID> --format=JobID,JobName,Partition,State,Elapsed,ExitCode,MaxRSS,ReqMem'' |
+| Show output log | ''cat logs/pupper_train_<JOB_ID>.out'' |
+| Show error log | ''cat logs/pupper_train_<JOB_ID>.err'' |
+| Show only reward lines | ''cat logs/pupper_train_<JOB_ID>.out | grep -E "Mean reward|rew_forward_velocity|episode length"'' |
+| Inspect reward functions | ''./lab8_reward_check.sh'' |
-In this mission, we penalise sudden movements.
+===== 20. Common problems =====
-Modify the configuration:
+==== Problem 1 - The reward stays zero ====
-<code yaml>
+If the output still shows:
-target_linear_velocity: 0.8
-target_angular_velocity: 0.0
-experiment:
+<code>
-    steps: 200
+Mean reward: 0.00
-    seed: 42
+Mean episode rew_forward_velocity: 0.0000
-    profile: "stable"
+</code>
-reward_weights:
+check that you actually modified:
-    tracking_lin_vel: 1.5
-    tracking_ang_vel: 0.5
-    effort: -0.2
-    stability: 3.0
-    smoothness: -0.2  </code>
-Run:
+<code>
+~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
+</code>
+and that the reward functions no longer return zero.
+Use:
 <code bash>
-python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config.yaml
+cd ~/pupper_lab8
+./lab8_reward_check.sh
 </code>
-Answer:
+or:
-    * How does the ''smoothness'' penalty influence the total score?
+<code bash>
-    * Why are smooth movements important for a real robot?
+grep -R "return 0" ~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
-    * What could happen if this penalty is too large?
+</code>
-===== Mission 6 - Team configuration =====
+==== Problem 2 - The job is killed with OOM ====
-Each team must propose its own configuration.
+If the job fails with:
-Create a new file:
+<code>
+Detected 1 oom_kill event
+Some of the step tasks have been OOM Killed
+</code>
+then the job used too much RAM.
+Use fewer environments:
 <code bash>
-cp reward_lab/reward_config.yaml reward_lab/reward_config_team.yaml
+NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
-nano reward_lab/reward_config_team.yaml
 </code>
-You may modify:
+or:
-    * the target linear velocity;
+<code bash>
-    * the target angular velocity;
+NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
-    * the profile: ''baseline'', ''fast'', ''stable'', or ''unstable'';
+</code>
-    * the reward coefficients.
-Run:
+If needed, the SLURM memory limit can be increased by the instructor in the training script:
 <code bash>
-python3 reward_lab/run_reward_experiment.py --config reward_lab/reward_config_team.yaml
+#SBATCH --mem=128G
 </code>
-The team must explain:
+==== Problem 3 - Task not registered ====
-    * what values were chosen;
+If you see:
-    * why those values were chosen;
-    * which term is considered the most important;
-    * what trade-off was accepted: speed, effort, stability, or smoothness;
-    * whether the result is better than the previous missions.
-===== Mission 7 - Comparing configurations =====
+<code>
+ValueError: Task with name: ... was not registered
+</code>
-Choose the 3 most important configurations you tested and complete the table:
+check that the task name is:
-^ Configuration ^ tracking_lin_vel ^ effort ^ stability ^ smoothness ^ Average total reward ^ Observation ^
+<code>
-| Speed-focused | | | | | | |
+pupper_flat
-| Effort penalty | | | | | | |
+</code>
-| Team configuration | | | | | | |
-Answer:
+You can inspect registered tasks with:
-    * Which configuration achieved the highest score?
+<code bash>
-    * Which configuration seems more suitable for a real robot?
+cd ~/pupper_lab8/leggedgym
-    * Is the configuration with the highest score always the best one? Why?
+grep -R "task_registry.register" -n legged_gym/envs
+</code>
-===== Preparing the neural controller on the real robot =====
+==== Problem 4 - You edited the wrong task ====
-This part is done only if you have access to the physical Pupper robot.
+This lab uses:
-The script ''rebuild_neural_controller.py'' must be executed on the robot's Raspberry Pi, not on Windows and not in the local folder used for reward tuning.
+<code>
+pupper_flat
+</code>
+Do not confuse it with:
+<code>
+pupper_standup
+</code>
+The standup task may have different reward scales, including zero forward velocity reward.
+==== Problem 5 - The output stops after gymtorch ====
+If the output stops around:
+<code>
+Building extension module gymtorch...
+ninja: no work to do.
+</code>
-Connect to the Raspberry Pi:
+check the error file:
 <code bash>
-ssh pi@IP_ROBOT
+cat logs/pupper_train_<JOB_ID>.err
 </code>
-On the Raspberry Pi:
+If the job completed successfully, use:
 <code bash>
-cd ~
+sacct -j <JOB_ID> --format=JobID,JobName,Partition,State,Elapsed,ExitCode,MaxRSS,ReqMem
-git clone https://github.com/cs123-stanford/lab_5_fall_2025.git
-cd lab_5_fall_2025
 </code>
-Check the required files:
+===== 21. Questions =====
+Answer the following questions in your report:
+    * Why did the initial training run produce zero reward?
+    * Why is forward velocity a useful reward term for locomotion?
+    * Why should large torques be penalized?
+    * Why can body height be used as a stability-related reward term?
+    * What is the purpose of training many environments in parallel?
+    * What changed after you implemented the reward functions?
+    * Did the robot learn better behavior after more iterations? Explain using the log values.
+    * Why might a policy that works in simulation behave differently on the real robot?
+===== 22. Deliverables =====
+Submit a short report containing:
+    * your implemented reward functions;
+    * a screenshot or copied log section from the baseline run;
+    * a screenshot or copied log section from the improved reward run;
+    * a small comparison table;
+    * short answers to the lab questions.
+Example comparison table:
+^ Run name ^ NUM_ENVS ^ MAX_ITERATIONS ^ Mean reward ^ rew_forward_velocity ^ Observation ^
+| baseline_zero_reward | 512 | 50 | 0.00 | 0.0000 | No useful learning signal |
+| reward_fixed_512 | 512 | 50 | ... | ... | Reward functions implemented |
+| reward_fixed_1000 | 1000 | 50 | ... | ... | More parallel environments |
+===== 23. What to remember =====
+The most important idea in this lab is that reinforcement learning does not magically learn the behavior we want.
+The agent learns what the reward function encourages.
+If the reward function is zero, the robot has no reason to improve.
+If the reward function encourages forward movement but also penalizes unstable or inefficient behavior, the robot has a better chance of learning useful locomotion.
+===== 24. Optional final step - Upload the trained policy to the real Pupper robot =====
+After training a policy in simulation, the next step is to test it on the real Pupper robot.
+This step should only be done under instructor supervision.
+Before uploading anything to the real robot, make sure that:
+    * the robot battery is charged;
+    * the robot is placed on the floor in a safe open area;
+    * the emergency stop is available;
+    * the policy was tested in simulation;
+    * the correct configuration file is used on the robot;
+    * the instructor or lab assistant is present.
+==== 24.1 Find the trained policy ====
+After training, the policy is saved inside the Legged Gym logs directory.
+Use:
 <code bash>
-ls config.yaml launch.py estop_controller.cpp parkour_policy.json test_policy.json
+cd ~/pupper_lab8/leggedgym
+find logs -name "    *.pt" | tail -20
 </code>
-Run:
+Look for a file similar to:
+<code>
+model_300.pt
+model_1500.pt
+</code>
+The exact name depends on the number of training iterations.
+==== 24.2 Copy the policy to a deployment folder ====
+Go to the deployment repository or folder provided by the instructor.
+Example:
 <code bash>
-python3 rebuild_neural_controller.py
+cd ~/pupper_lab8
+mkdir -p deploy_policy
 </code>
-If you get the error ''Source file does not exist'', check that the repository is located in:
+Copy the trained model:
+<code bash>
+cp ~/pupper_lab8/leggedgym/logs/<experiment_folder>/model_<iteration>.pt ~/pupper_lab8/deploy_policy/
+</code>
+Replace `<experiment_folder>` and `<iteration>` with the real names from your training output.
+==== 24.3 Convert or rebuild the neural controller ====
+Some Pupper deployment code does not use the raw `.pt` file directly. It may require rebuilding or exporting the neural controller.
+If the deployment folder contains a script such as:
 <code>
-/home/pi/lab_5_fall_2025
+rebuild_neural_controller.py
 </code>
-and that the files listed above exist.
+run it according to the instructor’s instructions.
-===== Deploying a policy on the robot =====
+Example:
-This part is done only on the real robot.
+<code bash>
+cd ~/pupper_lab8/deploy
+python rebuild_neural_controller.py
+</code>
-On the Raspberry Pi:
+The exact command may differ depending on the deployment package used in the lab.
+==== 24.4 Upload the controller to Pupper ====
+Connect to the Pupper robot using SSH.
+Example:
 <code bash>
-cd ~/lab_5_fall_2025
+ssh pi@pupper.local
-python3 deploy.py
 </code>
-After loading the policy, test the robot in a safe space.
+or, if the robot has a fixed IP address:
-Observe:
+<code bash>
+ssh pi@<PUPPER_IP_ADDRESS>
+</code>
-    * Does the robot start correctly?
+From your local or HPC environment, copy the generated controller or policy files to the robot:
-    * Is the policy activated?
-    * Is the walking behaviour stable?
-    * Are there sudden movements?
-    * Does the robot fall?
-    * Does the floor surface influence the behaviour?
-===== Difference between reward tuning and deploy =====
+<code bash>
+scp -r ~/pupper_lab8/deploy_policy/    * pi@<PUPPER_IP_ADDRESS>:~/pupper_deploy/policies/
+</code>
-Reward tuning:
+Replace `<PUPPER_IP_ADDRESS>` with the real IP address of the robot.
-    * is done in ''reward_lab/'';
+==== 24.5 Run the policy on the robot ====
-    * modifies ''reward_config.yaml'';
-    * runs ''run_reward_experiment.py'';
-    * helps us understand the reward function.
-Deploy:
+On the Pupper robot:
-    * is done on the Raspberry Pi;
+<code bash>
-    * uses ''rebuild_neural_controller.py'';
+cd ~/pupper_deploy
-    * uses already trained policies saved as ''.json'' files;
+python launch.py
-    * does not modify the reward function.
+</code>
-===== What to submit =====
+or use the command provided by the instructor for the specific robot setup.
-Each team must submit:
+Observe the robot carefully.
-    * the file ''reward_config_team.yaml'';
+Stop the program immediately if:
-    * at least 3 output files generated in ''reward_lab/outputs'';
-    * the comparison table between configurations;
-    * the answers to the mission questions;
-    * if the real robot is available: a video of the policy running on Pupper;
-    * if the real robot is not available: conclusions about the reward configuration.
-===== Report structure =====
+    * the robot moves violently;
+    * the joints oscillate strongly;
+    * the robot falls repeatedly;
+    * the motors overheat;
+    * the emergency stop is needed.
-The report must have a maximum of 2-3 pages and include:
+==== 24.6 Reflection question ====
-    * the names of the team members;
+Compare the behavior in simulation with the behavior on the real robot.
-    * a description of the reward function;
-    * the tested configurations;
-    * the obtained scores;
-    * the comparison between configurations;
-    * the explanation of the team's own configuration;
-    * the difference between reward tuning and deploy;
-    * conclusions.
-===== Questions for the report =====
+Answer:
-Answer briefly:
+    * Did the robot behave the same in simulation and reality?
+    * What differences did you observe?
+    * Why can a policy trained in simulation fail on a real robot?
+    * What is the sim-to-real gap?
+    * How could domain randomization help?
-    * What is the role of the reward function in Reinforcement Learning?
+===== 25. Instructor notes =====
-    * Why can a large coefficient for speed produce instability?
-    * What is the effect of penalising effort?
-    * Why is a stability term useful?
-    * What does smoother movement mean?
-    * Why are ''.json'' files not the place where we modify the reward?
-    * Why must ''rebuild_neural_controller.py'' be executed on the Raspberry Pi?
-    * What was the best configuration tested by your team?
-===== Common problems =====
+This section is for the instructor or lab assistant.
-If the ''reward_lab'' folder is missing, create it using the instructions at the beginning of the lab.
+The tested working stack on the HPC cluster was:
-If ''run_reward_experiment.py'' does not start, check that you are in the main repository folder:
+    * SLURM job on the `dgxa100` partition;
+    * NVIDIA A100-SXM4-80GB GPU;
+    * Apptainer with `--nv`;
+    * PyTorch 1.10.0 CUDA 11.3 container;
+    * Isaac Gym Preview 4;
+    * Python 3.7;
+    * Legged Gym;
+    * `pupper_flat` task.
+The final import check must confirm:
+<code>
+isaacgym import: OK
+gymtorch import: OK
+rsl_rl import: OK
+legged_gym import: OK
+CUDA available: True
+</code>
+The final training script must export the same environment variables used during the successful import check, especially:
 <code bash>
-pwd
+export PYTHONUSERBASE=$LAB_DIR/pyuser_isaac
-ls
+export TOOL_PREFIX=$LAB_DIR/conda_tools
-ls reward_lab
+export PATH=$TOOL_PREFIX/bin:$PYTHONUSERBASE/bin:$PATH
+export PYTHONPATH=$PYTHONUSERBASE/lib/python3.7/site-packages:${PYTHONPATH:-}
+export LD_LIBRARY_PATH=$TOOL_PREFIX/lib:/opt/conda/lib:${LD_LIBRARY_PATH:-}
+export LD_PRELOAD=$TOOL_PREFIX/lib/libstdc++.so.6:$TOOL_PREFIX/lib/libgcc_s.so.1
+export CPATH=$LAB_DIR/local_include:$TOOL_PREFIX/include:${CPATH:-}
+export CC=$TOOL_PREFIX/bin/x86_64-conda-linux-gnu-gcc
+export CXX=$TOOL_PREFIX/bin/x86_64-conda-linux-gnu-c++
+export TORCH_EXTENSIONS_DIR=$LAB_DIR/torch_extensions
+export MAX_JOBS=1
 </code>
-If you get a ''No such file or directory'' error, check the path to the configuration file.
+The final `lab8_train_isaacgym.slurm` should pass the task variables into Apptainer using:
-If you get an error related to ''yaml'', run:
+<code bash>
+export APPTAINERENV_LAB_DIR="$LAB_DIR"
+export APPTAINERENV_TASK="$TASK"
+export APPTAINERENV_NUM_ENVS="$NUM_ENVS"
+export APPTAINERENV_MAX_ITERATIONS="$MAX_ITERATIONS"
+export APPTAINERENV_RUN_NAME="$RUN_NAME"
+</code>
+A safe starting point for students is:
 <code bash>
-pip install pyyaml
+RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
 </code>
-If you run ''rebuild_neural_controller.py'' on your laptop and get errors related to ''/home/pi'', the script was executed in the wrong environment. This script is intended for the robot's Raspberry Pi.
+A tested larger run is:
-If the physical robot is not available, the deploy part is optional.
+<code bash>
+RUN_NAME="baseline_zero_reward_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
+If `NUM_ENVS=2000` causes an OOM kill, reduce the number of environments or increase the requested memory in the SLURM script.
+=====Manual Install=====
+https://developer.nvidia.com/isaac-gym/download
+<code>
+#!/bin/bash
+#SBATCH --job-name=lab8_setup
+#SBATCH --partition=dgxa100
+#SBATCH --gres=gpu:1
+#SBATCH --cpus-per-task=8
+#SBATCH --mem=32G
+#SBATCH --time=02:00:00
+#SBATCH --output=lab8_setup_%j.out
+#SBATCH --error=lab8_setup_%j.err
+set -e
+LAB_DIR="$HOME/pupper_lab8"
+mkdir -p "$LAB_DIR"
+cd "$LAB_DIR"
+echo "LAB_DIR=$LAB_DIR"
+echo "1. Check Apptainer image"
+if [ ! -f pytorch_isaacgym.sif ]; then
+    echo "pytorch_isaacgym.sif not found."
+    echo "Pulling base PyTorch image..."
+    apptainer pull pytorch_isaacgym.sif docker://pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
+else
+    echo "pytorch_isaacgym.sif already exists."
+fi
+echo "2. Check Isaac Gym package"
+if [ ! -f IsaacGym_Preview_4_Package.tar.gz ]; then
+    echo "ERROR: IsaacGym_Preview_4_Package.tar.gz is missing."
+    echo "The instructor must upload it manually to $LAB_DIR."
+    exit 1
+fi
+if [ ! -d isaacgym ]; then
+    echo "Extracting Isaac Gym..."
+    tar -xzf IsaacGym_Preview_4_Package.tar.gz
+else
+    echo "isaacgym folder already exists."
+fi
+echo "3. Clone rsl_rl if missing"
+if [ ! -d rsl_rl ]; then
+    git clone https://github.com/leggedrobotics/rsl_rl.git
+else
+    echo "rsl_rl already exists."
+fi
+echo "4. Clone leggedgym if missing"
+if [ ! -d leggedgym ]; then
+    git clone https://github.com/cs123-stanford/leggedgym.git
+else
+    echo "leggedgym already exists."
+fi
+echo "5. Install Python packages inside Apptainer user base"
+export PYTHONUSERBASE="$LAB_DIR/pyuser_isaac"
+export PATH="$PYTHONUSERBASE/bin:$PATH"
+export PYTHONPATH="$PYTHONUSERBASE/lib/python3.7/site-packages:${PYTHONPATH:-}"
+apptainer exec --nv pytorch_isaacgym.sif bash -lc "
+set -e
+cd $LAB_DIR
+export PYTHONUSERBASE=$PYTHONUSERBASE
+export PATH=$PATH
+export PYTHONPATH=$PYTHONPATH
+python -m pip install --user --upgrade pip
+python -m pip install --user -e isaacgym/python
+python -m pip install --user -e rsl_rl
+python -m pip install --user -e leggedgym
+python - <<'PY'
+import isaacgym
+import torch
+import rsl_rl
+import legged_gym
+print('isaacgym OK')
+print('torch', torch.__version__)
+print('cuda available', torch.cuda.is_available())
+print('rsl_rl OK')
+print('legged_gym OK')
+PY
+"
+echo "Lab 8 setup finished successfully."
+</code>

Labs

Resources

rasb/lab/08.1781528104.txt.gz · Last modified: 2026/06/15 15:55 by vlad.radulescu2901

Show page Old revisions

Media Manager Back to top