Show page

Differences

This shows you the differences between two versions of the page.

--- rasb:lab:08 [2026/06/16 09:25]
vlad.radulescu2901 [The story of the lab: teaching Pupper to walk]
+++ rasb:lab:08 [2026/07/08 19:29] (current)
vlad.radulescu2901 [Manual Install]
@@ Line 1: / Line 1: @@
 ====== Lab 8: Reinforcement Learning for Robotics ======
+===== 1. Lab idea =====
-===== The story of the lab: teaching Pupper to walk =====
+In this lab, you will train a quadruped robot called Pupper in simulation using Reinforcement Learning.
-In this lab, we study how a quadruped robot, Pupper, can learn to walk using Reinforcement Learning.
+The robot will be trained inside NVIDIA Isaac Gym, a GPU-based physics simulator that can run many environments in parallel. Instead of training one robot at a time, we can train hundreds or thousands of simulated robots at the same time.
-Instead of manually programming each leg movement, we define a reward function and train a neural policy in simulation. The robot receives higher rewards for behaviours we want, such as tracking a desired velocity, staying stable and moving efficiently. It receives penalties for behaviours we do not want, such as using too much effort, falling, tilting too much or producing jerky movements.
+The learning algorithm used in this lab is PPO - Proximal Policy Optimization.
-The original Stanford CS123 workflow uses large-scale parallel simulation for training. This means that many virtual Pupper robots are simulated at the same time. Each simulated robot collects experience, and the Reinforcement Learning algorithm uses that experience to improve the policy.
+The main goal of the lab is not to install Isaac Gym. The simulator and training pipeline are already prepared. Your task is to complete the reward functions used by the Pupper robot.
-In this lab, you will follow this workflow:
+Initially, the reward functions return zero, so the robot has no useful learning signal. You will implement reward terms that encourage the robot to:
-    * run or inspect the default policy;
+    * move forward;
-    * open the official training notebook or training environment;
+    * keep a stable body height;
-    * connect the experiment to Weights & Biases;
+    * avoid using unnecessarily large motor torques.
-    * modify reward terms;
-    * train a policy using parallel simulation;
-    * compare reward curves and generated videos;
-    * optionally deploy a trained policy on the physical Pupper robot.
+At the end of the lab, you will compare the training results before and after modifying the reward function.
-===== What students will learn =====
+===== 2. Learning objectives =====
-By the end of the lab, students should be able to:
+After this lab, you should be able to:
-    * explain the role of a reward function in Reinforcement Learning;
+    * explain why reinforcement learning needs a reward function;
-    * describe why simulation is used before deploying on a real robot;
+    * understand why many simulated environments are used in parallel;
-    * identify the main reward terms used for locomotion;
+    * run a training job on the HPC cluster using SLURM;
-    * modify reward coefficients and explain their effect;
+    * identify and modify reward functions in a Legged Gym environment;
-    * run a training experiment using the official Stanford/Pupper workflow;
+    * compare multiple training runs using logs;
-    * read WandB curves and compare different policies;
+    * explain how reward shaping influences the behavior learned by a robot;
-    * explain the sim-to-real gap;
+    * understand the basic idea of transferring a trained policy from simulation to the real Pupper robot.
-    * optionally deploy a trained policy on Pupper.
-===== Reinforcement Learning in this lab =====
+===== 3. Background =====
-In Reinforcement Learning, an agent interacts with an environment. At each step, the agent observes the state of the environment, chooses an action and receives a reward.
+A reinforcement learning agent learns by interacting with an environment.
-In this lab:
+For a robot, the environment contains:
-    * the agent is the neural policy controlling Pupper;
+    * the robot body;
-    * the environment is the Pupper simulation;
+    * the physics simulation;
-    * the actions are motor/controller commands;
+    * gravity;
-    * the reward measures how good the walking behaviour is;
+    * contacts with the ground;
-    * the policy is improved through training.
+    * joint positions and velocities;
+    * actions applied to the motors.
-The algorithm used in the original workflow is PPO, which stands for Proximal Policy Optimization. We do not focus on the full mathematical derivation of PPO in this lab. The important idea is that PPO updates the neural policy so that actions leading to better rewards become more likely.
+At every step, the agent receives an observation and outputs an action. The simulator applies the action and returns a reward.
-===== Why Pupper is trained in simulation =====
+The reward tells the agent whether its behavior is good or bad.
-Training directly on the real robot would be slow, expensive and risky. During early training, the robot may fall, move aggressively, damage its servos or behave unpredictably.
+For example:
-Simulation solves this problem. In the simulator, thousands of trials can be run safely. If a virtual robot falls, the simulation simply resets. No hardware is damaged.
+    * if the robot moves forward, it should receive a positive reward;
+    * if it falls, it should receive a penalty;
+    * if it uses too much torque, it should receive a penalty;
+    * if it keeps a stable body height, it should receive a better score.
-The Stanford workflow uses parallel simulation with MuJoCo/MJX and JAX. Instead of simulating one Pupper at a time, the system simulates many Pupper robots in parallel. This allows the policy to collect experience much faster.
+A bad reward function can make the robot learn nothing. A good reward function can make the robot learn useful locomotion.
-The general idea is:
+===== 4. Important note about this lab =====
+The HPC environment, container, Isaac Gym, PyTorch and Legged Gym are already prepared for you.
+You should not try to reinstall Isaac Gym manually during the lab.
+The important part of this lab is inside the file:
 <code>
-many simulated Pupper environments
+~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
-        ↓
-parallel experience collection
-        ↓
-PPO policy update
-        ↓
-improved walking behaviour
 </code>
-===== Reward functions: how we define good walking =====
+The initial version contains TODO functions similar to this:
-Pupper does not know what “good walking” means. We must define it through a reward function.
+<code python>
+def _reward_base_height(self):
+    return 0.0
-A reward function is usually built from several components. Each component encourages or discourages a specific behaviour.
+def _reward_forward_velocity(self):
+return 0
-Common reward terms for locomotion include:
+def _reward_torques(self):
+return 0 </code>
-    * velocity tracking - reward for following the desired linear velocity;
+As long as these functions return zero, the robot has no meaningful learning signal.
-    * angular velocity tracking - reward for following the desired turning velocity;
-    * effort penalty - penalty for using too much torque or energy;
-    * stability - reward for keeping the body balanced;
-    * orientation penalty - penalty for excessive body tilt;
-    * smoothness - penalty for sudden changes in movement;
-    * height tracking - reward for maintaining an appropriate body height;
-    * fall penalty - penalty if the robot falls or becomes unstable.
-A policy trained with a reward focused only on speed may become fast but unstable. A policy trained with too much effort penalty may become energy-efficient but too slow. A good reward function balances speed, stability, energy efficiency and smoothness.
+===== 5. Files used in this lab =====
-===== Important distinction: training code vs deploy code =====
+Download the starter archive from OCW:
-The repository may contain files used for deploying a trained policy on the real robot, for example:
+{{ :rasb:lab:lab8_hpc_pack.zip | Download Lab 8 HPC starter pack }}
+The starter pack contains the SLURM scripts needed for running the training jobs.
+Expected working directory:
 <code>
-config.yaml
+~/pupper_lab8
-launch.py
-estop_controller.cpp
-parkour_policy.json
-test_policy.json
-rebuild_neural_controller.py
-deploy.py
 </code>
-These files are used for controller setup and deployment. They are not necessarily the full training environment.
+Expected repository structure:
-The reward function is not modified inside ''config.yaml'' and it is not modified inside ''.json'' policy files.
+<code>
+~/pupper_lab8/
+├── isaacgym/
+├── leggedgym/
+├── rsl_rl/
+├── pytorch_isaacgym.sif
+├── pyuser_isaac/
+├── conda_tools/
+├── local_include/
+├── torch_extensions/
+├── logs/
+├── lab8_gpu_check.slurm
+├── lab8_import_check.slurm
+├── lab8_train_isaacgym.slurm
+└── lab8_reward_check.sh
+</code>
-Important:
+The archive contains only the small helper scripts. It does not contain the large simulator files, the container image or the full repositories.
-    * ''.json'' files are already trained policies;
+===== 6. Connect to the HPC cluster =====
-    * ''config.yaml'' points the controller to a policy file;
-    * ''rebuild_neural_controller.py'' prepares the controller on the Raspberry Pi;
-    * ''deploy.py'' loads a trained policy on the robot;
-    * reward tuning is done in the official training notebook or training code.
-If the full training notebook/environment is not available, students must ask the instructor for the official Lab 5 training material. The deploy repository alone is not enough for full Reinforcement Learning training.
+Connect to the frontend node:
-===== Required resources =====
+<code bash>
+ssh your_username@fep.grid.pub.ro
+</code>
-Software:
+Go to the lab directory:
-    * Python 3.9 or newer;
+<code bash>
-    * Google Colab or a local Linux/WSL environment;
+cd ~/pupper_lab8
-    * MuJoCo/MJX;
+</code>
-    * JAX;
-    * Weights & Biases;
-    * the official Lab 5 training notebook or training repository;
-    * the Lab 5 deploy repository, if the real robot is used.
-Optional hardware:
+Create the logs directory if it does not already exist:
-    * physical Pupper robot;
+<code bash>
-    * Raspberry Pi on the robot;
+mkdir -p logs
-    * controller;
+</code>
-    * stable power supply;
-    * safe testing area.
-===== Preparing the Lab 5 repository =====
+===== 7. Check that the GPU is available =====
-Clone the Lab 5 repository indicated by the instructor:
+Before running Isaac Gym, check that your SLURM job can access the GPU.
+Submit the GPU check job:
 <code bash>
-cd ~
+cd ~/pupper_lab8
-git clone https://github.com/cs123-stanford/lab_5_fall_2025.git
+sbatch lab8_gpu_check.slurm
-cd lab_5_fall_2025
 </code>
-Check the files:
+Check the queue:
 <code bash>
-ls
+squeue -u $USER
 </code>
-If you only see deploy/controller files and policy ''.json'' files, this repository is mainly for deploying a trained policy. The actual reward tuning and training must be done in the official training notebook or training environment.
+After the job finishes, inspect the output:
-===== Opening the official training environment =====
+<code bash>
+ls logs
+cat logs/gpu_check_<JOB_ID>.out
+cat logs/gpu_check_<JOB_ID>.err
+</code>
-Open the official Lab 5 training notebook or training environment provided by the instructor.
+Replace `<JOB_ID>` with the job id printed by `sbatch`.
-The training environment is the place where you should find:
+A successful run should show an NVIDIA GPU through `nvidia-smi`.
-    * reward configuration;
+===== 8. Check that Isaac Gym and Legged Gym work =====
-    * reward terms;
-    * PPO configuration;
-    * simulation configuration;
-    * command sampling;
-    * termination conditions;
-    * training loop;
-    * video generation;
-    * WandB logging.
-Do not try to modify the reward function in ''config.yaml'' from the deploy folder. That file is for loading a trained policy, not for training a new one.
+Before changing the reward function, check that the simulator imports correctly.
-===== Setting up Weights & Biases =====
+Submit the import check job:
-Weights & Biases is used to monitor training experiments.
+<code bash>
+cd ~/pupper_lab8
-Create or use an existing WandB account. Then follow the instructions from the training notebook to log in.
+sbatch lab8_import_check.slurm
+</code>
-Typical command:
+After the job finishes:
 <code bash>
-wandb login
+cat logs/import_check_<JOB_ID>.out
+cat logs/import_check_<JOB_ID>.err
 </code>
-or, inside the notebook, paste your WandB API key when requested.
+A successful import check should contain:
-During training, WandB will record:
+<code>
+isaacgym import: OK
+gymtorch import: OK
+rsl_rl import: OK
+legged_gym import: OK
+CUDA available: True
+</code>
-    * total reward;
+If this step fails, do not continue to the reward task. Ask the instructor or lab assistant for help.
-    * reward components;
-    * policy performance;
-    * training videos;
-    * run name;
-    * training configuration.
-Each team should save the WandB run link for the final report.
+===== 9. Run a small debug training job =====
-===== Mission 1 - Running the baseline policy =====
+Now check that the Pupper training task can start.
-Before modifying the reward function, run or inspect the baseline policy.
+Submit a small debug job:
-The goal is to understand how the default policy behaves before training a new version.
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
+</code>
-Observe:
+Check the queue:
-    * Does Pupper walk forward?
+<code bash>
-    * Is the movement stable?
+squeue -u $USER
-    * Does it fall?
+</code>
-    * Does it move smoothly?
-    * Does it respond to velocity commands?
-    * What does the reward curve look like?
-Deliverable for this mission:
+After the job finishes, check its output:
-    * one screenshot or video of the baseline behaviour;
+<code bash>
-    * 3-5 lines describing the baseline walking behaviour.
+cat logs/pupper_train_<JOB_ID>.out
+cat logs/pupper_train_<JOB_ID>.err
+</code>
-===== Mission 2 - Inspecting the reward function =====
+A successful run should contain messages similar to:
-Find the reward section in the official training notebook or training code.
+<code>
+Physics Engine: PhysX
+Physics Device: cuda:0
+GPU Pipeline: enabled
+Learning iteration 0/10
+Learning iteration 9/10
+Training finished.
+</code>
-Identify the terms used for the reward function.
+This means that Isaac Gym, PyTorch, CUDA and the Pupper environment are working.
-Look for terms such as:
+===== 10. Run the initial baseline =====
+Now run a slightly larger baseline:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="baseline_zero_reward_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
+After the job finishes, inspect the reward values:
+<code bash>
+cat logs/pupper_train_<JOB_ID>.out | grep -E "Learning iteration|Mean reward|rew_forward_velocity|episode length" | tail -80
+</code>
+You should observe that the reward is not useful. In the initial version, the relevant reward functions return zero.
+Example:
 <code>
-tracking_lin_vel
+Mean reward: 0.00
-tracking_ang_vel
+Mean episode rew_forward_velocity: 0.0000
-effort
-torque
-stability
-orientation
-smoothness
-height
-termination
 </code>
-Write down:
+This is expected before solving the lab.
-    * the name of each reward term;
+===== 11. Inspect the reward functions =====
-    * whether it is a reward or a penalty;
-    * what behaviour it encourages or discourages;
-    * the coefficient used for that term.
-Complete the table:
+Open the Pupper environment file:
-^ Reward term ^ Reward or penalty? ^ Behaviour affected ^ Coefficient ^
+<code bash>
-| tracking_lin_vel | | | |
+cd ~/pupper_lab8/leggedgym
-| tracking_ang_vel | | | |
+nano legged_gym/envs/pupper/pupper.py
-| effort / torque | | | |
+</code>
-| stability / orientation | | | |
-| smoothness | | | |
-===== Mission 3 - Training for velocity tracking =====
+Find the following functions:
-In this mission, the goal is to encourage Pupper to follow a desired forward velocity.
+<code python>
+def _reward_base_height(self):
+    return 0.0
-Modify the reward configuration so that velocity tracking has a stronger influence.
+def _reward_forward_velocity(self):
+return 0
-Example idea:
+def _reward_torques(self):
+return 0 </code>
+These functions are the main TODOs of this lab.
+You can also use the helper script:
+<code bash>
+cd ~/pupper_lab8
+./lab8_reward_check.sh
+</code>
+This script prints the reward functions and checks whether there are still `return 0` statements inside `pupper.py`.
+===== 12. Inspect the reward configuration =====
+Open the configuration file:
+<code bash>
+cd ~/pupper_lab8/leggedgym
+nano legged_gym/envs/pupper/pupper_config.py
+</code>
+Look for the reward section. It should contain values similar to:
 <code python>
-tracking_lin_vel = higher value
+class rewards:
-tracking_ang_vel = moderate value
+    forward_velocity_clip = 1.0
-effort penalty = low or disabled
+```
+class scales:
+    forward_velocity = 3.0
+```
 </code>
-Use the exact variable names and configuration format from the official training notebook.
+The exact values may differ depending on the starter code version.
-Run the training experiment.
+The important idea is that the config file contains the coefficients used to scale the reward terms.
-While training, monitor:
+For example:
-    * total reward;
+    * a positive scale means the term increases the reward;
-    * tracking reward;
+    * a negative scale means the term becomes a penalty;
-    * generated videos;
+    * a zero scale disables the term.
-    * stability of the policy;
-    * whether the robot moves too aggressively.
-Deliverables:
+===== 13. Task 1 - Implement forward velocity reward =====
-    * WandB run link;
+The robot should receive a positive reward when it moves forward.
-    * screenshot of reward curve;
-    * video or generated animation;
-    * short explanation of how increasing velocity tracking affected the behaviour.
-===== Mission 4 - Adding effort penalty =====
+In Legged Gym, the forward velocity of the robot base is usually stored in:
-A robot that moves fast but uses too much energy is not necessarily a good robot. In this mission, add or increase the effort penalty.
+<code python>
+self.base_lin_vel[:, 0]
+</code>
-Example idea:
+This is the x-axis linear velocity of the robot base.
+Replace the initial function:
 <code python>
-tracking_lin_vel = keep similar value
+def _reward_forward_velocity(self):
-effort / torque penalty = increase penalty
+    return 0
-smoothness = optional
 </code>
-Run the training again.
+with:
-Compare this run with the velocity-focused run.
+<code python>
+def _reward_forward_velocity(self):
+    return torch.clip(
+        self.base_lin_vel[:, 0],
+        min=0.0,
+        max=self.cfg.rewards.forward_velocity_clip
+    )
+</code>
-Observe:
+This reward gives positive values only when the robot moves forward. Negative velocity is clipped to zero, so moving backwards is not rewarded.
-    * Did the total reward increase or decrease?
+If `torch` is not imported at the top of the file, add:
-    * Did the movement become slower?
-    * Did the movement become smoother?
-    * Did the robot become more stable?
-    * Was there a trade-off between speed and energy use?
-Deliverables:
+<code python>
+import torch
+</code>
-    * WandB run link;
+===== 14. Task 2 - Implement torque penalty =====
-    * reward curve comparison;
-    * short explanation of the trade-off between speed and effort.
-===== Mission 5 - Stability and smoothness =====
+A robot should not learn to move by using extremely large motor torques. Large torques are inefficient and can produce unstable behavior.
-In this mission, the goal is to obtain a more natural and stable gait.
+The torque values are stored in:
-Modify the reward configuration by adding or increasing terms related to:
+<code python>
+self.torques
+</code>
-    * body stability;
+Replace:
-    * body orientation;
-    * smooth movement;
-    * fall prevention;
-    * height maintenance.
-Example idea:
+<code python>
+def _reward_torques(self):
+    return 0
+</code>
+with:
 <code python>
-tracking_lin_vel = medium value
+def _reward_torques(self):
-effort penalty = moderate value
+    return torch.sum(torch.square(self.torques), dim=1)
-stability / orientation = higher value
-smoothness = moderate penalty
 </code>
-Run the training experiment.
+This function returns a positive value representing how much torque the robot uses.
-Observe:
+Important: this function returns a positive value, but it becomes a penalty if the scale in the config file is negative.
-    * Does the robot fall less often?
+For example:
-    * Are the movements less aggressive?
-    * Does the robot maintain a better body posture?
-    * Is the walking behaviour more natural?
-    * Did the robot become slower?
-Deliverables:
+<code python>
+torques = -0.0002
+</code>
-    * WandB run link;
+means that large torques reduce the final reward.
-    * training video or animation;
-    * short explanation of how stability and smoothness terms affected the gait.
-===== Mission 6 - Team reward configuration =====
+===== 15. Task 3 - Implement base height penalty =====
-Each team must design its own reward configuration.
+The robot should keep its body at a reasonable height. If the body is too low or too high, the behavior is probably unstable.
-The goal is to obtain the best balance between:
+The base height is stored in:
-    * forward walking;
+<code python>
-    * stability;
+self.root_states[:, 2]
-    * energy efficiency;
+</code>
-    * smoothness;
-    * command following.
-Your team may modify:
+Replace:
-    * reward coefficients;
+<code python>
-    * target velocity range;
+def _reward_base_height(self):
-    * command sampling;
+    return 0.0
-    * termination conditions, if allowed by the notebook;
+</code>
-    * number of training steps, if time allows.
-The team must explain:
+with:
-    * which reward terms were changed;
+<code python>
-    * why those terms were changed;
+def _reward_base_height(self):
-    * what behaviour was expected;
+    base_height = self.root_states[:, 2]
-    * what behaviour was obtained;
+    return torch.square(base_height - self.cfg.rewards.base_height_target)
-    * what trade-off was accepted.
+</code>
-Deliverables:
+This function returns the squared error between the current base height and the target base height.
-    * final configuration;
+Again, this becomes a penalty if its scale in the config file is negative.
-    * WandB run link;
-    * final video or generated animation;
-    * explanation of the design choices.
-===== Mission 7 - Comparing policies =====
+===== 16. Check your code =====
-Compare at least three policies:
+After editing `pupper.py`, check the relevant lines:
-    * baseline policy;
+<code bash>
-    * velocity-focused policy;
+cd ~/pupper_lab8/leggedgym
-    * effort/stability-focused policy;
+nl -ba legged_gym/envs/pupper/pupper.py | sed -n '80,120p'
-    * team policy.
+</code>
-Complete the table:
+You should see the implemented reward functions, not `return 0`.
-^ Policy ^ Main reward focus ^ Average reward / final reward ^ Behaviour observed ^ Main problem ^
+You can also run:
-| Baseline | | | | |
-| Velocity-focused | | | | |
-| Effort/stability-focused | | | | |
-| Team policy | | | | |
-Answer:
+<code bash>
+cd ~/pupper_lab8
+./lab8_reward_check.sh
+</code>
-    * Which policy achieved the highest reward?
+If the script still shows `return 0` inside the reward functions, your implementation is not complete.
-    * Which policy looked most stable?
-    * Which policy looked most natural?
-    * Is the policy with the highest reward always the best one?
-    * What would you change if you had more training time?
-===== Mission 8 - Understanding sim-to-real gap =====
+===== 17. Train again after implementing the rewards =====
-A policy trained in simulation may not work perfectly on the real robot. This difference is called the sim-to-real gap.
+Run the training job again:
-Possible causes:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_512_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-    * friction is different in the real world;
+Check the job:
-    * servos do not behave exactly like the simulated motors;
-    * the robot mass may not match the simulated model;
-    * the floor surface may change the behaviour;
-    * sensors may be noisy;
-    * the battery level may affect the robot;
-    * the controller may have delays.
-Answer in the report:
+<code bash>
+squeue -u $USER
+</code>
-    * Why is sim-to-real transfer difficult?
+After it finishes:
-    * Which reward terms could help make the policy more robust?
-    * How could domain randomization help?
-===== Optional mission - Domain randomization =====
+<code bash>
+cat logs/pupper_train_<JOB_ID>.out | grep -E "Learning iteration|Mean reward|rew_forward_velocity|episode length" | tail -100
+cat logs/pupper_train_<JOB_ID>.err
+</code>
-If the training environment supports it, enable or modify domain randomization.
+Compare the new output with the initial baseline.
-Examples of randomized parameters:
+You should focus on:
-    * robot mass;
+    * `Mean reward`;
-    * friction;
+    * `Mean episode rew_forward_velocity`;
-    * motor strength;
+    * `Mean episode length`;
-    * sensor noise;
+    * total timesteps;
-    * terrain;
+    * whether the job completed successfully.
-    * control delay.
-Train again and compare the policy with the previous version.
+===== 18. Larger training runs =====
-Observe:
+After the small run works, you can try larger experiments.
-    * Is the policy more robust?
+Run with 1000 environments:
-    * Does it work better on varied terrain?
-    * Did the training become harder?
-    * Did the reward curve become noisier?
-Deliverable:
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_1000_${USER}" NUM_ENVS=1000 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-    * short explanation of what was randomized;
+If that works, try 2000 environments:
-    * comparison with a non-randomized policy.
-===== Optional mission - Preparing the real Pupper robot =====
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_2000_${USER}" NUM_ENVS=2000 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
-This part is done only if the physical Pupper robot is available.
+For a longer training run:
-The script ''rebuild_neural_controller.py'' must be executed on the robot's Raspberry Pi, not on Windows and not in the training notebook.
+<code bash>
+cd ~/pupper_lab8
+RUN_NAME="reward_fixed_long_${USER}" NUM_ENVS=2000 MAX_ITERATIONS=300 sbatch lab8_train_isaacgym.slurm
+</code>
-Connect to the Raspberry Pi:
+Do not start with the largest experiment. First check that the short run works.
+===== 19. Useful SLURM commands =====
+^ What you want to do ^ Command ^
+| Submit GPU check | ''sbatch lab8_gpu_check.slurm'' |
+| Submit import check | ''sbatch lab8_import_check.slurm'' |
+| Submit training | ''sbatch lab8_train_isaacgym.slurm'' |
+| Submit debug training | ''RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm'' |
+| Check your jobs | ''squeue -u $USER'' |
+| Check finished job status | ''sacct -j <JOB_ID> --format=JobID,JobName,Partition,State,Elapsed,ExitCode,MaxRSS,ReqMem'' |
+| Show output log | ''cat logs/pupper_train_<JOB_ID>.out'' |
+| Show error log | ''cat logs/pupper_train_<JOB_ID>.err'' |
+| Show only reward lines | ''cat logs/pupper_train_<JOB_ID>.out | grep -E "Mean reward|rew_forward_velocity|episode length"'' |
+| Inspect reward functions | ''./lab8_reward_check.sh'' |
+===== 20. Common problems =====
+==== Problem 1 - The reward stays zero ====
+If the output still shows:
+<code>
+Mean reward: 0.00
+Mean episode rew_forward_velocity: 0.0000
+</code>
+check that you actually modified:
+<code>
+~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
+</code>
+and that the reward functions no longer return zero.
+Use:
 <code bash>
-ssh pi@IP_ROBOT
+cd ~/pupper_lab8
+./lab8_reward_check.sh
 </code>
-Clone the deploy repository on the Raspberry Pi:
+or:
 <code bash>
-cd ~
+grep -R "return 0" ~/pupper_lab8/leggedgym/legged_gym/envs/pupper/pupper.py
-git clone https://github.com/cs123-stanford/lab_5_fall_2025.git
-cd lab_5_fall_2025
 </code>
-Check the required files:
+==== Problem 2 - The job is killed with OOM ====
+If the job fails with:
+<code>
+Detected 1 oom_kill event
+Some of the step tasks have been OOM Killed
+</code>
+then the job used too much RAM.
+Use fewer environments:
 <code bash>
-ls config.yaml launch.py estop_controller.cpp parkour_policy.json test_policy.json
+NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
 </code>
-Run:
+or:
 <code bash>
-python3 rebuild_neural_controller.py
+NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
 </code>
-If you get the error ''Source file does not exist'', check that:
+If needed, the SLURM memory limit can be increased by the instructor in the training script:
-    * you are on the Raspberry Pi;
+<code bash>
-    * the repository is located in ''/home/pi/lab_5_fall_2025'';
+#SBATCH --mem=128G
-    * the required files exist;
+</code>
-    * the Pupper ROS2 workspace exists.
-===== Optional mission - Deploying the trained policy on Pupper =====
+==== Problem 3 - Task not registered ====
-After training a policy, deploy it only if the physical robot is available and the instructor confirms that the robot is ready for testing.
+If you see:
+<code>
+ValueError: Task with name: ... was not registered
+</code>
+check that the task name is:
+<code>
+pupper_flat
+</code>
-On the Raspberry Pi:
+You can inspect registered tasks with:
 <code bash>
-cd ~/lab_5_fall_2025
+cd ~/pupper_lab8/leggedgym
-python3 deploy.py
+grep -R "task_registry.register" -n legged_gym/envs
 </code>
-Select the trained policy or WandB run indicated by the instructor.
+==== Problem 4 - You edited the wrong task ====
-Test the robot in a safe space.
+This lab uses:
-Observe:
+<code>
+pupper_flat
+</code>
-    * Does the policy load correctly?
+Do not confuse it with:
-    * Does the robot stand safely?
-    * Does the robot walk?
-    * Does it fall?
-    * Are the movements smooth or aggressive?
-    * Does the behaviour match the simulation?
-Deliverables:
+<code>
+pupper_standup
+</code>
-    * short video of the real robot;
+The standup task may have different reward scales, including zero forward velocity reward.
-    * short comparison between simulation and real behaviour.
-===== What to submit =====
+==== Problem 5 - The output stops after gymtorch ====
-Each team must submit:
+If the output stops around:
-    * the reward table from Mission 2;
+<code>
-    * WandB links for the training runs;
+Building extension module gymtorch...
-    * screenshots of the reward curves;
+ninja: no work to do.
-    * generated videos or animations;
+</code>
-    * the final reward configuration;
-    * the policy comparison table;
-    * answers to the report questions;
-    * optional: real robot video, if deployed.
-===== Report structure =====
+check the error file:
-The report should have 2-3 pages and include:
+<code bash>
+cat logs/pupper_train_<JOB_ID>.err
+</code>
-    * team members;
+If the job completed successfully, use:
-    * baseline policy observations;
-    * reward terms identified;
-    * experiments performed;
-    * reward curve screenshots;
-    * comparison between policies;
-    * final reward configuration;
-    * sim-to-real discussion;
-    * conclusions.
-===== Questions for the report =====
+<code bash>
+sacct -j <JOB_ID> --format=JobID,JobName,Partition,State,Elapsed,ExitCode,MaxRSS,ReqMem
+</code>
-Answer briefly:
+===== 21. Questions =====
-    * What is the role of the reward function in Reinforcement Learning?
+Answer the following questions in your report:
-    * Why is Pupper trained in simulation before being tested on the real robot?
-    * What is the advantage of running many simulated environments in parallel?
+    * Why did the initial training run produce zero reward?
-    * What happened when velocity tracking was emphasized?
+    * Why is forward velocity a useful reward term for locomotion?
-    * What happened when effort was penalized?
+    * Why should large torques be penalized?
-    * Why are stability and smoothness important?
+    * Why can body height be used as a stability-related reward term?
+    * What is the purpose of training many environments in parallel?
+    * What changed after you implemented the reward functions?
+    * Did the robot learn better behavior after more iterations? Explain using the log values.
+    * Why might a policy that works in simulation behave differently on the real robot?
+===== 22. Deliverables =====
+Submit a short report containing:
+    * your implemented reward functions;
+    * a screenshot or copied log section from the baseline run;
+    * a screenshot or copied log section from the improved reward run;
+    * a small comparison table;
+    * short answers to the lab questions.
+Example comparison table:
+^ Run name ^ NUM_ENVS ^ MAX_ITERATIONS ^ Mean reward ^ rew_forward_velocity ^ Observation ^
+| baseline_zero_reward | 512 | 50 | 0.00 | 0.0000 | No useful learning signal |
+| reward_fixed_512 | 512 | 50 | ... | ... | Reward functions implemented |
+| reward_fixed_1000 | 1000 | 50 | ... | ... | More parallel environments |
+===== 23. What to remember =====
+The most important idea in this lab is that reinforcement learning does not magically learn the behavior we want.
+The agent learns what the reward function encourages.
+If the reward function is zero, the robot has no reason to improve.
+If the reward function encourages forward movement but also penalizes unstable or inefficient behavior, the robot has a better chance of learning useful locomotion.
+===== 24. Optional final step - Upload the trained policy to the real Pupper robot =====
+After training a policy in simulation, the next step is to test it on the real Pupper robot.
+This step should only be done under instructor supervision.
+Before uploading anything to the real robot, make sure that:
+    * the robot battery is charged;
+    * the robot is placed on the floor in a safe open area;
+    * the emergency stop is available;
+    * the policy was tested in simulation;
+    * the correct configuration file is used on the robot;
+    * the instructor or lab assistant is present.
+==== 24.1 Find the trained policy ====
+After training, the policy is saved inside the Legged Gym logs directory.
+Use:
+<code bash>
+cd ~/pupper_lab8/leggedgym
+find logs -name "    *.pt" | tail -20
+</code>
+Look for a file similar to:
+<code>
+model_300.pt
+model_1500.pt
+</code>
+The exact name depends on the number of training iterations.
+==== 24.2 Copy the policy to a deployment folder ====
+Go to the deployment repository or folder provided by the instructor.
+Example:
+<code bash>
+cd ~/pupper_lab8
+mkdir -p deploy_policy
+</code>
+Copy the trained model:
+<code bash>
+cp ~/pupper_lab8/leggedgym/logs/<experiment_folder>/model_<iteration>.pt ~/pupper_lab8/deploy_policy/
+</code>
+Replace `<experiment_folder>` and `<iteration>` with the real names from your training output.
+==== 24.3 Convert or rebuild the neural controller ====
+Some Pupper deployment code does not use the raw `.pt` file directly. It may require rebuilding or exporting the neural controller.
+If the deployment folder contains a script such as:
+<code>
+rebuild_neural_controller.py
+</code>
+run it according to the instructor’s instructions.
+Example:
+<code bash>
+cd ~/pupper_lab8/deploy
+python rebuild_neural_controller.py
+</code>
+The exact command may differ depending on the deployment package used in the lab.
+==== 24.4 Upload the controller to Pupper ====
+Connect to the Pupper robot using SSH.
+Example:
+<code bash>
+ssh pi@pupper.local
+</code>
+or, if the robot has a fixed IP address:
+<code bash>
+ssh pi@<PUPPER_IP_ADDRESS>
+</code>
+From your local or HPC environment, copy the generated controller or policy files to the robot:
+<code bash>
+scp -r ~/pupper_lab8/deploy_policy/    * pi@<PUPPER_IP_ADDRESS>:~/pupper_deploy/policies/
+</code>
+Replace `<PUPPER_IP_ADDRESS>` with the real IP address of the robot.
+==== 24.5 Run the policy on the robot ====
+On the Pupper robot:
+<code bash>
+cd ~/pupper_deploy
+python launch.py
+</code>
+or use the command provided by the instructor for the specific robot setup.
+Observe the robot carefully.
+Stop the program immediately if:
+    * the robot moves violently;
+    * the joints oscillate strongly;
+    * the robot falls repeatedly;
+    * the motors overheat;
+    * the emergency stop is needed.
+==== 24.6 Reflection question ====
+Compare the behavior in simulation with the behavior on the real robot.
+Answer:
+    * Did the robot behave the same in simulation and reality?
+    * What differences did you observe?
+    * Why can a policy trained in simulation fail on a real robot?
     * What is the sim-to-real gap?
     * How could domain randomization help?
-    * Which policy was the best and why?
-===== Common problems =====
+===== 25. Instructor notes =====
-If the training notebook is not available, ask the instructor for the official Lab 5 training material. The deploy repository alone is not enough for full RL training.
+This section is for the instructor or lab assistant.
-If WandB does not work, check your API key and internet connection.
+The tested working stack on the HPC cluster was:
-If training is very slow, check whether GPU/accelerator support is enabled.
+    * SLURM job on the `dgxa100` partition;
+    * NVIDIA A100-SXM4-80GB GPU;
+    * Apptainer with `--nv`;
+    * PyTorch 1.10.0 CUDA 11.3 container;
+    * Isaac Gym Preview 4;
+    * Python 3.7;
+    * Legged Gym;
+    * `pupper_flat` task.
-If MuJoCo, MJX or JAX fails to import, check the environment setup instructions from the official training notebook.
+The final import check must confirm:
-If the generated videos do not appear, check the notebook output folder and WandB media tab.
+<code>
+isaacgym import: OK
+gymtorch import: OK
+rsl_rl import: OK
+legged_gym import: OK
+CUDA available: True
+</code>
-If ''rebuild_neural_controller.py'' gives errors involving ''/home/pi'', it is probably being run in the wrong environment. It must be run on the Raspberry Pi of the robot.
+The final training script must export the same environment variables used during the successful import check, especially:
-If the policy works in simulation but not on the real robot, discuss the sim-to-real gap instead of assuming that the training failed.
+<code bash>
+export PYTHONUSERBASE=$LAB_DIR/pyuser_isaac
+export TOOL_PREFIX=$LAB_DIR/conda_tools
+export PATH=$TOOL_PREFIX/bin:$PYTHONUSERBASE/bin:$PATH
+export PYTHONPATH=$PYTHONUSERBASE/lib/python3.7/site-packages:${PYTHONPATH:-}
+export LD_LIBRARY_PATH=$TOOL_PREFIX/lib:/opt/conda/lib:${LD_LIBRARY_PATH:-}
+export LD_PRELOAD=$TOOL_PREFIX/lib/libstdc++.so.6:$TOOL_PREFIX/lib/libgcc_s.so.1
+export CPATH=$LAB_DIR/local_include:$TOOL_PREFIX/include:${CPATH:-}
+export CC=$TOOL_PREFIX/bin/x86_64-conda-linux-gnu-gcc
+export CXX=$TOOL_PREFIX/bin/x86_64-conda-linux-gnu-c++
+export TORCH_EXTENSIONS_DIR=$LAB_DIR/torch_extensions
+export MAX_JOBS=1
+</code>
+The final `lab8_train_isaacgym.slurm` should pass the task variables into Apptainer using:
+<code bash>
+export APPTAINERENV_LAB_DIR="$LAB_DIR"
+export APPTAINERENV_TASK="$TASK"
+export APPTAINERENV_NUM_ENVS="$NUM_ENVS"
+export APPTAINERENV_MAX_ITERATIONS="$MAX_ITERATIONS"
+export APPTAINERENV_RUN_NAME="$RUN_NAME"
+</code>
+A safe starting point for students is:
+<code bash>
+RUN_NAME="debug_128_${USER}" NUM_ENVS=128 MAX_ITERATIONS=10 sbatch lab8_train_isaacgym.slurm
+</code>
+A tested larger run is:
+<code bash>
+RUN_NAME="baseline_zero_reward_${USER}" NUM_ENVS=512 MAX_ITERATIONS=50 sbatch lab8_train_isaacgym.slurm
+</code>
+If `NUM_ENVS=2000` causes an OOM kill, reduce the number of environments or increase the requested memory in the SLURM script.
+=====Manual Install=====
+https://developer.nvidia.com/isaac-gym/download
+<code>
+#!/bin/bash
+#SBATCH --job-name=lab8_setup
+#SBATCH --partition=dgxa100
+#SBATCH --gres=gpu:1
+#SBATCH --cpus-per-task=8
+#SBATCH --mem=32G
+#SBATCH --time=02:00:00
+#SBATCH --output=lab8_setup_%j.out
+#SBATCH --error=lab8_setup_%j.err
+set -e
+LAB_DIR="$HOME/pupper_lab8"
+mkdir -p "$LAB_DIR"
+cd "$LAB_DIR"
+echo "LAB_DIR=$LAB_DIR"
+echo "1. Check Apptainer image"
+if [ ! -f pytorch_isaacgym.sif ]; then
+    echo "pytorch_isaacgym.sif not found."
+    echo "Pulling base PyTorch image..."
+    apptainer pull pytorch_isaacgym.sif docker://pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
+else
+    echo "pytorch_isaacgym.sif already exists."
+fi
+echo "2. Check Isaac Gym package"
+if [ ! -f IsaacGym_Preview_4_Package.tar.gz ]; then
+    echo "ERROR: IsaacGym_Preview_4_Package.tar.gz is missing."
+    echo "The instructor must upload it manually to $LAB_DIR."
+    exit 1
+fi
+if [ ! -d isaacgym ]; then
+    echo "Extracting Isaac Gym..."
+    tar -xzf IsaacGym_Preview_4_Package.tar.gz
+else
+    echo "isaacgym folder already exists."
+fi
+echo "3. Clone rsl_rl if missing"
+if [ ! -d rsl_rl ]; then
+    git clone https://github.com/leggedrobotics/rsl_rl.git
+else
+    echo "rsl_rl already exists."
+fi
+echo "4. Clone leggedgym if missing"
+if [ ! -d leggedgym ]; then
+    git clone https://github.com/cs123-stanford/leggedgym.git
+else
+    echo "leggedgym already exists."
+fi
+echo "5. Install Python packages inside Apptainer user base"
+export PYTHONUSERBASE="$LAB_DIR/pyuser_isaac"
+export PATH="$PYTHONUSERBASE/bin:$PATH"
+export PYTHONPATH="$PYTHONUSERBASE/lib/python3.7/site-packages:${PYTHONPATH:-}"
+apptainer exec --nv pytorch_isaacgym.sif bash -lc "
+set -e
+cd $LAB_DIR
+export PYTHONUSERBASE=$PYTHONUSERBASE
+export PATH=$PATH
+export PYTHONPATH=$PYTHONPATH
+python -m pip install --user --upgrade pip
+python -m pip install --user -e isaacgym/python
+python -m pip install --user -e rsl_rl
+python -m pip install --user -e leggedgym
+python - <<'PY'
+import isaacgym
+import torch
+import rsl_rl
+import legged_gym
+print('isaacgym OK')
+print('torch', torch.__version__)
+print('cuda available', torch.cuda.is_available())
+print('rsl_rl OK')
+print('legged_gym OK')
+PY
+"
+echo "Lab 8 setup finished successfully."
+</code>

Labs

Resources

rasb/lab/08.1781591151.txt.gz · Last modified: 2026/06/16 09:25 by vlad.radulescu2901

Show page Old revisions

Media Manager Back to top