Show page

Differences

This shows you the differences between two versions of the page.

--- rasb:lab:09 [2026/06/29 10:55]
vlad.radulescu2901 [4.5 Safety check before the real robot]
+++ rasb:lab:09 [2026/06/29 12:12] (current)
vlad.radulescu2901 [Exercise 6 - Demonstrate the full voice-to-Pupper pipeline]
@@ Line 35: / Line 35: @@
 The folder `llm_lab` is used first for local testing. After the local pipeline works, students must connect the command parser to a real LLM and then use the validated commands to control the real Pupper robot.
+===== Environment setup for ROS2 simulation =====
+This lab uses a ROS2 simulator before commands are tested on the real Pupper robot.
+The recommended environment is:
+<code>
+Ubuntu 24.04 / WSL2
+ROS2 Jazzy
+Python 3 virtual environment
+Ollama or another approved local/instructor-provided LLM
+</code>
+The archive for this lab contains both the LLM sandbox and a minimal ROS2 simulator package:
+<code>
+lab_9_fall_2025/
+├── llm_lab/
+│   ├── command_parser.py
+│   ├── karel_commander.py
+│   ├── karel_pupper.py
+│   ├── ros2_pupper_bridge.py
+│   ├── real_llm.py
+│   ├── run_sim_llm_pipeline.py
+│   ├── run_voice_sim_pipeline.py
+│   └── ...
+└── src/
+    └── llm_pupper_sim/
+        ├── package.xml
+        ├── setup.py
+        ├── launch/
+        │   └── pupper_sim.launch.py
+        └── llm_pupper_sim/
+            ├── pupper_sim_node.py
+            └── pupper_marker_node.py
+</code>
+The ROS2 simulator package subscribes to:
+<code>
+/cmd_vel
+/pupper/high_level_command
+</code>
+and publishes:
+<code>
+/pupper/sim_pose
+/pupper/marker
+</code>
+The LLM must never publish directly to these ROS2 topics.
+The safe structure is:
+<code>
+LLM output
+    -> sanitize_commands()
+    -> KarelPupper API
+    -> ros2_pupper_bridge.py
+    -> ROS2 topics
+    -> simulator
+</code>
+==== Installing ROS2 Jazzy on Ubuntu 24.04 ====
+If ROS2 is not installed, install ROS2 Jazzy first.
+Check the Ubuntu version:
+<code bash>
+lsb_release -a
+</code>
+For Ubuntu 24.04, use ROS2 Jazzy.
+Enable the required repositories:
+<code bash>
+sudo apt update
+sudo apt install software-properties-common curl -y
+sudo add-apt-repository universe -y
+sudo apt update
+</code>
+Add the ROS2 apt source:
+<code bash>
+export ROS_APT_SOURCE_VERSION=$(curl -s https://api.github.com/repos/ros-infrastructure/ros-apt-source/releases/latest | grep -F "tag_name" | awk -F'"' '{print $4}')
+curl -L -o /tmp/ros2-apt-source.deb "https://github.com/ros-infrastructure/ros-apt-source/releases/download/${ROS_APT_SOURCE_VERSION}/ros2-apt-source_${ROS_APT_SOURCE_VERSION}.$(. /etc/os-release && echo $VERSION_CODENAME)_all.deb"
+sudo dpkg -i /tmp/ros2-apt-source.deb
+sudo apt update </code>
+Install ROS2 Jazzy and development tools:
+<code bash>
+sudo apt install ros-jazzy-desktop python3-argcomplete ros-dev-tools -y
+</code>
+Source ROS2:
+<code bash>
+source /opt/ros/jazzy/setup.bash
+</code>
+Optional: add ROS2 to `.bashrc`:
+<code bash>
+echo "source /opt/ros/jazzy/setup.bash" >> ~/.bashrc
+source ~/.bashrc
+</code>
+Check that ROS2 works:
+<code bash>
+ros2 --help
+</code>
+==== Building the simulator package ====
+Go to the root folder of the lab:
+<code bash>
+cd ~/lab_9_fall_2025
+</code>
+or, if the lab is stored inside the Windows filesystem from WSL:
+<code bash>
+cd /mnt/c/Users/<your_user>/Documents/sumer_school/llm_lab/lab_9_fall_2025
+</code>
+Build the simulator package:
+<code bash>
+source /opt/ros/jazzy/setup.bash
+colcon build --packages-select llm_pupper_sim
+source install/setup.bash
+</code>
+A successful build should create:
+<code>
+build/
+install/
+log/
+</code>
+and the local setup file:
+<code>
+install/setup.bash
+</code>
+==== Python environment for the LLM lab ====
+Ubuntu 24 may block global `pip install` commands because the system Python environment is externally managed.
+Use a virtual environment instead.
+From the `llm_lab` folder:
+<code bash>
+cd ~/lab_9_fall_2025/llm_lab
+python3 -m venv --system-site-packages .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install requests sounddevice scipy faster-whisper </code>
+The option `--system-site-packages` is important because it allows the virtual environment to see ROS2 Python packages such as `rclpy`.
+Check that both ROS2 and the lab dependencies are visible:
+<code bash>
+python -c "import rclpy; print('rclpy OK')"
+python -c "import requests; print('requests OK')"
+</code>
+==== Local LLM access ====
+The lab can use a local LLM through Ollama.
+If Ollama runs inside the same Ubuntu / WSL environment, the default URL is:
+<code>
+http://localhost:11434/api/generate
+</code>
+If Ollama runs on Windows and the ROS2 pipeline runs in WSL, expose Ollama from Windows and call it from WSL.
+In Windows PowerShell, start Ollama with:
+<code powershell>
+$env:OLLAMA_HOST="0.0.0.0:11434"
+ollama serve
+</code>
+If Windows Firewall blocks access from WSL, add an inbound rule for port 11434.
+In WSL, find the Windows host IP:
+<code bash>
+WIN_HOST=$(ip route | awk '/default/ {print $3}')
+echo $WIN_HOST
+</code>
+Test access to Ollama:
+<code bash>
+curl http://$WIN_HOST:11434/api/tags
+</code>
+Then set the LLM URL:
+<code bash>
+export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"
+export OLLAMA_MODEL="llama3.2:3b"
+</code>
+If Ollama runs directly in WSL, use:
+<code bash>
+export OLLAMA_URL="http://localhost:11434/api/generate"
+export OLLAMA_MODEL="llama3.2:3b"
+</code>
 ===== Lab idea =====
@@ Line 628: / Line 861: @@
 Before running commands on the real Pupper robot, test the validated command pipeline in simulation.
-The simulator step is important because it lets you check whether the command generated by the LLM is correctly mapped to a robot action before using the physical robot.
+The simulator step is important because it checks whether the command generated by the LLM is correctly mapped to a robot action before using the physical robot.
 The simulated pipeline is:
@@ Line 638: / Line 871: @@
     -> sanitize_commands()
     -> KarelPupper API
-    -> ROS2 simulator / RViz / Gazebo
+    -> ROS2 simulator
 </code>
-The LLM must still not publish directly to ROS2 topics. The output of the LLM must always pass through:
+The LLM must not publish directly to ROS2 topics.
+The output of the LLM must always pass through:
 <code python>
@@ Line 649: / Line 884: @@
 Only validated commands may be sent to the robot API.
-==== 4.1 Start the simulator ====
+==== 4.1 Start the ROS2 simulator ====
-Open a terminal and source the ROS2 workspace used by the Pupper repository.
+Open a terminal and go to the root folder of the lab:
-Example:
 <code bash>
 cd ~/lab_9_fall_2025
+</code>
+or, if the lab is stored inside the Windows filesystem from WSL:
+<code bash>
+cd /mnt/c/Users/<your_user>/Documents/sumer_school/llm_lab/lab_9_fall_2025
+</code>
+Source ROS2 and the local workspace:
+<code bash>
+source /opt/ros/jazzy/setup.bash
 source install/setup.bash
 </code>
-or, if the workspace is in another folder:
+Start the simulator:
 <code bash>
-source ~/ros2_ws/install/setup.bash
+ros2 launch llm_pupper_sim pupper_sim.launch.py
 </code>
-Then start the simulator using the launch file provided by the repository.
+The simulator should print messages similar to:
-Example structure:
+<code>
+Pupper ROS2 simulator started.
+Listening on /cmd_vel and /pupper/high_level_command.
+Pupper RViz marker node started.
+pose x=0.00, y=0.00, theta=0.00, state=STAND
+</code>
+Leave this terminal open.
+==== 4.2 Check the simulator topics ====
+Open a second terminal.
+Go to the same lab folder and source the environment:
 <code bash>
-ros2 launch <pupper_simulation_package> <simulation_launch_file>.py
+cd ~/lab_9_fall_2025
+source /opt/ros/jazzy/setup.bash
+source install/setup.bash </code>
+List the ROS2 topics:
+<code bash>
+ros2 topic list
 </code>
-Use the exact simulation launch command provided by the instructor or by the repository.
+You should see:
-Depending on the repository, the simulator may use RViz, Gazebo, or both.
+<code>
+/cmd_vel
+/pupper/high_level_command
+/pupper/sim_pose
+/pupper/marker
+</code>
-==== 4.2 Connect KarelPupper to the simulated robot ====
+Manually test forward movement:
-The starter version of `KarelPupper` runs in mock mode and only prints actions.
+<code bash>
+ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}"
+</code>
-For the simulator, you must connect the high-level KarelPupper functions to the simulated robot interface.
+Let it run for a few seconds, then stop it with `Ctrl+C`.
-Open:
+In the simulator terminal, the value of `pose x` should change.
+Stop the simulated robot:
 <code bash>
-nano llm_lab/karel_pupper.py
+ros2 topic pub --once /pupper/high_level_command std_msgs/msg/String "{data: 'STOP'}"
 </code>
-Find the high-level methods:
+Manually test turning:
-<code python>
+<code bash>
-def move_forward(self):
+ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.8}}"
-    ...
+</code>
-def turn_left(self):
+Let it run for a few seconds, then stop it with `Ctrl+C`.
-...
-def turn_right(self):
+In the simulator terminal, the value of `theta` should change.
-...
-def sit(self):
+==== 4.3 Connect KarelPupper to the ROS2 simulator ====
-...
-def stand(self):
+The archive already contains a ROS2 bridge:
-...
-def wave(self):
+<code>
-...
+llm_lab/ros2_pupper_bridge.py
+</code>
-def stop(self):
+and a simulator-aware version of:
-... </code>
-Modify the implementation so that these functions send commands to the simulated Pupper robot using the existing ROS2/Pupper interface.
+<code>
+llm_lab/karel_pupper.py
+</code>
-Do not send raw LLM output directly to ROS2.
+The connection is:
+<code>
+KarelPupper(mode="sim")
+    -> ros2_pupper_bridge.py
+    -> /cmd_vel
+    -> /pupper/high_level_command
+    -> ROS2 simulator
+</code>
+The command mapping is:
+<code>
+MOVE_FORWARD  -> /cmd_vel linear.x = 0.20
+MOVE_BACKWARD -> /cmd_vel linear.x = -0.20
+TURN_LEFT     -> /cmd_vel angular.z = 0.80
+TURN_RIGHT    -> /cmd_vel angular.z = -0.80
+STOP          -> zero velocity + STOP high-level command
+SIT           -> /pupper/high_level_command
+STAND         -> /pupper/high_level_command
+WAVE          -> /pupper/high_level_command
+</code>
+Do not modify the LLM so that it publishes directly to ROS2.
 The correct structure is:
@@ Line 723: / Line 1019: @@
 LLM output
     -> sanitize_commands()
-    -> MOVE_FORWARD / TURN_LEFT / SIT / ...
+    -> allowed command
     -> KarelPupper method
     -> ROS2 simulator command
 </code>
-==== 4.3 Test text commands in simulation ====
+==== 4.4 Test text commands in simulation ====
-First test the simulator with text input.
+Keep the simulator running in the first terminal.
-Use at least the following commands:
+In the second terminal, go to the lab folder:
+<code bash>
+cd ~/lab_9_fall_2025
+source /opt/ros/jazzy/setup.bash
+source install/setup.bash
+cd llm_lab
+source .venv/bin/activate </code>
+If the LLM runs through Ollama on Windows, set:
+<code bash>
+WIN_HOST=$(ip route | awk '/default/ {print $3}')
+export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"
+export OLLAMA_MODEL="llama3.2:3b"
+</code>
+If the LLM runs through Ollama inside WSL, set:
+<code bash>
+export OLLAMA_URL="http://localhost:11434/api/generate"
+export OLLAMA_MODEL="llama3.2:3b"
+</code>
+Run the text-to-simulator pipeline:
+<code bash>
+python run_sim_llm_pipeline.py
+</code>
+Test at least the following commands:
 <code>
@@ Line 741: / Line 1069: @@
 Sit down.
 Stop.
+Run into the wall.
 </code>
@@ Line 746: / Line 1075: @@
     * the input text;
-    * the LLM output;
+    * the raw LLM output;
     * the validated command list;
     * the simulator behavior.
@@ Line 753: / Line 1082: @@
 ^ Input command ^ LLM output ^ Validated commands ^ Simulator behavior ^
-| Stand up | STAND | STAND | Simulated robot stands |
+| Stand up | STAND | STAND | Simulator receives STAND high-level command |
-| Move forward | MOVE_FORWARD | MOVE_FORWARD | Simulated robot moves forward |
+| Move forward | MOVE_FORWARD | MOVE_FORWARD | `pose x` changes in the ROS2 simulator |
-| Turn left | TURN_LEFT | TURN_LEFT | Simulated robot turns left |
+| Turn left | TURN_LEFT | TURN_LEFT | `theta` changes in the ROS2 simulator |
-| Stop | STOP | STOP | Simulated robot stops |
+| Stop | STOP | STOP | Simulator receives STOP |
+| Run into the wall | STOP | STOP | Unsafe command is rejected and converted to STOP |
-==== 4.4 Test voice commands in simulation ====
+==== 4.5 Test voice commands in simulation ====
 After text commands work, test the same pipeline using voice input.
@@ Line 765: / Line 1095: @@
 <code>
-microphone
+microphone or recorded audio file
     -> speech-to-text
     -> transcribed text
@@ Line 771: / Line 1101: @@
     -> sanitize_commands()
     -> KarelPupper API
-    -> ROS2 simulator / RViz / Gazebo
+    -> ROS2 simulator
+</code>
+Run:
+<code bash>
+python run_voice_sim_pipeline.py
 </code>
@@ Line 782: / Line 1118: @@
 Turn right and sit.
 Stop.
+Run into the wall.
 </code>
@@ Line 788: / Line 1125: @@
     * the spoken command;
     * the transcription;
-    * the LLM output;
+    * the raw LLM output;
     * the validated commands;
     * the simulator behavior.
@@ Line 795: / Line 1132: @@
 ^ Spoken command ^ Transcription ^ LLM output ^ Validated commands ^ Simulator behavior ^
-| Move forward | Move forward | MOVE_FORWARD | MOVE_FORWARD | Simulated robot moves forward |
+| Move forward | Move forward | MOVE_FORWARD | MOVE_FORWARD | `pose x` changes in the ROS2 simulator |
-| Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | Simulated robot turns left, then sits |
+| Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | `theta` changes, then simulator receives SIT |
-| Stop | Stop | STOP | STOP | Simulated robot stops |
+| Stop | Stop | STOP | STOP | Simulator receives STOP |
+| Run into the wall | Run into the wall | STOP | STOP | Unsafe command is rejected and converted to STOP |
-==== 4.5 Safety check before the real robot ====
+If the microphone is not available inside WSL, record a short `.wav` file on Windows and process that file from WSL.
+The pipeline is still valid:
+<code>
+recorded voice command
+    -> .wav file
+    -> speech-to-text
+    -> transcribed text
+    -> real LLM parser
+    -> sanitize_commands()
+    -> ROS2 simulator
+</code>
+==== 4.6 Safety check before the real robot ====
 Before moving to the real robot, verify that:
@@ Line 807: / Line 1159: @@
     * unsafe commands are converted to `STOP`;
     * the robot does not execute raw text;
-    * the same `KarelPupper` interface can be reused for the real robot.
+    * the LLM does not publish directly to ROS2 topics;
+    * the same `KarelPupper` interface can later be reused for the real robot.
 Only after the simulator pipeline works should you continue to the real Pupper robot.
@@ Line 819: / Line 1172: @@
 Before running anything on the real robot:
-* test the command in mock mode;
+    * test the command in mock mode;
-* test the command in the simulator;
+    * test the command in the simulator;
-* make sure the emergency stop is available;
+    * make sure the emergency stop is available;
-* place the robot in a safe open area;
+    * place the robot in a safe open area;
-* use only high-level KarelPupper commands;
+    * use only high-level KarelPupper commands;
-* do not bypass the safety parser.
+    * do not bypass the safety parser.
 The real robot pipeline must be:
@@ Line 865: / Line 1218: @@
 Your demonstration must include:
-* one single-step command;
+    * one single-step command;
-* one two-step command;
+    * one two-step command;
-* one command that makes the robot stop;
+    * one command that makes the robot stop;
-* one command that is rejected or converted to `STOP`;
+    * one command that is rejected or converted to `STOP`;
-* one command chosen by you.
+    * one command chosen by you.
 Example spoken commands:
@@ Line 884: / Line 1237: @@
 For each demonstrated command, record:
-* the spoken command;
+    * the spoken command;
-* the transcription;
+    * the transcription;
-* the LLM output;
+    * the LLM output;
-* the validated command list;
+    * the validated command list;
-* the observed robot behavior.
+    * the observed robot behavior.
 Example table:
@@ Line 897: / Line 1250: @@
 | Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | Robot turns left, then sits |
 | Run into the wall | Run into the wall | STOP | STOP | Robot stops / does not execute unsafe movement |
+===== Common setup problems =====
+==== Problem 1 - `ros2: command not found` ====
+ROS2 is not sourced or not installed.
+Check:
+<code bash>
+ls /opt/ros
+</code>
+For Ubuntu 24.04, source Jazzy:
+<code bash>
+source /opt/ros/jazzy/setup.bash
+</code>
+Then source the local workspace:
+<code bash>
+source install/setup.bash
+</code>
+==== Problem 2 - `externally-managed-environment` when using pip ====
+Ubuntu 24 protects the system Python environment.
+Use a virtual environment:
+<code bash>
+cd llm_lab
+python3 -m venv --system-site-packages .venv
+source .venv/bin/activate
+python -m pip install requests sounddevice scipy faster-whisper
+</code>
+Do not install lab packages globally into the system Python.
+==== Problem 3 - WSL cannot connect to Ollama running on Windows ====
+If Ollama runs on Windows and the pipeline runs in WSL, Ollama must be accessible from WSL.
+In Windows PowerShell:
+<code powershell>
+$env:OLLAMA_HOST="0.0.0.0:11434"
+ollama serve
+</code>
+If WSL still cannot connect, allow inbound TCP traffic on port 11434 in Windows Firewall.
+In WSL:
+<code bash>
+WIN_HOST=$(ip route | awk '/default/ {print $3}')
+curl http://$WIN_HOST:11434/api/tags
+</code>
+Then set:
+<code bash>
+export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"
+</code>
+==== Problem 4 - The simulator does not move ====
+First check that the ROS2 topics exist:
+<code bash>
+ros2 topic list
+</code>
+Then publish a manual velocity command:
+<code bash>
+ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}"
+</code>
+If `pose x` changes in the simulator terminal, the simulator works.
+==== Problem 5 - Microphone does not work in WSL ====
+Some WSL installations do not expose the microphone correctly.
+If microphone recording fails, record a short `.wav` file on Windows and process the audio file from WSL.
+This still satisfies the voice pipeline requirement because the command starts as spoken audio.
 ===== Deliverables =====

Labs

Resources

rasb/lab/09.1782719729.txt.gz · Last modified: 2026/06/29 10:55 by vlad.radulescu2901

Show page Old revisions

Media Manager Back to top