Show page

Differences

This shows you the differences between two versions of the page.

--- rasb:lab:09 [2026/06/29 12:09]
vlad.radulescu2901 [Downloading the llm_lab starter folder]
+++ rasb:lab:09 [2026/06/29 12:12] (current)
vlad.radulescu2901 [Exercise 6 - Demonstrate the full voice-to-Pupper pipeline]
@@ Line 861: / Line 861: @@
 Before running commands on the real Pupper robot, test the validated command pipeline in simulation.
-The simulator step is important because it lets you check whether the command generated by the LLM is correctly mapped to a robot action before using the physical robot.
+The simulator step is important because it checks whether the command generated by the LLM is correctly mapped to a robot action before using the physical robot.
 The simulated pipeline is:
@@ Line 871: / Line 871: @@
     -> sanitize_commands()
     -> KarelPupper API
-    -> ROS2 simulator / RViz / Gazebo
+    -> ROS2 simulator
 </code>
-The LLM must still not publish directly to ROS2 topics. The output of the LLM must always pass through:
+The LLM must not publish directly to ROS2 topics.
+The output of the LLM must always pass through:
 <code python>
@@ Line 882: / Line 884: @@
 Only validated commands may be sent to the robot API.
-==== 4.1 Start the simulator ====
+==== 4.1 Start the ROS2 simulator ====
-Open a terminal and source the ROS2 workspace used by the Pupper repository.
+Open a terminal and go to the root folder of the lab:
-Example:
 <code bash>
 cd ~/lab_9_fall_2025
+</code>
+or, if the lab is stored inside the Windows filesystem from WSL:
+<code bash>
+cd /mnt/c/Users/<your_user>/Documents/sumer_school/llm_lab/lab_9_fall_2025
+</code>
+Source ROS2 and the local workspace:
+<code bash>
+source /opt/ros/jazzy/setup.bash
 source install/setup.bash
 </code>
-or, if the workspace is in another folder:
+Start the simulator:
 <code bash>
-source ~/ros2_ws/install/setup.bash
+ros2 launch llm_pupper_sim pupper_sim.launch.py
 </code>
-Then start the simulator using the launch file provided by the repository.
+The simulator should print messages similar to:
-Example structure:
+<code>
+Pupper ROS2 simulator started.
+Listening on /cmd_vel and /pupper/high_level_command.
+Pupper RViz marker node started.
+pose x=0.00, y=0.00, theta=0.00, state=STAND
+</code>
+Leave this terminal open.
+==== 4.2 Check the simulator topics ====
+Open a second terminal.
+Go to the same lab folder and source the environment:
 <code bash>
-ros2 launch <pupper_simulation_package> <simulation_launch_file>.py
+cd ~/lab_9_fall_2025
+source /opt/ros/jazzy/setup.bash
+source install/setup.bash </code>
+List the ROS2 topics:
+<code bash>
+ros2 topic list
 </code>
-Use the exact simulation launch command provided by the instructor or by the repository.
+You should see:
-Depending on the repository, the simulator may use RViz, Gazebo, or both.
+<code>
+/cmd_vel
+/pupper/high_level_command
+/pupper/sim_pose
+/pupper/marker
+</code>
-==== 4.2 Connect KarelPupper to the simulated robot ====
+Manually test forward movement:
-The starter version of `KarelPupper` runs in mock mode and only prints actions.
+<code bash>
+ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}"
+</code>
-For the simulator, you must connect the high-level KarelPupper functions to the simulated robot interface.
+Let it run for a few seconds, then stop it with `Ctrl+C`.
-Open:
+In the simulator terminal, the value of `pose x` should change.
+Stop the simulated robot:
 <code bash>
-nano llm_lab/karel_pupper.py
+ros2 topic pub --once /pupper/high_level_command std_msgs/msg/String "{data: 'STOP'}"
 </code>
-Find the high-level methods:
+Manually test turning:
-<code python>
+<code bash>
-def move_forward(self):
+ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.8}}"
-    ...
+</code>
-def turn_left(self):
+Let it run for a few seconds, then stop it with `Ctrl+C`.
-...
-def turn_right(self):
+In the simulator terminal, the value of `theta` should change.
-...
-def sit(self):
+==== 4.3 Connect KarelPupper to the ROS2 simulator ====
-...
-def stand(self):
+The archive already contains a ROS2 bridge:
-...
-def wave(self):
+<code>
-...
+llm_lab/ros2_pupper_bridge.py
+</code>
-def stop(self):
+and a simulator-aware version of:
-... </code>
-Modify the implementation so that these functions send commands to the simulated Pupper robot using the existing ROS2/Pupper interface.
+<code>
+llm_lab/karel_pupper.py
+</code>
+The connection is:
+<code>
+KarelPupper(mode="sim")
+    -> ros2_pupper_bridge.py
+    -> /cmd_vel
+    -> /pupper/high_level_command
+    -> ROS2 simulator
+</code>
+The command mapping is:
+<code>
+MOVE_FORWARD  -> /cmd_vel linear.x = 0.20
+MOVE_BACKWARD -> /cmd_vel linear.x = -0.20
+TURN_LEFT     -> /cmd_vel angular.z = 0.80
+TURN_RIGHT    -> /cmd_vel angular.z = -0.80
+STOP          -> zero velocity + STOP high-level command
+SIT           -> /pupper/high_level_command
+STAND         -> /pupper/high_level_command
+WAVE          -> /pupper/high_level_command
+</code>
-Do not send raw LLM output directly to ROS2.
+Do not modify the LLM so that it publishes directly to ROS2.
 The correct structure is:
@@ Line 956: / Line 1019: @@
 LLM output
     -> sanitize_commands()
-    -> MOVE_FORWARD / TURN_LEFT / SIT / ...
+    -> allowed command
     -> KarelPupper method
     -> ROS2 simulator command
 </code>
-==== 4.3 Test text commands in simulation ====
+==== 4.4 Test text commands in simulation ====
-First test the simulator with text input.
+Keep the simulator running in the first terminal.
-Use at least the following commands:
+In the second terminal, go to the lab folder:
+<code bash>
+cd ~/lab_9_fall_2025
+source /opt/ros/jazzy/setup.bash
+source install/setup.bash
+cd llm_lab
+source .venv/bin/activate </code>
+If the LLM runs through Ollama on Windows, set:
+<code bash>
+WIN_HOST=$(ip route | awk '/default/ {print $3}')
+export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"
+export OLLAMA_MODEL="llama3.2:3b"
+</code>
+If the LLM runs through Ollama inside WSL, set:
+<code bash>
+export OLLAMA_URL="http://localhost:11434/api/generate"
+export OLLAMA_MODEL="llama3.2:3b"
+</code>
+Run the text-to-simulator pipeline:
+<code bash>
+python run_sim_llm_pipeline.py
+</code>
+Test at least the following commands:
 <code>
@@ Line 974: / Line 1069: @@
 Sit down.
 Stop.
+Run into the wall.
 </code>
@@ Line 979: / Line 1075: @@
     * the input text;
-    * the LLM output;
+    * the raw LLM output;
     * the validated command list;
     * the simulator behavior.
@@ Line 986: / Line 1082: @@
 ^ Input command ^ LLM output ^ Validated commands ^ Simulator behavior ^
-| Stand up | STAND | STAND | Simulated robot stands |
+| Stand up | STAND | STAND | Simulator receives STAND high-level command |
-| Move forward | MOVE_FORWARD | MOVE_FORWARD | Simulated robot moves forward |
+| Move forward | MOVE_FORWARD | MOVE_FORWARD | `pose x` changes in the ROS2 simulator |
-| Turn left | TURN_LEFT | TURN_LEFT | Simulated robot turns left |
+| Turn left | TURN_LEFT | TURN_LEFT | `theta` changes in the ROS2 simulator |
-| Stop | STOP | STOP | Simulated robot stops |
+| Stop | STOP | STOP | Simulator receives STOP |
+| Run into the wall | STOP | STOP | Unsafe command is rejected and converted to STOP |
-==== 4.4 Test voice commands in simulation ====
+==== 4.5 Test voice commands in simulation ====
 After text commands work, test the same pipeline using voice input.
@@ Line 998: / Line 1095: @@
 <code>
-microphone
+microphone or recorded audio file
     -> speech-to-text
     -> transcribed text
@@ Line 1004: / Line 1101: @@
     -> sanitize_commands()
     -> KarelPupper API
-    -> ROS2 simulator / RViz / Gazebo
+    -> ROS2 simulator
+</code>
+Run:
+<code bash>
+python run_voice_sim_pipeline.py
 </code>
@@ Line 1015: / Line 1118: @@
 Turn right and sit.
 Stop.
+Run into the wall.
 </code>
@@ Line 1021: / Line 1125: @@
     * the spoken command;
     * the transcription;
-    * the LLM output;
+    * the raw LLM output;
     * the validated commands;
     * the simulator behavior.
@@ Line 1028: / Line 1132: @@
 ^ Spoken command ^ Transcription ^ LLM output ^ Validated commands ^ Simulator behavior ^
-| Move forward | Move forward | MOVE_FORWARD | MOVE_FORWARD | Simulated robot moves forward |
+| Move forward | Move forward | MOVE_FORWARD | MOVE_FORWARD | `pose x` changes in the ROS2 simulator |
-| Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | Simulated robot turns left, then sits |
+| Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | `theta` changes, then simulator receives SIT |
-| Stop | Stop | STOP | STOP | Simulated robot stops |
+| Stop | Stop | STOP | STOP | Simulator receives STOP |
+| Run into the wall | Run into the wall | STOP | STOP | Unsafe command is rejected and converted to STOP |
-==== 4.5 Safety check before the real robot ====
+If the microphone is not available inside WSL, record a short `.wav` file on Windows and process that file from WSL.
+The pipeline is still valid:
+<code>
+recorded voice command
+    -> .wav file
+    -> speech-to-text
+    -> transcribed text
+    -> real LLM parser
+    -> sanitize_commands()
+    -> ROS2 simulator
+</code>
+==== 4.6 Safety check before the real robot ====
 Before moving to the real robot, verify that:
@@ Line 1040: / Line 1159: @@
     * unsafe commands are converted to `STOP`;
     * the robot does not execute raw text;
-    * the same `KarelPupper` interface can be reused for the real robot.
+    * the LLM does not publish directly to ROS2 topics;
+    * the same `KarelPupper` interface can later be reused for the real robot.
 Only after the simulator pipeline works should you continue to the real Pupper robot.
@@ Line 1130: / Line 1250: @@
 | Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | Robot turns left, then sits |
 | Run into the wall | Run into the wall | STOP | STOP | Robot stops / does not execute unsafe movement |
+===== Common setup problems =====
+==== Problem 1 - `ros2: command not found` ====
+ROS2 is not sourced or not installed.
+Check:
+<code bash>
+ls /opt/ros
+</code>
+For Ubuntu 24.04, source Jazzy:
+<code bash>
+source /opt/ros/jazzy/setup.bash
+</code>
+Then source the local workspace:
+<code bash>
+source install/setup.bash
+</code>
+==== Problem 2 - `externally-managed-environment` when using pip ====
+Ubuntu 24 protects the system Python environment.
+Use a virtual environment:
+<code bash>
+cd llm_lab
+python3 -m venv --system-site-packages .venv
+source .venv/bin/activate
+python -m pip install requests sounddevice scipy faster-whisper
+</code>
+Do not install lab packages globally into the system Python.
+==== Problem 3 - WSL cannot connect to Ollama running on Windows ====
+If Ollama runs on Windows and the pipeline runs in WSL, Ollama must be accessible from WSL.
+In Windows PowerShell:
+<code powershell>
+$env:OLLAMA_HOST="0.0.0.0:11434"
+ollama serve
+</code>
+If WSL still cannot connect, allow inbound TCP traffic on port 11434 in Windows Firewall.
+In WSL:
+<code bash>
+WIN_HOST=$(ip route | awk '/default/ {print $3}')
+curl http://$WIN_HOST:11434/api/tags
+</code>
+Then set:
+<code bash>
+export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"
+</code>
+==== Problem 4 - The simulator does not move ====
+First check that the ROS2 topics exist:
+<code bash>
+ros2 topic list
+</code>
+Then publish a manual velocity command:
+<code bash>
+ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}"
+</code>
+If `pose x` changes in the simulator terminal, the simulator works.
+==== Problem 5 - Microphone does not work in WSL ====
+Some WSL installations do not expose the microphone correctly.
+If microphone recording fails, record a short `.wav` file on Windows and process the audio file from WSL.
+This still satisfies the voice pipeline requirement because the command starts as spoken audio.
 ===== Deliverables =====

Labs

Resources

rasb/lab/09.1782724182.txt.gz · Last modified: 2026/06/29 12:09 by vlad.radulescu2901

Show page Old revisions

Media Manager Back to top