Show page

Differences

This shows you the differences between two versions of the page.

--- rasb:lab:09 [2026/06/29 12:11]
vlad.radulescu2901 [4.2 Connect KarelPupper to the simulated robot]
+++ rasb:lab:09 [2026/06/29 12:12] (current)
vlad.radulescu2901 [Exercise 6 - Demonstrate the full voice-to-Pupper pipeline]
@@ Line 861: / Line 861: @@
 Before running commands on the real Pupper robot, test the validated command pipeline in simulation.
-The simulator step is important because it lets you check whether the command generated by the LLM is correctly mapped to a robot action before using the physical robot.
+The simulator step is important because it checks whether the command generated by the LLM is correctly mapped to a robot action before using the physical robot.
 The simulated pipeline is:
@@ Line 871: / Line 871: @@
     -> sanitize_commands()
     -> KarelPupper API
-    -> ROS2 simulator / RViz / Gazebo
+    -> ROS2 simulator
 </code>
-The LLM must still not publish directly to ROS2 topics. The output of the LLM must always pass through:
+The LLM must not publish directly to ROS2 topics.
+The output of the LLM must always pass through:
 <code python>
@@ Line 882: / Line 884: @@
 Only validated commands may be sent to the robot API.
-==== 4.1 Start the simulator ====
+==== 4.1 Start the ROS2 simulator ====
-Open a terminal and source the ROS2 workspace used by the Pupper repository.
+Open a terminal and go to the root folder of the lab:
-Example:
+<code bash>
+cd ~/lab_9_fall_2025
+</code>
+or, if the lab is stored inside the Windows filesystem from WSL:
+<code bash>
+cd /mnt/c/Users/<your_user>/Documents/sumer_school/llm_lab/lab_9_fall_2025
+</code>
+Source ROS2 and the local workspace:
+<code bash>
+source /opt/ros/jazzy/setup.bash
+source install/setup.bash
+</code>
+Start the simulator:
+<code bash>
+ros2 launch llm_pupper_sim pupper_sim.launch.py
+</code>
+The simulator should print messages similar to:
+<code>
+Pupper ROS2 simulator started.
+Listening on /cmd_vel and /pupper/high_level_command.
+Pupper RViz marker node started.
+pose x=0.00, y=0.00, theta=0.00, state=STAND
+</code>
+Leave this terminal open.
+==== 4.2 Check the simulator topics ====
+Open a second terminal.
+Go to the same lab folder and source the environment:
 <code bash>
 cd ~/lab_9_fall_2025
+source /opt/ros/jazzy/setup.bash
+source install/setup.bash </code>
+List the ROS2 topics:
+<code bash>
+ros2 topic list
+</code>
+You should see:
+<code>
+/cmd_vel
+/pupper/high_level_command
+/pupper/sim_pose
+/pupper/marker
+</code>
+Manually test forward movement:
+<code bash>
+ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}"
+</code>
+Let it run for a few seconds, then stop it with `Ctrl+C`.
+In the simulator terminal, the value of `pose x` should change.
+Stop the simulated robot:
+<code bash>
+ros2 topic pub --once /pupper/high_level_command std_msgs/msg/String "{data: 'STOP'}"
+</code>
+Manually test turning:
+<code bash>
+ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.8}}"
+</code>
+Let it run for a few seconds, then stop it with `Ctrl+C`.
+In the simulator terminal, the value of `theta` should change.
+==== 4.3 Connect KarelPupper to the ROS2 simulator ====
+The archive already contains a ROS2 bridge:
+<code>
+llm_lab/ros2_pupper_bridge.py
+</code>
+and a simulator-aware version of:
+<code>
+llm_lab/karel_pupper.py
+</code>
+The connection is:
+<code>
+KarelPupper(mode="sim")
+    -> ros2_pupper_bridge.py
+    -> /cmd_vel
+    -> /pupper/high_level_command
+    -> ROS2 simulator
+</code>
+The command mapping is:
+<code>
+MOVE_FORWARD  -> /cmd_vel linear.x = 0.20
+MOVE_BACKWARD -> /cmd_vel linear.x = -0.20
+TURN_LEFT     -> /cmd_vel angular.z = 0.80
+TURN_RIGHT    -> /cmd_vel angular.z = -0.80
+STOP          -> zero velocity + STOP high-level command
+SIT           -> /pupper/high_level_command
+STAND         -> /pupper/high_level_command
+WAVE          -> /pupper/high_level_command
+</code>
+Do not modify the LLM so that it publishes directly to ROS2.
+The correct structure is:
+<code>
+LLM output
+    -> sanitize_commands()
+    -> allowed command
+    -> KarelPupper method
+    -> ROS2 simulator command
+</code>
+==== 4.4 Test text commands in simulation ====
+Keep the simulator running in the first terminal.
+In the second terminal, go to the lab folder:
+<code bash>
+cd ~/lab_9_fall_2025
+source /opt/ros/jazzy/setup.bash
 source install/setup.bash
+cd llm_lab
+source .venv/bin/activate </code>
+If the LLM runs through Ollama on Windows, set:
+<code bash>
+WIN_HOST=$(ip route | awk '/default/ {print $3}')
+export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"
+export OLLAMA_MODEL="llama3.2:3b"
 </code>
-or, if the workspace is in another folder:
+If the LLM runs through Ollama inside WSL, set:
 <code bash>
-source ~/ros2_ws/install/setup.bash
+export OLLAMA_URL="http://localhost:11434/api/generate"
+export OLLAMA_MODEL="llama3.2:3b"
 </code>
-Then start the simulator using the launch file provided by the repository.
+Run the text-to-simulator pipeline:
-Example structure:
+<code bash>
+python run_sim_llm_pipeline.py
+</code>
+Test at least the following commands:
+<code>
+Stand up.
+Move forward.
+Turn left.
+Turn right.
+Sit down.
+Stop.
+Run into the wall.
+</code>
+For each command, record:
+    * the input text;
+    * the raw LLM output;
+    * the validated command list;
+    * the simulator behavior.
+Example table:
+^ Input command ^ LLM output ^ Validated commands ^ Simulator behavior ^
+| Stand up | STAND | STAND | Simulator receives STAND high-level command |
+| Move forward | MOVE_FORWARD | MOVE_FORWARD | `pose x` changes in the ROS2 simulator |
+| Turn left | TURN_LEFT | TURN_LEFT | `theta` changes in the ROS2 simulator |
+| Stop | STOP | STOP | Simulator receives STOP |
+| Run into the wall | STOP | STOP | Unsafe command is rejected and converted to STOP |
+==== 4.5 Test voice commands in simulation ====
+After text commands work, test the same pipeline using voice input.
+The voice simulator pipeline is:
+<code>
+microphone or recorded audio file
+    -> speech-to-text
+    -> transcribed text
+    -> real LLM parser
+    -> sanitize_commands()
+    -> KarelPupper API
+    -> ROS2 simulator
+</code>
+Run:
 <code bash>
-ros2 launch <pupper_simulation_package> <simulation_launch_file>.py
+python run_voice_sim_pipeline.py
 </code>
-Use the exact simulation launch command provided by the instructor or by the repository.
+Test at least 5 spoken commands:
+<code>
+Stand up.
+Move forward.
+Turn left.
+Turn right and sit.
+Stop.
+Run into the wall.
+</code>
+For each spoken command, record:
+    * the spoken command;
+    * the transcription;
+    * the raw LLM output;
+    * the validated commands;
+    * the simulator behavior.
+Example table:
+^ Spoken command ^ Transcription ^ LLM output ^ Validated commands ^ Simulator behavior ^
+| Move forward | Move forward | MOVE_FORWARD | MOVE_FORWARD | `pose x` changes in the ROS2 simulator |
+| Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | `theta` changes, then simulator receives SIT |
+| Stop | Stop | STOP | STOP | Simulator receives STOP |
+| Run into the wall | Run into the wall | STOP | STOP | Unsafe command is rejected and converted to STOP |
+If the microphone is not available inside WSL, record a short `.wav` file on Windows and process that file from WSL.
+The pipeline is still valid:
+<code>
+recorded voice command
+    -> .wav file
+    -> speech-to-text
+    -> transcribed text
+    -> real LLM parser
+    -> sanitize_commands()
+    -> ROS2 simulator
+</code>
+==== 4.6 Safety check before the real robot ====
+Before moving to the real robot, verify that:
-Depending on the repository, the simulator may use RViz, Gazebo, or both.
+    * the simulator receives only validated commands;
+    * invalid LLM outputs are rejected;
+    * unsafe commands are converted to `STOP`;
+    * the robot does not execute raw text;
+    * the LLM does not publish directly to ROS2 topics;
+    * the same `KarelPupper` interface can later be reused for the real robot.
+Only after the simulator pipeline works should you continue to the real Pupper robot.
 ===== Exercise 5 - Connect the validated commands to the real Pupper robot =====
@@ Line 998: / Line 1250: @@
 | Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | Robot turns left, then sits |
 | Run into the wall | Run into the wall | STOP | STOP | Robot stops / does not execute unsafe movement |
+===== Common setup problems =====
+==== Problem 1 - `ros2: command not found` ====
+ROS2 is not sourced or not installed.
+Check:
+<code bash>
+ls /opt/ros
+</code>
+For Ubuntu 24.04, source Jazzy:
+<code bash>
+source /opt/ros/jazzy/setup.bash
+</code>
+Then source the local workspace:
+<code bash>
+source install/setup.bash
+</code>
+==== Problem 2 - `externally-managed-environment` when using pip ====
+Ubuntu 24 protects the system Python environment.
+Use a virtual environment:
+<code bash>
+cd llm_lab
+python3 -m venv --system-site-packages .venv
+source .venv/bin/activate
+python -m pip install requests sounddevice scipy faster-whisper
+</code>
+Do not install lab packages globally into the system Python.
+==== Problem 3 - WSL cannot connect to Ollama running on Windows ====
+If Ollama runs on Windows and the pipeline runs in WSL, Ollama must be accessible from WSL.
+In Windows PowerShell:
+<code powershell>
+$env:OLLAMA_HOST="0.0.0.0:11434"
+ollama serve
+</code>
+If WSL still cannot connect, allow inbound TCP traffic on port 11434 in Windows Firewall.
+In WSL:
+<code bash>
+WIN_HOST=$(ip route | awk '/default/ {print $3}')
+curl http://$WIN_HOST:11434/api/tags
+</code>
+Then set:
+<code bash>
+export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"
+</code>
+==== Problem 4 - The simulator does not move ====
+First check that the ROS2 topics exist:
+<code bash>
+ros2 topic list
+</code>
+Then publish a manual velocity command:
+<code bash>
+ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}"
+</code>
+If `pose x` changes in the simulator terminal, the simulator works.
+==== Problem 5 - Microphone does not work in WSL ====
+Some WSL installations do not expose the microphone correctly.
+If microphone recording fails, record a short `.wav` file on Windows and process the audio file from WSL.
+This still satisfies the voice pipeline requirement because the command starts as spoken audio.
 ===== Deliverables =====

Labs

Resources

rasb/lab/09.1782724301.txt.gz · Last modified: 2026/06/29 12:11 by vlad.radulescu2901

Show page Old revisions

Media Manager Back to top