Differences
This shows you the differences between two versions of the page.
|
rasb:lab:09 [2026/06/29 12:11] vlad.radulescu2901 [4.3 Test text commands in simulation] |
rasb:lab:09 [2026/06/29 12:12] (current) vlad.radulescu2901 [Exercise 6 - Demonstrate the full voice-to-Pupper pipeline] |
||
|---|---|---|---|
| Line 861: | Line 861: | ||
| Before running commands on the real Pupper robot, test the validated command pipeline in simulation. | Before running commands on the real Pupper robot, test the validated command pipeline in simulation. | ||
| - | The simulator step is important because it lets you check whether the command generated by the LLM is correctly mapped to a robot action before using the physical robot. | + | The simulator step is important because it checks whether the command generated by the LLM is correctly mapped to a robot action before using the physical robot. |
| The simulated pipeline is: | The simulated pipeline is: | ||
| Line 871: | Line 871: | ||
| -> sanitize_commands() | -> sanitize_commands() | ||
| -> KarelPupper API | -> KarelPupper API | ||
| - | -> ROS2 simulator / RViz / Gazebo | + | -> ROS2 simulator |
| </code> | </code> | ||
| - | The LLM must still not publish directly to ROS2 topics. The output of the LLM must always pass through: | + | The LLM must not publish directly to ROS2 topics. |
| + | |||
| + | The output of the LLM must always pass through: | ||
| <code python> | <code python> | ||
| Line 882: | Line 884: | ||
| Only validated commands may be sent to the robot API. | Only validated commands may be sent to the robot API. | ||
| - | ==== 4.1 Start the simulator ==== | + | ==== 4.1 Start the ROS2 simulator ==== |
| - | Open a terminal and source the ROS2 workspace used by the Pupper repository. | + | Open a terminal and go to the root folder of the lab: |
| - | + | ||
| - | Example: | + | |
| <code bash> | <code bash> | ||
| cd ~/lab_9_fall_2025 | cd ~/lab_9_fall_2025 | ||
| + | </code> | ||
| + | |||
| + | or, if the lab is stored inside the Windows filesystem from WSL: | ||
| + | |||
| + | <code bash> | ||
| + | cd /mnt/c/Users/<your_user>/Documents/sumer_school/llm_lab/lab_9_fall_2025 | ||
| + | </code> | ||
| + | |||
| + | Source ROS2 and the local workspace: | ||
| + | |||
| + | <code bash> | ||
| + | source /opt/ros/jazzy/setup.bash | ||
| source install/setup.bash | source install/setup.bash | ||
| </code> | </code> | ||
| - | or, if the workspace is in another folder: | + | Start the simulator: |
| <code bash> | <code bash> | ||
| - | source ~/ros2_ws/install/setup.bash | + | ros2 launch llm_pupper_sim pupper_sim.launch.py |
| </code> | </code> | ||
| - | Then start the simulator using the launch file provided by the repository. | + | The simulator should print messages similar to: |
| - | Example structure: | + | <code> |
| + | Pupper ROS2 simulator started. | ||
| + | Listening on /cmd_vel and /pupper/high_level_command. | ||
| + | Pupper RViz marker node started. | ||
| + | pose x=0.00, y=0.00, theta=0.00, state=STAND | ||
| + | </code> | ||
| + | |||
| + | Leave this terminal open. | ||
| + | |||
| + | ==== 4.2 Check the simulator topics ==== | ||
| + | |||
| + | Open a second terminal. | ||
| + | |||
| + | Go to the same lab folder and source the environment: | ||
| <code bash> | <code bash> | ||
| - | ros2 launch <pupper_simulation_package> <simulation_launch_file>.py | + | cd ~/lab_9_fall_2025 |
| + | |||
| + | source /opt/ros/jazzy/setup.bash | ||
| + | source install/setup.bash </code> | ||
| + | |||
| + | List the ROS2 topics: | ||
| + | |||
| + | <code bash> | ||
| + | ros2 topic list | ||
| </code> | </code> | ||
| - | Use the exact simulation launch command provided by the instructor or by the repository. | + | You should see: |
| - | Depending on the repository, the simulator may use RViz, Gazebo, or both. | + | <code> |
| + | /cmd_vel | ||
| + | /pupper/high_level_command | ||
| + | /pupper/sim_pose | ||
| + | /pupper/marker | ||
| + | </code> | ||
| - | ==== 4.2 Connect KarelPupper to the simulated robot ==== | + | Manually test forward movement: |
| - | The starter version of `KarelPupper` runs in mock mode and only prints actions. | + | <code bash> |
| + | ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}" | ||
| + | </code> | ||
| - | For the simulator, you must connect the high-level KarelPupper functions to the simulated robot interface. | + | Let it run for a few seconds, then stop it with `Ctrl+C`. |
| - | Open: | + | In the simulator terminal, the value of `pose x` should change. |
| + | |||
| + | Stop the simulated robot: | ||
| <code bash> | <code bash> | ||
| - | nano llm_lab/karel_pupper.py | + | ros2 topic pub --once /pupper/high_level_command std_msgs/msg/String "{data: 'STOP'}" |
| </code> | </code> | ||
| - | Find the high-level methods: | + | Manually test turning: |
| - | <code python> | + | <code bash> |
| - | def move_forward(self): | + | ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.8}}" |
| - | ... | + | </code> |
| - | def turn_left(self): | + | Let it run for a few seconds, then stop it with `Ctrl+C`. |
| - | ... | + | |
| - | def turn_right(self): | + | In the simulator terminal, the value of `theta` should change. |
| - | ... | + | |
| - | def sit(self): | + | ==== 4.3 Connect KarelPupper to the ROS2 simulator ==== |
| - | ... | + | |
| - | def stand(self): | + | The archive already contains a ROS2 bridge: |
| - | ... | + | |
| - | def wave(self): | + | <code> |
| - | ... | + | llm_lab/ros2_pupper_bridge.py |
| + | </code> | ||
| - | def stop(self): | + | and a simulator-aware version of: |
| - | ... </code> | + | |
| - | Modify the implementation so that these functions send commands to the simulated Pupper robot using the existing ROS2/Pupper interface. | + | <code> |
| + | llm_lab/karel_pupper.py | ||
| + | </code> | ||
| + | |||
| + | The connection is: | ||
| + | |||
| + | <code> | ||
| + | KarelPupper(mode="sim") | ||
| + | -> ros2_pupper_bridge.py | ||
| + | -> /cmd_vel | ||
| + | -> /pupper/high_level_command | ||
| + | -> ROS2 simulator | ||
| + | </code> | ||
| + | |||
| + | The command mapping is: | ||
| + | |||
| + | <code> | ||
| + | MOVE_FORWARD -> /cmd_vel linear.x = 0.20 | ||
| + | MOVE_BACKWARD -> /cmd_vel linear.x = -0.20 | ||
| + | TURN_LEFT -> /cmd_vel angular.z = 0.80 | ||
| + | TURN_RIGHT -> /cmd_vel angular.z = -0.80 | ||
| + | STOP -> zero velocity + STOP high-level command | ||
| + | SIT -> /pupper/high_level_command | ||
| + | STAND -> /pupper/high_level_command | ||
| + | WAVE -> /pupper/high_level_command | ||
| + | </code> | ||
| - | Do not send raw LLM output directly to ROS2. | + | Do not modify the LLM so that it publishes directly to ROS2. |
| The correct structure is: | The correct structure is: | ||
| Line 956: | Line 1019: | ||
| LLM output | LLM output | ||
| -> sanitize_commands() | -> sanitize_commands() | ||
| - | -> MOVE_FORWARD / TURN_LEFT / SIT / ... | + | -> allowed command |
| -> KarelPupper method | -> KarelPupper method | ||
| -> ROS2 simulator command | -> ROS2 simulator command | ||
| </code> | </code> | ||
| + | ==== 4.4 Test text commands in simulation ==== | ||
| + | |||
| + | Keep the simulator running in the first terminal. | ||
| + | |||
| + | In the second terminal, go to the lab folder: | ||
| + | |||
| + | <code bash> | ||
| + | cd ~/lab_9_fall_2025 | ||
| + | |||
| + | source /opt/ros/jazzy/setup.bash | ||
| + | source install/setup.bash | ||
| + | |||
| + | cd llm_lab | ||
| + | source .venv/bin/activate </code> | ||
| + | |||
| + | If the LLM runs through Ollama on Windows, set: | ||
| + | |||
| + | <code bash> | ||
| + | WIN_HOST=$(ip route | awk '/default/ {print $3}') | ||
| + | export OLLAMA_URL="http://$WIN_HOST:11434/api/generate" | ||
| + | export OLLAMA_MODEL="llama3.2:3b" | ||
| + | </code> | ||
| + | |||
| + | If the LLM runs through Ollama inside WSL, set: | ||
| + | |||
| + | <code bash> | ||
| + | export OLLAMA_URL="http://localhost:11434/api/generate" | ||
| + | export OLLAMA_MODEL="llama3.2:3b" | ||
| + | </code> | ||
| + | |||
| + | Run the text-to-simulator pipeline: | ||
| + | |||
| + | <code bash> | ||
| + | python run_sim_llm_pipeline.py | ||
| + | </code> | ||
| + | |||
| + | Test at least the following commands: | ||
| + | |||
| + | <code> | ||
| + | Stand up. | ||
| + | Move forward. | ||
| + | Turn left. | ||
| + | Turn right. | ||
| + | Sit down. | ||
| + | Stop. | ||
| + | Run into the wall. | ||
| + | </code> | ||
| + | |||
| + | For each command, record: | ||
| + | |||
| + | * the input text; | ||
| + | * the raw LLM output; | ||
| + | * the validated command list; | ||
| + | * the simulator behavior. | ||
| + | |||
| + | Example table: | ||
| + | |||
| + | ^ Input command ^ LLM output ^ Validated commands ^ Simulator behavior ^ | ||
| + | | Stand up | STAND | STAND | Simulator receives STAND high-level command | | ||
| + | | Move forward | MOVE_FORWARD | MOVE_FORWARD | `pose x` changes in the ROS2 simulator | | ||
| + | | Turn left | TURN_LEFT | TURN_LEFT | `theta` changes in the ROS2 simulator | | ||
| + | | Stop | STOP | STOP | Simulator receives STOP | | ||
| + | | Run into the wall | STOP | STOP | Unsafe command is rejected and converted to STOP | | ||
| + | |||
| + | ==== 4.5 Test voice commands in simulation ==== | ||
| + | |||
| + | After text commands work, test the same pipeline using voice input. | ||
| + | |||
| + | The voice simulator pipeline is: | ||
| + | |||
| + | <code> | ||
| + | microphone or recorded audio file | ||
| + | -> speech-to-text | ||
| + | -> transcribed text | ||
| + | -> real LLM parser | ||
| + | -> sanitize_commands() | ||
| + | -> KarelPupper API | ||
| + | -> ROS2 simulator | ||
| + | </code> | ||
| + | |||
| + | Run: | ||
| + | |||
| + | <code bash> | ||
| + | python run_voice_sim_pipeline.py | ||
| + | </code> | ||
| + | |||
| + | Test at least 5 spoken commands: | ||
| + | |||
| + | <code> | ||
| + | Stand up. | ||
| + | Move forward. | ||
| + | Turn left. | ||
| + | Turn right and sit. | ||
| + | Stop. | ||
| + | Run into the wall. | ||
| + | </code> | ||
| + | |||
| + | For each spoken command, record: | ||
| + | |||
| + | * the spoken command; | ||
| + | * the transcription; | ||
| + | * the raw LLM output; | ||
| + | * the validated commands; | ||
| + | * the simulator behavior. | ||
| + | |||
| + | Example table: | ||
| + | |||
| + | ^ Spoken command ^ Transcription ^ LLM output ^ Validated commands ^ Simulator behavior ^ | ||
| + | | Move forward | Move forward | MOVE_FORWARD | MOVE_FORWARD | `pose x` changes in the ROS2 simulator | | ||
| + | | Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | `theta` changes, then simulator receives SIT | | ||
| + | | Stop | Stop | STOP | STOP | Simulator receives STOP | | ||
| + | | Run into the wall | Run into the wall | STOP | STOP | Unsafe command is rejected and converted to STOP | | ||
| + | |||
| + | If the microphone is not available inside WSL, record a short `.wav` file on Windows and process that file from WSL. | ||
| + | |||
| + | The pipeline is still valid: | ||
| + | |||
| + | <code> | ||
| + | recorded voice command | ||
| + | -> .wav file | ||
| + | -> speech-to-text | ||
| + | -> transcribed text | ||
| + | -> real LLM parser | ||
| + | -> sanitize_commands() | ||
| + | -> ROS2 simulator | ||
| + | </code> | ||
| + | |||
| + | ==== 4.6 Safety check before the real robot ==== | ||
| + | |||
| + | Before moving to the real robot, verify that: | ||
| + | |||
| + | * the simulator receives only validated commands; | ||
| + | * invalid LLM outputs are rejected; | ||
| + | * unsafe commands are converted to `STOP`; | ||
| + | * the robot does not execute raw text; | ||
| + | * the LLM does not publish directly to ROS2 topics; | ||
| + | * the same `KarelPupper` interface can later be reused for the real robot. | ||
| + | |||
| + | Only after the simulator pipeline works should you continue to the real Pupper robot. | ||
| ===== Exercise 5 - Connect the validated commands to the real Pupper robot ===== | ===== Exercise 5 - Connect the validated commands to the real Pupper robot ===== | ||
| Line 1048: | Line 1250: | ||
| | Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | Robot turns left, then sits | | | Turn left and sit | Turn left and sit | TURN_LEFT, SIT | TURN_LEFT, SIT | Robot turns left, then sits | | ||
| | Run into the wall | Run into the wall | STOP | STOP | Robot stops / does not execute unsafe movement | | | Run into the wall | Run into the wall | STOP | STOP | Robot stops / does not execute unsafe movement | | ||
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
| + | |||
| + | ===== Common setup problems ===== | ||
| + | |||
| + | ==== Problem 1 - `ros2: command not found` ==== | ||
| + | |||
| + | ROS2 is not sourced or not installed. | ||
| + | |||
| + | Check: | ||
| + | |||
| + | <code bash> | ||
| + | ls /opt/ros | ||
| + | </code> | ||
| + | |||
| + | For Ubuntu 24.04, source Jazzy: | ||
| + | |||
| + | <code bash> | ||
| + | source /opt/ros/jazzy/setup.bash | ||
| + | </code> | ||
| + | |||
| + | Then source the local workspace: | ||
| + | |||
| + | <code bash> | ||
| + | source install/setup.bash | ||
| + | </code> | ||
| + | |||
| + | ==== Problem 2 - `externally-managed-environment` when using pip ==== | ||
| + | |||
| + | Ubuntu 24 protects the system Python environment. | ||
| + | |||
| + | Use a virtual environment: | ||
| + | |||
| + | <code bash> | ||
| + | cd llm_lab | ||
| + | python3 -m venv --system-site-packages .venv | ||
| + | source .venv/bin/activate | ||
| + | python -m pip install requests sounddevice scipy faster-whisper | ||
| + | </code> | ||
| + | |||
| + | Do not install lab packages globally into the system Python. | ||
| + | |||
| + | ==== Problem 3 - WSL cannot connect to Ollama running on Windows ==== | ||
| + | |||
| + | If Ollama runs on Windows and the pipeline runs in WSL, Ollama must be accessible from WSL. | ||
| + | |||
| + | In Windows PowerShell: | ||
| + | |||
| + | <code powershell> | ||
| + | $env:OLLAMA_HOST="0.0.0.0:11434" | ||
| + | ollama serve | ||
| + | </code> | ||
| + | |||
| + | If WSL still cannot connect, allow inbound TCP traffic on port 11434 in Windows Firewall. | ||
| + | |||
| + | In WSL: | ||
| + | |||
| + | <code bash> | ||
| + | WIN_HOST=$(ip route | awk '/default/ {print $3}') | ||
| + | curl http://$WIN_HOST:11434/api/tags | ||
| + | </code> | ||
| + | |||
| + | Then set: | ||
| + | |||
| + | <code bash> | ||
| + | export OLLAMA_URL="http://$WIN_HOST:11434/api/generate" | ||
| + | </code> | ||
| + | |||
| + | ==== Problem 4 - The simulator does not move ==== | ||
| + | |||
| + | First check that the ROS2 topics exist: | ||
| + | |||
| + | <code bash> | ||
| + | ros2 topic list | ||
| + | </code> | ||
| + | |||
| + | Then publish a manual velocity command: | ||
| + | |||
| + | <code bash> | ||
| + | ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}" | ||
| + | </code> | ||
| + | |||
| + | If `pose x` changes in the simulator terminal, the simulator works. | ||
| + | |||
| + | ==== Problem 5 - Microphone does not work in WSL ==== | ||
| + | |||
| + | Some WSL installations do not expose the microphone correctly. | ||
| + | |||
| + | If microphone recording fails, record a short `.wav` file on Windows and process the audio file from WSL. | ||
| + | |||
| + | This still satisfies the voice pipeline requirement because the command starts as spoken audio. | ||
| + | |||
| ===== Deliverables ===== | ===== Deliverables ===== | ||
