Lab 9: Do What I Say: LLM Control for Pupper

For this lab, use the starter archive provided on OCW:

Download llm_lab.zip

The original repository contains the ROS2/Pupper files. The archive `llm_lab.zip` adds a simplified sandbox for testing the command pipeline before running it on the real robot.

Download `llm_lab.zip` and extract it in the root folder of the repository:

cd ~/lab_9_fall_2025
unzip llm_lab.zip
ls llm_lab

After extraction, the repository should contain:

lab_9_fall_2025/
└── llm_lab/
    ├── commands_config.json
    ├── llm_prompt.txt
    ├── command_parser.py
    ├── karel_pupper.py
    ├── karel_commander.py
    ├── mock_llm.py
    ├── run_karel_test.py
    ├── run_mock_pipeline.py
    ├── README.md
    └── outputs/

The folder `llm_lab` is used first for local testing. After the local pipeline works, students must connect the command parser to a real LLM and then use the validated commands to control the real Pupper robot.

This lab uses a ROS2 simulator before commands are tested on the real Pupper robot.

The recommended environment is:

Ubuntu 24.04 / WSL2
ROS2 Jazzy
Python 3 virtual environment
Ollama or another approved local/instructor-provided LLM

The archive for this lab contains both the LLM sandbox and a minimal ROS2 simulator package:

lab_9_fall_2025/
├── llm_lab/
│   ├── command_parser.py
│   ├── karel_commander.py
│   ├── karel_pupper.py
│   ├── ros2_pupper_bridge.py
│   ├── real_llm.py
│   ├── run_sim_llm_pipeline.py
│   ├── run_voice_sim_pipeline.py
│   └── ...
└── src/
    └── llm_pupper_sim/
        ├── package.xml
        ├── setup.py
        ├── launch/
        │   └── pupper_sim.launch.py
        └── llm_pupper_sim/
            ├── pupper_sim_node.py
            └── pupper_marker_node.py

The ROS2 simulator package subscribes to:

/cmd_vel
/pupper/high_level_command

and publishes:

/pupper/sim_pose
/pupper/marker

The LLM must never publish directly to these ROS2 topics.

The safe structure is:

LLM output
    -> sanitize_commands()
    -> KarelPupper API
    -> ros2_pupper_bridge.py
    -> ROS2 topics
    -> simulator

If ROS2 is not installed, install ROS2 Jazzy first.

Check the Ubuntu version:

lsb_release -a

For Ubuntu 24.04, use ROS2 Jazzy.

Enable the required repositories:

sudo apt update
sudo apt install software-properties-common curl -y
sudo add-apt-repository universe -y
sudo apt update

Add the ROS2 apt source:

export ROS_APT_SOURCE_VERSION=$(curl -s https://api.github.com/repos/ros-infrastructure/ros-apt-source/releases/latest | grep -F "tag_name" | awk -F'"' '{print $4}')
 
curl -L -o /tmp/ros2-apt-source.deb "https://github.com/ros-infrastructure/ros-apt-source/releases/download/${ROS_APT_SOURCE_VERSION}/ros2-apt-source_${ROS_APT_SOURCE_VERSION}.$(. /etc/os-release && echo $VERSION_CODENAME)_all.deb"
 
sudo dpkg -i /tmp/ros2-apt-source.deb
sudo apt update 

Install ROS2 Jazzy and development tools:

sudo apt install ros-jazzy-desktop python3-argcomplete ros-dev-tools -y

Source ROS2:

source /opt/ros/jazzy/setup.bash

Optional: add ROS2 to `.bashrc`:

echo "source /opt/ros/jazzy/setup.bash" >> ~/.bashrc
source ~/.bashrc

Check that ROS2 works:

ros2 --help

Go to the root folder of the lab:

cd ~/lab_9_fall_2025

or, if the lab is stored inside the Windows filesystem from WSL:

cd /mnt/c/Users/<your_user>/Documents/sumer_school/llm_lab/lab_9_fall_2025

Build the simulator package:

source /opt/ros/jazzy/setup.bash
colcon build --packages-select llm_pupper_sim
source install/setup.bash

A successful build should create:

build/
install/
log/

and the local setup file:

install/setup.bash

Ubuntu 24 may block global `pip install` commands because the system Python environment is externally managed.

Use a virtual environment instead.

From the `llm_lab` folder:

cd ~/lab_9_fall_2025/llm_lab
 
python3 -m venv --system-site-packages .venv
source .venv/bin/activate
 
python -m pip install --upgrade pip
python -m pip install requests sounddevice scipy faster-whisper 

The option `–system-site-packages` is important because it allows the virtual environment to see ROS2 Python packages such as `rclpy`.

Check that both ROS2 and the lab dependencies are visible:

python -c "import rclpy; print('rclpy OK')"
python -c "import requests; print('requests OK')"

The lab can use a local LLM through Ollama.

If Ollama runs inside the same Ubuntu / WSL environment, the default URL is:

http://localhost:11434/api/generate

If Ollama runs on Windows and the ROS2 pipeline runs in WSL, expose Ollama from Windows and call it from WSL.

In Windows PowerShell, start Ollama with:

$env:OLLAMA_HOST="0.0.0.0:11434"
ollama serve

If Windows Firewall blocks access from WSL, add an inbound rule for port 11434.

In WSL, find the Windows host IP:

WIN_HOST=$(ip route | awk '/default/ {print $3}')
echo $WIN_HOST

Test access to Ollama:

curl http://$WIN_HOST:11434/api/tags

Then set the LLM URL:

export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"
export OLLAMA_MODEL="llama3.2:3b"

If Ollama runs directly in WSL, use:

export OLLAMA_URL="http://localhost:11434/api/generate"
export OLLAMA_MODEL="llama3.2:3b"

In this lab, you will build a natural language command pipeline for Pupper.

The final pipeline is:

human command
    -> real LLM parser
    -> safety filter
    -> KarelPupper API
    -> real Pupper action

The LLM must not execute Python code directly. It is only allowed to return commands from a fixed list.

Allowed commands:

MOVE_FORWARD
MOVE_BACKWARD
TURN_LEFT
TURN_RIGHT
SIT
STAND
WAVE
STOP

The safety filter checks the LLM output before any robot command is executed.

Go to the lab folder:

cd ~/lab_9_fall_2025/llm_lab

Run:

python3 run_karel_test.py

Expected output:

[MOCK PUPPER] stand
[MOCK PUPPER] move forward
[MOCK PUPPER] turn left
[MOCK PUPPER] wave
[MOCK PUPPER] sit
[MOCK PUPPER] stop

This confirms that the high-level Pupper API works in mock mode.

Open:

nano commands_config.json

This file contains the commands that the robot is allowed to execute.

The safety rule is:

• if a command is in the allowed list, it may be executed;
• if a command is not in the allowed list, it must be rejected.

Open:

nano command_parser.py

Find the function:

sanitize_commands(raw_commands)

This function validates the output generated by the LLM.

The LLM output is never trusted directly. It must always pass through this safety layer before reaching the robot.

Run:

python3 run_mock_pipeline.py

Try commands such as:

Please move forward and then turn left.
Can you say hello?
Go back and stop.
Run into the wall.

The mock pipeline uses a simple local parser. This is only the baseline. It is used to understand the command flow before connecting a real LLM.

Your first task is to replace the mock parser with a real LLM call.

The LLM must receive the user command and return only commands from the allowed list.

The LLM may be:

• a local LLM running on your computer;
• an instructor-provided API;
• another LLM option approved by the instructor.

You are not required to use a paid personal API key.

The important requirement is that the LLM must transform natural language into robot commands.

Open the prompt file:

nano llm_prompt.txt

The prompt must force the model to follow these rules:

• return only commands from the allowed list;
• return one command per line;
• do not explain;
• do not invent new commands;
• return `STOP` for unsafe or impossible requests.

Allowed commands:

MOVE_FORWARD
MOVE_BACKWARD
TURN_LEFT
TURN_RIGHT
SIT
STAND
WAVE
STOP

You must implement or adapt a real LLM call so that a command such as:

Please move forward and then turn left.

returns:

MOVE_FORWARD
TURN_LEFT

You can create a new file for the real LLM parser, for example:

nano real_llm.py

The function should have this structure:

def real_llm_response(user_text):
    """
    Sends user_text to a real LLM and returns the raw model output.
    The model output must contain only commands, one per line.
    """
    # TODO: call a real LLM here
    return raw_llm_output

If you use a local LLM, the function may call a local server such as Ollama.

Example structure:

def real_llm_response(user_text):
    prompt = read_prompt_from_file("llm_prompt.txt")
 
```
# TODO: send prompt + user_text to a local LLM
# Example target: a local LLM server running on your computer
 
return raw_llm_output
```

If you use an instructor-provided API, the function may call that API instead.

After receiving the LLM output, pass it through the safety parser before executing it.

The required pipeline is:

user text
    -> real LLM
    -> raw command output
    -> sanitize_commands()
    -> KarelPupper API

Do not execute code generated by the LLM.

The LLM is allowed to output only text commands. The program decides whether those commands are valid.

A paid API key is not required for this lab.

A free option is to run a local LLM on your computer and call it from Python.

One possible setup is:

student command
    -> local LLM
    -> command text
    -> sanitize_commands()
    -> KarelPupper API

Example local LLM flow:

1. Start a local LLM server.
2. Load a small instruction-following model.
3. Send the contents of llm_prompt.txt plus the user command.
4. Receive the model output.
5. Validate the output with sanitize_commands().

Example Python structure:

import requests
from pathlib import Path
 
ROOT = Path(**file**).resolve().parent
PROMPT_PATH = ROOT / "llm_prompt.txt"
 
def real_llm_response(user_text):
prompt = PROMPT_PATH.read_text(encoding="utf-8")
 
```
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2:3b",
        "prompt": prompt + "\\n\\nUser: " + user_text + "\\nCommands:",
        "stream": False,
        "options": {
            "temperature": 0
        }
    },
    timeout=60
)
 
response.raise_for_status()
return response.json()["response"].strip()
```

This is only an example. The exact local model may be changed by the instructor.

Even when using a local LLM, the output must still pass through:

sanitize_commands()

The local LLM is not allowed to execute code or send commands directly to the robot.

Test at least 10 natural language commands.

Your tests must include:

• simple commands;
• multi-step commands;
• polite commands;
• ambiguous commands;
• unsafe commands.

Use the mock parser first, then the real LLM parser.

Example table:

Write a short conclusion explaining where the real LLM performs better than the simple mock parser.

Before controlling Pupper using voice, you need to transform spoken audio into text.

This step is called speech-to-text or audio transcription.

The idea is simple:

spoken command
    -> audio recording
    -> speech-to-text model
    -> transcribed text

For example:

Audio: "Please move forward and then turn left."

should become:

Please move forward and then turn left.

After that, the transcribed text is sent to the LLM command parser, just like a normal typed command.

The full voice pipeline becomes:

microphone
    -> audio file
    -> speech-to-text
    -> transcribed text
    -> LLM command parser
    -> sanitize_commands()
    -> KarelPupper API

Important: the speech-to-text result is only text. It must not be executed directly. It must still go through the LLM parser and the safety filter.

There are several ways to implement speech-to-text.

You are not required to use a paid transcription API.

The recommended order is:

1. Try a local speech-to-text option.
2. If available, use an instructor-provided API.
3. Use another approved transcription tool if needed.

A local speech-to-text model runs on your own computer.

Advantages:

• does not require a paid API key;
• can work offline after installation;
• is suitable for a lab environment;
• avoids sending audio to an external service.

Disadvantages:

• may be slower on weak laptops;
• may require additional installation;
• transcription quality depends on the model and microphone.

The pipeline is:

microphone
    -> audio recording
    -> local speech-to-text model
    -> transcribed text
    -> LLM command parser

Example structure:

def speech_to_text(audio_path):
    """
    Receives the path to an audio file.
    Returns the transcribed text.
    """
    # TODO: run a local speech-to-text model here
    return transcribed_text

If the instructor provides an API key or a shared transcription service, you may use it.

Advantages:

• easier to integrate;
• usually good transcription quality;
• less local setup.

Disadvantages:

• requires internet access;
• may have usage limits;
• should not use students' personal paid keys unless explicitly allowed.

Example structure:

def speech_to_text(audio_path):
    with open(audio_path, "rb") as audio_file:
        # send audio_file to the instructor-provided transcription API
        # receive transcription
        pass
 
```
return transcription_text
```

Realtime speech-to-text processes microphone input continuously.

This is more advanced and should not be the first implementation.

For this lab, start with short audio recordings of 3-5 seconds. After that works, realtime transcription can be added later.

For this lab, use the simple recording-based approach:

1. Record 3-5 seconds of audio.
2. Save the audio as a .wav file.
3. Transcribe the .wav file using a local model or instructor-provided service.
4. Receive the transcription.
5. Send the transcription to the LLM command parser.
6. Validate the LLM output using sanitize_commands().
7. Execute only validated commands.

The voice command pipeline should look like this:

audio_path = record_audio()
text = speech_to_text(audio_path)
 
raw_llm_output = real_llm_response(text)
commands = sanitize_commands(raw_llm_output.splitlines())
 
execute_commands(robot, commands) 

Using the speech-to-text approach explained above, extend the pipeline so that the user can speak commands instead of typing them.

Start with short recorded commands of 3-5 seconds. Do not begin with realtime transcription.

The required pipeline is:

microphone
    -> audio recording
    -> speech-to-text
    -> transcribed text
    -> real LLM parser
    -> sanitize_commands()
    -> KarelPupper API

Test the voice pipeline in mock mode first.

Example spoken commands:

Stand up.
Move forward.
Turn left.
Turn right.
Sit down.
Say hello.
Stop.
Run into the wall.

For each test, record:

• the spoken command;
• the transcription;
• the LLM output;
• the validated commands;
• the mock robot output.

Example table:

Check that your system can access the microphone.

Try recording a short audio file first, without using the LLM.

Speak clearly and use short commands.

Bad input:

Can you maybe, like, go there and do the thing?

Better input:

Move forward and turn left.

If the transcription is wrong, the LLM may produce the wrong command.

That is why the report must include the transcription, not only the final robot action.

This is normal. The LLM parser should still extract the intended command.

Example:

Please, can you move forward a little bit?

Expected command:

MOVE_FORWARD

If the user says something unsafe, the final validated command should be:

STOP

Example:

Run into the wall.

Expected output:

STOP

Before running commands on the real Pupper robot, test the validated command pipeline in simulation.

The simulator step is important because it checks whether the command generated by the LLM is correctly mapped to a robot action before using the physical robot.

The simulated pipeline is:

voice or text command
    -> speech-to-text, if voice is used
    -> real LLM parser
    -> sanitize_commands()
    -> KarelPupper API
    -> ROS2 simulator

The LLM must not publish directly to ROS2 topics.

The output of the LLM must always pass through:

sanitize_commands()

Only validated commands may be sent to the robot API.

Open a terminal and go to the root folder of the lab:

cd ~/lab_9_fall_2025

or, if the lab is stored inside the Windows filesystem from WSL:

cd /mnt/c/Users/<your_user>/Documents/sumer_school/llm_lab/lab_9_fall_2025

Source ROS2 and the local workspace:

source /opt/ros/jazzy/setup.bash
source install/setup.bash

Start the simulator:

ros2 launch llm_pupper_sim pupper_sim.launch.py

The simulator should print messages similar to:

Pupper ROS2 simulator started.
Listening on /cmd_vel and /pupper/high_level_command.
Pupper RViz marker node started.
pose x=0.00, y=0.00, theta=0.00, state=STAND

Leave this terminal open.

Open a second terminal.

Go to the same lab folder and source the environment:

cd ~/lab_9_fall_2025
 
source /opt/ros/jazzy/setup.bash
source install/setup.bash 

List the ROS2 topics:

ros2 topic list

You should see:

/cmd_vel
/pupper/high_level_command
/pupper/sim_pose
/pupper/marker

Manually test forward movement:

ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}"

Let it run for a few seconds, then stop it with `Ctrl+C`.

In the simulator terminal, the value of `pose x` should change.

Stop the simulated robot:

ros2 topic pub --once /pupper/high_level_command std_msgs/msg/String "{data: 'STOP'}"

Manually test turning:

ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.8}}"

Let it run for a few seconds, then stop it with `Ctrl+C`.

In the simulator terminal, the value of `theta` should change.

The archive already contains a ROS2 bridge:

llm_lab/ros2_pupper_bridge.py

and a simulator-aware version of:

llm_lab/karel_pupper.py

The connection is:

KarelPupper(mode="sim")
    -> ros2_pupper_bridge.py
    -> /cmd_vel
    -> /pupper/high_level_command
    -> ROS2 simulator

The command mapping is:

MOVE_FORWARD  -> /cmd_vel linear.x = 0.20
MOVE_BACKWARD -> /cmd_vel linear.x = -0.20
TURN_LEFT     -> /cmd_vel angular.z = 0.80
TURN_RIGHT    -> /cmd_vel angular.z = -0.80
STOP          -> zero velocity + STOP high-level command
SIT           -> /pupper/high_level_command
STAND         -> /pupper/high_level_command
WAVE          -> /pupper/high_level_command

Do not modify the LLM so that it publishes directly to ROS2.

The correct structure is:

LLM output
    -> sanitize_commands()
    -> allowed command
    -> KarelPupper method
    -> ROS2 simulator command

Keep the simulator running in the first terminal.

In the second terminal, go to the lab folder:

cd ~/lab_9_fall_2025
 
source /opt/ros/jazzy/setup.bash
source install/setup.bash
 
cd llm_lab
source .venv/bin/activate 

If the LLM runs through Ollama on Windows, set:

WIN_HOST=$(ip route | awk '/default/ {print $3}')
export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"
export OLLAMA_MODEL="llama3.2:3b"

If the LLM runs through Ollama inside WSL, set:

export OLLAMA_URL="http://localhost:11434/api/generate"
export OLLAMA_MODEL="llama3.2:3b"

Run the text-to-simulator pipeline:

python run_sim_llm_pipeline.py

Test at least the following commands:

Stand up.
Move forward.
Turn left.
Turn right.
Sit down.
Stop.
Run into the wall.

For each command, record:

• the input text;
• the raw LLM output;
• the validated command list;
• the simulator behavior.

Example table:

After text commands work, test the same pipeline using voice input.

The voice simulator pipeline is:

microphone or recorded audio file
    -> speech-to-text
    -> transcribed text
    -> real LLM parser
    -> sanitize_commands()
    -> KarelPupper API
    -> ROS2 simulator

Run:

python run_voice_sim_pipeline.py

Test at least 5 spoken commands:

Stand up.
Move forward.
Turn left.
Turn right and sit.
Stop.
Run into the wall.

For each spoken command, record:

• the spoken command;
• the transcription;
• the raw LLM output;
• the validated commands;
• the simulator behavior.

Example table:

If the microphone is not available inside WSL, record a short `.wav` file on Windows and process that file from WSL.

The pipeline is still valid:

recorded voice command
    -> .wav file
    -> speech-to-text
    -> transcribed text
    -> real LLM parser
    -> sanitize_commands()
    -> ROS2 simulator

Before moving to the real robot, verify that:

• the simulator receives only validated commands;
• invalid LLM outputs are rejected;
• unsafe commands are converted to `STOP`;
• the robot does not execute raw text;
• the LLM does not publish directly to ROS2 topics;
• the same `KarelPupper` interface can later be reused for the real robot.

Only after the simulator pipeline works should you continue to the real Pupper robot.

After the text pipeline, voice pipeline and simulator pipeline work, connect the validated command pipeline to the real Pupper robot.

This step must be done under instructor supervision.

Before running anything on the real robot:

• test the command in mock mode;
• test the command in the simulator;
• make sure the emergency stop is available;
• place the robot in a safe open area;
• use only high-level KarelPupper commands;
• do not bypass the safety parser.

The real robot pipeline must be:

human text or voice command
    -> speech-to-text, if voice is used
    -> real LLM
    -> sanitize_commands()
    -> KarelPupper command
    -> real Pupper action

The LLM is not allowed to control motors directly. It may only select one or more commands from the allowed command list.

Modify the real robot backend in:

nano llm_lab/karel_pupper.py

The same high-level methods should be used:

robot.stand()
robot.move_forward()
robot.turn_left()
robot.turn_right()
robot.sit()
robot.wave()
robot.stop()

The implementation behind these methods may use ROS2, an existing Pupper API, or another interface provided by the repository.

Demonstrate at least 5 commands on the real Pupper robot using voice input.

Your demonstration must include:

• one single-step command;
• one two-step command;
• one command that makes the robot stop;
• one command that is rejected or converted to `STOP`;
• one command chosen by you.

Example spoken commands:

Stand up.
Move forward.
Turn left and sit.
Say hello.
Stop.
Run into the wall.

For each demonstrated command, record:

• the spoken command;
• the transcription;
• the LLM output;
• the validated command list;
• the observed robot behavior.

Example table:

ROS2 is not sourced or not installed.

Check:

ls /opt/ros

For Ubuntu 24.04, source Jazzy:

source /opt/ros/jazzy/setup.bash

Then source the local workspace:

source install/setup.bash

Ubuntu 24 protects the system Python environment.

Use a virtual environment:

cd llm_lab
python3 -m venv --system-site-packages .venv
source .venv/bin/activate
python -m pip install requests sounddevice scipy faster-whisper

Do not install lab packages globally into the system Python.

If Ollama runs on Windows and the pipeline runs in WSL, Ollama must be accessible from WSL.

In Windows PowerShell:

$env:OLLAMA_HOST="0.0.0.0:11434"
ollama serve

If WSL still cannot connect, allow inbound TCP traffic on port 11434 in Windows Firewall.

In WSL:

WIN_HOST=$(ip route | awk '/default/ {print $3}')
curl http://$WIN_HOST:11434/api/tags

Then set:

export OLLAMA_URL="http://$WIN_HOST:11434/api/generate"

First check that the ROS2 topics exist:

ros2 topic list

Then publish a manual velocity command:

ros2 topic pub --rate 10 /cmd_vel geometry_msgs/msg/Twist "{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}"

If `pose x` changes in the simulator terminal, the simulator works.

Some WSL installations do not expose the microphone correctly.

If microphone recording fails, record a short `.wav` file on Windows and process the audio file from WSL.

This still satisfies the voice pipeline requirement because the command starts as spoken audio.

Submit:

• the final LLM prompt;
• the code or function used to call the real LLM;
• a short note specifying which LLM option you used: local LLM, instructor-provided API, or another approved option;
• the code or function used for speech-to-text;
• a short note specifying which speech-to-text option you used: local model, instructor-provided API, or another approved option;
• the code section that connects validated commands to the simulator;
• the code section that connects validated commands to the real Pupper robot;
• a comparison table between the mock parser and the real LLM parser;
• a table with at least 5 voice commands, transcriptions, LLM outputs and validated commands;
• proof that the command pipeline was tested in the ROS2 simulator;
• proof of running at least 5 voice commands on the real Pupper robot;
• a short explanation of the safety filter;
• a short explanation of why simulation is used before the real robot;
• a short explanation of why the robot should not execute raw LLM output directly.

A language model should not be allowed to control a robot directly.

The safe design is:

voice or text input
    -> transcription, if voice is used
    -> LLM output
    -> validation
    -> allowed command
    -> high-level robot API

The LLM can interpret natural language, but the program must decide what is safe to execute.