In this lab, Pupper finally gets to use its eyes.

You will build a complete perception-to-action pipeline: the camera image goes through an object detector, the detector output feeds a PD controller, and the PD controller makes Pupper walk towards a ball. After that works, you will put the LLM from Lab 9 back in the loop so that Pupper decides what to follow based on natural language, and finally you will use a vision-language model to follow objects that no classic detector knows about.

The full pipeline you will build:

camera image
    -> object detector (YOLOv11)
    -> bounding box of the target
    -> PD controller (visual servoing)
    -> velocity command (/cmd_vel)
    -> Pupper walks towards the target

The key idea from the Stanford lecture: if we know our location relative to an object, we do not need to know where we are, or where the object is, in absolute terms. A bounding box in the image is enough to walk to the object — no map, no localization, no GPS.

Object detection means answering two questions at once: what is in the image and where is it. The detector returns a list of boxes, each with a class label and a confidence score. Today, two families of architectures dominate.

YOLO is a single-shot detector: one forward pass of a convolutional network produces all the boxes at the same time.

The classic YOLO formulation divides the image into a grid (7×7 in the original paper). Each grid cell predicts:

• a few candidate bounding boxes, each described by center coordinates (cx, cy), size (w, h) and a confidence score;
• the class probabilities for the object centered in that cell.

In the original paper this meant 30 output values per cell: 2 candidate boxes x (cx, cy, w, h, conf) + 20 class scores. Localization and classification happen simultaneously — this is why YOLO is fast enough for robots and why it “only looks once”.

Because several cells may fire on the same object, YOLO needs a post-processing step called non-maximum suppression (NMS): keep the highest-confidence box, drop all boxes that overlap it too much, repeat.

Modern versions (we will use YOLOv11 from Ultralytics) are anchor-free, NMS is built into the pipeline, and the models come in sizes from n (nano, ~2.6M parameters) to x (extra large). They are trained on the COCO dataset: 80 everyday classes such as person, dog, chair and — conveniently for us — sports ball.

DETR treats detection as a set prediction problem using a transformer:

• a CNN backbone extracts image features;
• a transformer encoder-decoder processes them;
• a fixed set of learned object queries (e.g. 100) each “asks” the image about one potential object;
• each query outputs a box + class directly.

During training, a bipartite (Hungarian) matching assigns each predicted box to at most one ground-truth object, so DETR needs no NMS and no anchors — the architecture is conceptually much cleaner. The price is slower convergence in training and, for the original DETR, slower inference. Newer variants such as RT-DETR close the speed gap.

On a robot with a Raspberry Pi CPU, the practical answer today is YOLO, and the nano variant. Keep in mind the fundamental limitation of both: they only know the classes they were trained on. A COCO-trained model cannot detect “a fire extinguisher” or “the blue whiteboard marker”. We return to this problem in Task 4.

A note on hardware. Pupper's head also contains a Hailo-8L neural accelerator. The production stack already runs a YOLO model on it: the hailo_detection_node subscribes to the camera and publishes /detections at camera rate, which is what the built-in person-following demo uses. In this lab you will run YOLOv11 on the CPU to own the whole pipeline yourself — measured on the robot, yolo11n at 320 px input takes about 230 ms/frame (~4 FPS), which is enough for a walking robot. Once your pipeline works, comparing your node with the Hailo one is a nice exercise.

SSH into your Pupper. The ROS 2 stack runs as a systemd service (sudo systemctl status robot). The topics relevant for this lab:

/camera/image_raw/compressed      sensor_msgs/CompressedImage   camera feed, 1400x1050
/detections                       vision_msgs/Detection2DArray  onboard Hailo YOLO (reference)
/cmd_vel                          geometry_msgs/Twist           input of the walking controller

You must never publish to /cmd_vel directly. A multiplexer node, cmd_vel_mux, decides which command source is in control:

/teleop_cmd_vel            (joystick)          priority 0 - highest
/llm_cmd_vel               (Lab 9 pipeline)    priority 1
/person_following_cmd_vel  (vision pipeline)   priority 2 - lowest
        |
        v
    cmd_vel_mux  ->  /cmd_vel  ->  walking controller

A source is considered active only while it keeps publishing: after 500 ms of silence the mux drops to the next source. Two consequences:

• your controller must publish continuously (e.g. at 10 Hz) to stay in control;
• the joystick always wins — grabbing the controller is your first emergency stop. Keep it in hand whenever your code can move the robot.

Create the lab folder and a Python virtual environment with access to the system ROS 2 packages:

mkdir -p ~/lab_10_fall_2025 && cd ~/lab_10_fall_2025
python3 -m venv --system-site-packages venv
source venv/bin/activate
pip install ultralytics

The option –system-site-packages is important: it lets the venv see rclpy and cv2 that are already installed for ROS 2 (same trick as in Lab 9).

Check that everything is visible:

python -c "import rclpy, cv2, ultralytics; print('OK')"
ros2 topic hz /camera/image_raw/compressed

To see what Pupper sees, save one frame to disk:

#!/usr/bin/env python3
"""grab_frame.py - save one camera frame to disk."""
import cv2, numpy as np, rclpy
from rclpy.node import Node
from sensor_msgs.msg import CompressedImage
 
class FrameGrabber(Node):
    def __init__(self):
        super().__init__("frame_grabber")
        self.done = False
        self.create_subscription(CompressedImage, "/camera/image_raw/compressed", self.cb, 10)
 
    def cb(self, msg):
        img = cv2.imdecode(np.frombuffer(msg.data, np.uint8), cv2.IMREAD_COLOR)
        cv2.imwrite("frame.jpg", img)
        self.get_logger().info(f"Saved {img.shape}")
        self.done = True
 
rclpy.init()
node = FrameGrabber()
while rclpy.ok() and not node.done:
    rclpy.spin_once(node, timeout_sec=1.0)

Copy frame.jpg to your laptop with scp and look at it, or use Foxglove (the foxglove_bridge is already running on the robot) and add an Image panel on /camera/image_raw/compressed.

Write a ROS 2 node ball_detector.py that:

• subscribes to /camera/image_raw/compressed;
• runs YOLOv11-nano on each frame, keeping only the class sports ball (COCO id 32);
• publishes the boxes as vision_msgs/Detection2DArray on /ball_detections;
• publishes an annotated JPEG on /ball_detections/annotated so you can watch it in Foxglove.

Skeleton:

#!/usr/bin/env python3
"""ball_detector.py - YOLOv11 ball detection on the Pupper camera."""
import cv2, numpy as np, rclpy
from rclpy.node import Node
from sensor_msgs.msg import CompressedImage
from vision_msgs.msg import Detection2D, Detection2DArray, ObjectHypothesisWithPose
from ultralytics import YOLO
 
TARGET_CLASS = "sports ball"   # COCO class id 32
CONF_THRESHOLD = 0.35
IMGSZ = 320                    # inference resolution: keep small on the Pi CPU
 
class BallDetectorNode(Node):
    def __init__(self):
        super().__init__("ball_detector_node")
        self.model = YOLO("yolo11n.pt")   # downloads the weights on first run
        # TODO 1: find the numeric id of TARGET_CLASS in self.model.names
        self.target_id = ...
 
        self.det_pub = self.create_publisher(Detection2DArray, "/ball_detections", 10)
        self.img_pub = self.create_publisher(CompressedImage, "/ball_detections/annotated", 10)
        self.create_subscription(CompressedImage, "/camera/image_raw/compressed",
                                 self.image_callback, 1)
 
    def image_callback(self, msg):
        # TODO 2: decode the JPEG into an OpenCV image
        # hint: cv2.imdecode(np.frombuffer(msg.data, np.uint8), cv2.IMREAD_COLOR)
        frame = ...
        h, w = frame.shape[:2]
 
        # TODO 3: run the model on the frame
        # hint: self.model.predict(frame, imgsz=IMGSZ, conf=CONF_THRESHOLD,
        #                          classes=[self.target_id], verbose=False)[0]
        results = ...
 
        det_array = Detection2DArray()
        det_array.header = msg.header
        for box in results.boxes:
            x1, y1, x2, y2 = box.xyxy[0].tolist()
            det = Detection2D()
            det.header = msg.header
            # TODO 4: fill in det.bbox.center.position.x / .y,
            #         det.bbox.size_x / .size_y from x1, y1, x2, y2
            hyp = ObjectHypothesisWithPose()
            hyp.hypothesis.class_id = str(int(box.cls[0]))
            hyp.hypothesis.score = float(box.conf[0])
            det.results.append(hyp)
            det_array.detections.append(det)
            # TODO 5: draw the rectangle and the score on `frame` with cv2
 
        self.det_pub.publish(det_array)
 
        annotated = CompressedImage()
        annotated.header = msg.header
        annotated.format = "jpeg"
        annotated.data = cv2.imencode(".jpg", frame,
                                      [cv2.IMWRITE_JPEG_QUALITY, 70])[1].tobytes()
        self.img_pub.publish(annotated)
 
rclpy.init()
rclpy.spin(BallDetectorNode())

Run it inside the venv:

python ball_detector.py

Testing without chasing a ball around the lab. You can inject any image into your pipeline instead of the live camera. Write a tiny publisher that sends a JPEG in a loop:

#!/usr/bin/env python3
"""publish_test_image.py <image.jpg> - publish a JPEG as CompressedImage on /test_image."""
import sys, rclpy
from rclpy.node import Node
from sensor_msgs.msg import CompressedImage
 
class TestImagePublisher(Node):
    def __init__(self, path):
        super().__init__("test_image_publisher")
        self.jpeg = open(path, "rb").read()
        self.pub = self.create_publisher(CompressedImage, "/test_image", 1)
        self.create_timer(0.5, self.tick)
 
    def tick(self):
        msg = CompressedImage()
        msg.header.stamp = self.get_clock().now().to_msg()
        msg.format = "jpeg"
        msg.data = self.jpeg
        self.pub.publish(msg)
 
rclpy.init()
rclpy.spin(TestImagePublisher(sys.argv[1]))

then start your detector with the input remapped:

python ball_detector.py --ros-args -r /camera/image_raw/compressed:=/test_image

(for the remap to work, call rclpy.init(args=sys.argv) in your node, or just accept the default which already parses sys.argv when you use rclpy.init() with no arguments in a script run directly)

Download a real photo of a soccer ball, publish it, and check:

ros2 topic echo /ball_detections --once

With a correct implementation and a real ball photo you should see one detection with confidence around 0.8-0.9. A word of warning from our own testing: a clip-art ball on a white background is out of distribution for COCO and will not be detected — use a photograph. Interpret this failure: the model learned balls in context (grass, fields, hands), not the abstract concept of “round thing with pentagons”.

Checkpoint 1: show your assistant the annotated Foxglove stream with a live ball detection and the /ball_detections echo.

Now close the loop. This is visual servoing: the control error is defined directly in image space, no 3D reconstruction needed.

Two errors, two velocity commands:

err_yaw  = (x_box - W/2) / (W/2)        horizontal offset, normalized to [-1, 1]
err_dist = A_target - A_box / A_image   bounding box area as a distance proxy

wz = -( Kp_ang * err_yaw  + Kd_ang * d(err_yaw)/dt )     angular velocity (turn)
vx =    Kp_lin * err_dist + Kd_lin * d(err_dist)/dt      linear velocity (walk)

Intuition:

• ball left of center → err_yaw < 0 → wz > 0 → Pupper turns left. The minus sign matters: check the sign convention (positive angular.z is counter-clockwise, i.e. left);
• ball far away → small box → err_dist > 0 → Pupper walks forward. When the box area reaches A_target, Pupper stops at a polite distance from the ball;
• the derivative terms damp the response: without them, a proportional-only controller overshoots and oscillates around the target, exactly like in the motor control lab (Lab 4).

Write ball_follower.py:

• subscribe to /ball_detections, keep the highest-confidence box;
• compute the PD law above;
• clamp: |vx| ⇐ 0.4 m/s, |wz| ⇐ 1.0 rad/s;
• add a small deadzone (if |err| < 0.05, treat it as 0) so Pupper does not dance around the setpoint;
• add a watchdog: if no ball was seen for 1 s, publish a zero Twist and reset the derivative state;
• publish the Twist on a topic given by a parameter.

Suggested starting gains (they are the ones used by the built-in person follower, which is a P-only controller — read its source in ~/pupperv3-monorepo/ros2_ws/src/person_follower/ for inspiration):

Kp_angular = 2.0     Kd_angular = 0.15
Kp_linear  = 1.5     Kd_linear  = 0.1
target_bbox_area = 0.06

Test dry first. Publish to a scratch topic that the mux does not listen to:

python ball_follower.py --cmd-topic /ball_following_cmd_vel
ros2 topic echo /ball_following_cmd_vel

Feed it the test image from Task 1 and verify the signs. Example from our reference run (ball on the left third of the image, box already at target size):

err_yaw=-0.45 area=0.060 -> vx=+0.00 wz=+0.90     # turns left, does not advance

Move the test image around (crop the ball to different corners) and fill in a table:

Then go live. With the robot standing in an open area, an assistant present, and the joystick in your hand (remember: teleop overrides you at any moment, and 500 ms of joystick silence gives control back to your node):

python ball_follower.py --cmd-topic /person_following_cmd_vel

Roll the ball around and tune:

• oscillates around the ball heading → reduce Kp_angular or increase Kd_angular;
• reacts too slowly → increase Kp_angular;
• walks into the ball → increase target_bbox_area;
• jitters when the ball is centered → increase the deadzone.

Checkpoint 2: Pupper turns towards the ball, walks to it, and stops at a fixed distance. Kick the ball away — Pupper should reacquire and follow.

Following a ball is cute, but hardcoded. In Lab 9 you built the safe LLM pipeline:

user text -> LLM -> safety filter -> whitelisted robot command

Now you will let an LLM choose which behavior the vision pipeline runs. On the lab Pupper there is a Gemma 4 31B instance (Google's open-weights model, released April 2026) served with an OpenAI-compatible API:

http://localhost:8111/v1/chat/completions      model name: gemma-4-31b-it

It will be up when you start the lab. Check it:

curl http://localhost:8111/v1/models

Instead of Lab 9's “return one command token per line”, you will use proper function calling (also called tool calling): you describe a set of functions in JSON Schema, and the model returns a structured call — name + arguments — instead of free text. The safety philosophy is unchanged and non-negotiable:

• the LLM never publishes to ROS 2 topics;
• the LLM only selects from functions you whitelisted;
• your code validates the name and the arguments before doing anything.

Write llm_commander.py:

import json, requests
 
GEMMA_URL = "http://localhost:8111/v1/chat/completions"
MODEL = "gemma-4-31b-it"
 
TRACKABLE = ["person", "sports ball", "dog", "cat", "backpack", "bottle", "chair", "cup"]
 
TOOLS = [
    {"type": "function", "function": {
        "name": "follow_object",
        "description": "Make Pupper walk towards and follow an object visible in its camera.",
        "parameters": {
            "type": "object",
            "properties": {"object_class": {
                "type": "string", "enum": TRACKABLE,
                "description": "COCO class of the object to follow"}},
            "required": ["object_class"]}}},
    {"type": "function", "function": {
        "name": "stop",
        "description": "Stop all motion and stand still.",
        "parameters": {"type": "object", "properties": {}}}},
]
 
SYSTEM_PROMPT = ("You control a quadruped robot named Pupper. Decide which tool call "
                 "satisfies the user's request. Only use the provided tools; if the "
                 "request cannot be satisfied, call stop() and explain briefly.")
 
def ask_gemma(user_text):
    r = requests.post(GEMMA_URL, json={
        "model": MODEL,
        "messages": [{"role": "system", "content": SYSTEM_PROMPT},
                     {"role": "user", "content": user_text}],
        "tools": TOOLS,
        "temperature": 0}, timeout=60)
    r.raise_for_status()
    return r.json()["choices"][0]["message"]
 
def validate_and_dispatch(message):
    # TODO 1: read message["tool_calls"]; if empty, print the text answer and return
    # TODO 2: extract name and json.loads(arguments) of the first call
    # TODO 3: if name == "follow_object" and the class is in TRACKABLE:
    #           reconfigure your detector to that class and activate the follower
    #         elif name == "stop": deactivate the follower, publish one zero Twist
    #         else: REJECT and log - this is your safety filter
    ...

To make follow_object actually take effect, generalize Task 1: make TARGET_CLASS a ROS 2 parameter of your detector node (target_class), so the commander can switch it at runtime:

ros2 param set /ball_detector_node target_class "dog"

(declare the parameter with self.declare_parameter and read it in the callback, or register an add_on_set_parameters_callback — both are fine for this lab)

When the LLM pipeline drives the robot, publish the follower output on /llm_cmd_vel — that is exactly why the mux has that input, and it has priority over the plain vision input.

Test with at least these commands and record the results in a table (input, tool call returned, validated?, robot behavior):

Follow the ball.
Can you chase the dog?
Follow my backpack.
Stop right there.
Follow the ghost in the room.
Bring me a beer from the fridge.

The last two must end in a rejected call or stop() — if your pipeline makes the robot move on them, your safety filter has a hole.

Checkpoint 3: demonstrate “follow the ball”, a switch to another class by voice or text, and a graceful rejection.

Ask your detector to follow “the red fire extinguisher” and it will shrug: COCO has 80 classes and none of them is a fire extinguisher. What are the options when the object you care about is not in the training set?

1. Fine-tune the detector. Collect and label images of your object, retrain YOLO. Works well, but costs hours and a dataset, and must be repeated for every new object.
2. Open-vocabulary detectors. Models such as YOLO-World, OWL-ViT or Grounding DINO embed text and image in the same space, so the “class list” becomes a free-text prompt at inference time. This is the clean production answer, but these models are heavier than yolo11n.
3. Use the VLM you already have. Gemma 4 is multimodal — it accepts images. You can simply show it the camera frame and ask, in plain language, where the object is.

In this task you take option 3, because the infrastructure is already running on localhost:8111. The trick that makes it drop into your existing pipeline: publish the VLM answer as a Detection2DArray on the same /ball_detections topic. Your PD follower from Task 2 does not care who produced the box.

camera frame (downscaled, base64)
    -> Gemma 4: "find <description>, answer ONLY with JSON
       {"found": bool, "box": [x_min, y_min, x_max, y_max]} normalized to [0,1]"
    -> parse + validate JSON
    -> Detection2DArray on /ball_detections
    -> your PD follower (unchanged)

Skeleton for vlm_detector.py:

import base64, json, cv2
 
def frame_to_b64(frame, width=448):
    h, w = frame.shape[:2]
    small = cv2.resize(frame, (width, int(h * width / w)))
    return base64.b64encode(cv2.imencode(".jpg", small,
                            [cv2.IMWRITE_JPEG_QUALITY, 80])[1]).decode()
 
def ask_vlm(jpeg_b64, query):
    payload = {
        "model": "gemma-4-31b-it",
        "messages": [{"role": "user", "content": [
            {"type": "image_url",
             "image_url": {"url": f"data:image/jpeg;base64,{jpeg_b64}"}},
            {"type": "text", "text":
             f"Find '{query}' in this image. Reply ONLY with JSON: "
             '{"found": bool, "box": [x_min, y_min, x_max, y_max]} '
             "with coordinates normalized to [0, 1]."}]}],
        "temperature": 0,
        "response_format": {"type": "json_object"},
    }
    # TODO: POST to the Gemma server, parse the JSON answer,
    #       convert the normalized box back to pixel coordinates,
    #       return it (or None if not found)

Build the node around it: keep the latest camera frame in the callback, query the VLM from a timer (a 31B model needs a few seconds per image — do NOT call it from the image callback), and publish the box.

Things to experiment with and discuss in your report:

• Latency. Your detections now arrive every ~3-5 s instead of 4 Hz. What does that do to the PD controller? Reduce max_linear_vel and let the watchdog stop the robot between detections, or hold the last command — try both and compare.
• Prompting. Does “the red mug” work better than “mug”? What happens with two similar objects in view? Can you ask Gemma to pick the leftmost one?
• Hallucinations. Ask for an object that is not in the room. Does Gemma ever return found: true with an invented box? Add a sanity check (box size, aspect ratio) to your validator. This is the same lesson as Lab 9: never trust raw model output.
• (bonus) Hybrid pipeline. Run YOLO for the 80 known classes at 4 Hz and the VLM only to choose which YOLO box matches an ambiguous description (“the person wearing red”). Best of both worlds: YOLO's rate, VLM's vocabulary.

Checkpoint 4: Pupper walks to an object of your choice that is NOT a COCO class, commanded in natural language through the full pipeline: text → LLM tool call → VLM detection → PD controller.

Use the virtual environment from the Setup section. Do not install into the system Python — the –break-system-packages flag risks breaking the actual robot stack.

Use yolo11n.pt (nano) and imgsz=320. The first inference after loading compiles the model and takes several seconds — that is normal, warm-up. Keep the subscriber queue depth at 1 so you always process the freshest frame instead of building a backlog.

Check the confidence threshold (start at 0.35, lower it while debugging), the distance (a far ball is a few pixels — YOLO needs some area), and the lighting. Test on a photograph first; remember the clip-art lesson from Task 1.

You are publishing to /cmd_vel directly (forbidden and ignored by design) or to a topic the mux does not subscribe to, or a higher-priority source (the joystick!) is active. Check what the mux sees:

ros2 topic echo /cmd_vel_mux/active_source

Also remember the 500 ms timeout: publishing a single Twist once is not enough, publish at ~10 Hz.

Sign error in wz. Positive angular.z turns the robot counter-clockwise (left). Re-derive the sign from err_yaw.

Keep “response_format”: {“type”: “json_object”} in the request, keep ONLY with JSON in the prompt, and still wrap the parse in a validator that rejects malformed answers. LLM output is input data, not gospel.

Submit:

• ball_detector.py, ball_follower.py, llm_commander.py, vlm_detector.py;
• a screenshot of the annotated detection stream in Foxglove;
• the sign-verification table from Task 2 and your final PD gains, with one sentence per gain explaining the tuning;
• a short video of Pupper following the ball (Task 2 checkpoint);
• the command table from Task 3, including the rejected commands;
• a short video of the open-vocabulary following from Task 4, plus the object description you used;
• answers to the Task 4 discussion points (latency, prompting, hallucinations) — a paragraph each;
• one paragraph: when would you choose DETR over YOLO on a robot, and when a VLM over both?

pixels -> boxes -> errors -> velocities

• A bounding box and a P(D) controller are enough to walk to an object — relative information beats absolute maps for reactive behaviors.
• Detector choice is an engineering tradeoff: YOLO for speed on the edge, DETR for architectural cleanliness, VLMs for open vocabulary at the price of latency.
• The LLM/VLM never touches actuators. It emits structured, whitelisted decisions; your code validates them; the controller executes them. This chain — model, filter, controller — is the same pattern from Lab 9, and it scales from “follow the ball” to any embodied AI system you will build after this course.

• Stanford CS123, Lecture 7: Computer Vision for Robotics
• Ultralytics YOLOv11 documentation
• DETR: End-to-End Object Detection with Transformers (Carion et al., 2020)
• YOLO-World: Real-Time Open-Vocabulary Object Detection
• Gemma 4 (Google DeepMind)
• vision_msgs message definitions
• Pupper v3 documentation