Show page

Differences

This shows you the differences between two versions of the page.

--- rasb:lab:10 [2026/06/04 19:28]
ciprian.popescu0411 created
+++ rasb:lab:10 [2026/07/11 19:50] (current)
atoader
@@ Line 1: / Line 1: @@
-===== Lab 10: =====
+====== Lab 10: Computer Vision for Robotics ======
+In this lab, Pupper finally gets to use its eyes.
+You will build a complete perception-to-action pipeline: the camera image goes through an object detector, the detector output feeds a PD controller, and the PD controller makes Pupper walk towards a ball. After that works, you will put the LLM from [[.:09|Lab 9]] back in the loop so that Pupper decides //what// to follow based on natural language, and finally you will use a vision-language model to follow objects that no classic detector knows about.
+The full pipeline you will build:
+<code>
+camera image
+    -> object detector (YOLOv11)
+    -> bounding box of the target
+    -> PD controller (visual servoing)
+    -> velocity command (/cmd_vel)
+    -> Pupper walks towards the target
+</code>
+The key idea from the Stanford lecture: **if we know our location relative to an object, we do not need to know where we are, or where the object is, in absolute terms**. A bounding box in the image is enough to walk to the object — no map, no localization, no GPS.
+===== Background: two ways to detect objects =====
+Object detection means answering two questions at once: //what// is in the image and //where// is it. The detector returns a list of boxes, each with a class label and a confidence score. Today, two families of architectures dominate.
+==== YOLO: You Only Look Once ====
+YOLO is a **single-shot** detector: one forward pass of a convolutional network produces all the boxes at the same time.
+The classic YOLO formulation divides the image into a grid (7x7 in the original paper). Each grid cell predicts:
+    * a few candidate bounding boxes, each described by center coordinates ''(cx, cy)'', size ''(w, h)'' and a confidence score;
+    * the class probabilities for the object centered in that cell.
+In the original paper this meant 30 output values per cell: 2 candidate boxes x (cx, cy, w, h, conf) + 20 class scores. Localization and classification happen simultaneously — this is why YOLO is fast enough for robots and why it "only looks once".
+Because several cells may fire on the same object, YOLO needs a post-processing step called **non-maximum suppression (NMS)**: keep the highest-confidence box, drop all boxes that overlap it too much, repeat.
+Modern versions (we will use **YOLOv11** from Ultralytics) are anchor-free, NMS is built into the pipeline, and the models come in sizes from ''n'' (nano, ~2.6M parameters) to ''x'' (extra large). They are trained on the **COCO dataset**: 80 everyday classes such as ''person'', ''dog'', ''chair'' and — conveniently for us — ''sports ball''.
+==== DETR: DEtection TRansformer ====
+DETR treats detection as a **set prediction** problem using a transformer:
+    * a CNN backbone extracts image features;
+    * a transformer encoder-decoder processes them;
+    * a fixed set of learned **object queries** (e.g. 100) each "asks" the image about one potential object;
+    * each query outputs a box + class directly.
+During training, a bipartite (Hungarian) matching assigns each predicted box to at most one ground-truth object, so DETR needs **no NMS and no anchors** — the architecture is conceptually much cleaner. The price is slower convergence in training and, for the original DETR, slower inference. Newer variants such as RT-DETR close the speed gap.
+==== Which one for a robot? ====
+^ ^ YOLO (v11) ^ DETR ^
+| Inference speed on edge hardware | very fast, designed for it | usually slower (RT-DETR is competitive) |
+| Post-processing | NMS (handled internally) | none — set prediction |
+| Small objects | good | weaker (original DETR) |
+| Deployment (ONNX, accelerators) | mature tooling | improving |
+| Conceptual elegance | grid + suppression heuristics | end-to-end, clean |
+On a robot with a Raspberry Pi CPU, the practical answer today is YOLO, and the nano variant. Keep in mind the fundamental limitation of both: **they only know the classes they were trained on**. A COCO-trained model cannot detect "a fire extinguisher" or "the blue whiteboard marker". We return to this problem in Task 4.
+**A note on hardware.** Pupper's head also contains a Hailo-8L neural accelerator. The production stack already runs a YOLO model on it: the ''hailo_detection_node'' subscribes to the camera and publishes ''/detections'' at camera rate, which is what the built-in person-following demo uses. In this lab you will run YOLOv11 on the CPU to own the whole pipeline yourself — measured on the robot, ''yolo11n'' at 320 px input takes about **230 ms/frame (~4 FPS)**, which is enough for a walking robot. Once your pipeline works, comparing your node with the Hailo one is a nice exercise.
+===== The Pupper vision and control stack =====
+SSH into your Pupper. The ROS 2 stack runs as a systemd service (''sudo systemctl status robot''). The topics relevant for this lab:
+<code>
+/camera/image_raw/compressed      sensor_msgs/CompressedImage   camera feed, 1400x1050
+/detections                       vision_msgs/Detection2DArray  onboard Hailo YOLO (reference)
+/cmd_vel                          geometry_msgs/Twist           input of the walking controller
+</code>
+You must **never publish to /cmd_vel directly**. A multiplexer node, ''cmd_vel_mux'', decides which command source is in control:
+<code>
+/teleop_cmd_vel            (joystick)          priority 0 - highest
+/llm_cmd_vel               (Lab 9 pipeline)    priority 1
+/person_following_cmd_vel  (vision pipeline)   priority 2 - lowest
+        |
+        v
+    cmd_vel_mux  ->  /cmd_vel  ->  walking controller
+</code>
+A source is considered active only while it keeps publishing: after **500 ms** of silence the mux drops to the next source. Two consequences:
+    * your controller must publish continuously (e.g. at 10 Hz) to stay in control;
+    * the **joystick always wins** — grabbing the controller is your first emergency stop. Keep it in hand whenever your code can move the robot.
+===== Setup =====
+Create the lab folder and a Python virtual environment **with access to the system ROS 2 packages**:
+<code bash>
+mkdir -p ~/lab_10_fall_2025 && cd ~/lab_10_fall_2025
+python3 -m venv --system-site-packages venv
+source venv/bin/activate
+pip install ultralytics
+</code>
+The option ''--system-site-packages'' is important: it lets the venv see ''rclpy'' and ''cv2'' that are already installed for ROS 2 (same trick as in Lab 9).
+Check that everything is visible:
+<code bash>
+python -c "import rclpy, cv2, ultralytics; print('OK')"
+ros2 topic hz /camera/image_raw/compressed
+</code>
+To see what Pupper sees, save one frame to disk:
+<code python>
+#!/usr/bin/env python3
+"""grab_frame.py - save one camera frame to disk."""
+import cv2, numpy as np, rclpy
+from rclpy.node import Node
+from sensor_msgs.msg import CompressedImage
+class FrameGrabber(Node):
+    def __init__(self):
+        super().__init__("frame_grabber")
+        self.done = False
+        self.create_subscription(CompressedImage, "/camera/image_raw/compressed", self.cb, 10)
+    def cb(self, msg):
+        img = cv2.imdecode(np.frombuffer(msg.data, np.uint8), cv2.IMREAD_COLOR)
+        cv2.imwrite("frame.jpg", img)
+        self.get_logger().info(f"Saved {img.shape}")
+        self.done = True
+rclpy.init()
+node = FrameGrabber()
+while rclpy.ok() and not node.done:
+    rclpy.spin_once(node, timeout_sec=1.0)
+</code>
+Copy ''frame.jpg'' to your laptop with ''scp'' and look at it, or use **Foxglove** (the ''foxglove_bridge'' is already running on the robot) and add an Image panel on ''/camera/image_raw/compressed''.
+===== Task 1 - Detect the ball with YOLOv11 =====
+Write a ROS 2 node ''ball_detector.py'' that:
+    * subscribes to ''/camera/image_raw/compressed'';
+    * runs YOLOv11-nano on each frame, keeping only the class ''sports ball'' (COCO id 32);
+    * publishes the boxes as ''vision_msgs/Detection2DArray'' on ''/ball_detections'';
+    * publishes an annotated JPEG on ''/ball_detections/annotated'' so you can watch it in Foxglove.
+Skeleton:
+<code python>
+#!/usr/bin/env python3
+"""ball_detector.py - YOLOv11 ball detection on the Pupper camera."""
+import cv2, numpy as np, rclpy
+from rclpy.node import Node
+from sensor_msgs.msg import CompressedImage
+from vision_msgs.msg import Detection2D, Detection2DArray, ObjectHypothesisWithPose
+from ultralytics import YOLO
+TARGET_CLASS = "sports ball"   # COCO class id 32
+CONF_THRESHOLD = 0.35
+IMGSZ = 320                    # inference resolution: keep small on the Pi CPU
+class BallDetectorNode(Node):
+    def __init__(self):
+        super().__init__("ball_detector_node")
+        self.model = YOLO("yolo11n.pt")   # downloads the weights on first run
+        # TODO 1: find the numeric id of TARGET_CLASS in self.model.names
+        self.target_id = ...
+        self.det_pub = self.create_publisher(Detection2DArray, "/ball_detections", 10)
+        self.img_pub = self.create_publisher(CompressedImage, "/ball_detections/annotated", 10)
+        self.create_subscription(CompressedImage, "/camera/image_raw/compressed",
+                                 self.image_callback, 1)
+    def image_callback(self, msg):
+        # TODO 2: decode the JPEG into an OpenCV image
+        # hint: cv2.imdecode(np.frombuffer(msg.data, np.uint8), cv2.IMREAD_COLOR)
+        frame = ...
+        h, w = frame.shape[:2]
+        # TODO 3: run the model on the frame
+        # hint: self.model.predict(frame, imgsz=IMGSZ, conf=CONF_THRESHOLD,
+        #                          classes=[self.target_id], verbose=False)[0]
+        results = ...
+        det_array = Detection2DArray()
+        det_array.header = msg.header
+        for box in results.boxes:
+            x1, y1, x2, y2 = box.xyxy[0].tolist()
+            det = Detection2D()
+            det.header = msg.header
+            # TODO 4: fill in det.bbox.center.position.x / .y,
+            #         det.bbox.size_x / .size_y from x1, y1, x2, y2
+            hyp = ObjectHypothesisWithPose()
+            hyp.hypothesis.class_id = str(int(box.cls[0]))
+            hyp.hypothesis.score = float(box.conf[0])
+            det.results.append(hyp)
+            det_array.detections.append(det)
+            # TODO 5: draw the rectangle and the score on `frame` with cv2
+        self.det_pub.publish(det_array)
+        annotated = CompressedImage()
+        annotated.header = msg.header
+        annotated.format = "jpeg"
+        annotated.data = cv2.imencode(".jpg", frame,
+                                      [cv2.IMWRITE_JPEG_QUALITY, 70])[1].tobytes()
+        self.img_pub.publish(annotated)
+rclpy.init()
+rclpy.spin(BallDetectorNode())
+</code>
+Run it inside the venv:
+<code bash>
+python ball_detector.py
+</code>
+**Testing without chasing a ball around the lab.** You can inject any image into your pipeline instead of the live camera. Write a tiny publisher that sends a JPEG in a loop:
+<code python>
+#!/usr/bin/env python3
+"""publish_test_image.py <image.jpg> - publish a JPEG as CompressedImage on /test_image."""
+import sys, rclpy
+from rclpy.node import Node
+from sensor_msgs.msg import CompressedImage
+class TestImagePublisher(Node):
+    def __init__(self, path):
+        super().__init__("test_image_publisher")
+        self.jpeg = open(path, "rb").read()
+        self.pub = self.create_publisher(CompressedImage, "/test_image", 1)
+        self.create_timer(0.5, self.tick)
+    def tick(self):
+        msg = CompressedImage()
+        msg.header.stamp = self.get_clock().now().to_msg()
+        msg.format = "jpeg"
+        msg.data = self.jpeg
+        self.pub.publish(msg)
+rclpy.init()
+rclpy.spin(TestImagePublisher(sys.argv[1]))
+</code>
+then start your detector with the input remapped:
+<code bash>
+python ball_detector.py --ros-args -r /camera/image_raw/compressed:=/test_image
+</code>
+(for the remap to work, call ''rclpy.init(args=sys.argv)'' in your node, or just accept the default which already parses ''sys.argv'' when you use ''rclpy.init()'' with no arguments in a script run directly)
+Download a real photo of a soccer ball, publish it, and check:
+<code bash>
+ros2 topic echo /ball_detections --once
+</code>
+With a correct implementation and a real ball photo you should see one detection with confidence around 0.8-0.9. A word of warning from our own testing: a clip-art ball on a white background is **out of distribution** for COCO and will not be detected — use a photograph. Interpret this failure: the model learned balls //in context// (grass, fields, hands), not the abstract concept of "round thing with pentagons".
+**Checkpoint 1:** show your assistant the annotated Foxglove stream with a live ball detection and the ''/ball_detections'' echo.
+===== Task 2 - A PD controller to walk to the ball =====
+Now close the loop. This is **visual servoing**: the control error is defined directly in image space, no 3D reconstruction needed.
+Two errors, two velocity commands:
+<code>
+err_yaw  = (x_box - W/2) / (W/2)        horizontal offset, normalized to [-1, 1]
+err_dist = A_target - A_box / A_image   bounding box area as a distance proxy
+wz = -( Kp_ang * err_yaw  + Kd_ang * d(err_yaw)/dt )     angular velocity (turn)
+vx =    Kp_lin * err_dist + Kd_lin * d(err_dist)/dt      linear velocity (walk)
+</code>
+Intuition:
+    * ball left of center -> ''err_yaw < 0'' -> ''wz > 0'' -> Pupper turns left. The minus sign matters: check the sign convention (positive ''angular.z'' is counter-clockwise, i.e. left);
+    * ball far away -> small box -> ''err_dist > 0'' -> Pupper walks forward. When the box area reaches ''A_target'', Pupper stops at a polite distance from the ball;
+    * the derivative terms damp the response: without them, a proportional-only controller overshoots and oscillates around the target, exactly like in the motor control lab (Lab 4).
+Write ''ball_follower.py'':
+    * subscribe to ''/ball_detections'', keep the highest-confidence box;
+    * compute the PD law above;
+    * clamp: ''|vx| <= 0.4 m/s'', ''|wz| <= 1.0 rad/s'';
+    * add a small **deadzone** (if ''|err| < 0.05'', treat it as 0) so Pupper does not dance around the setpoint;
+    * add a **watchdog**: if no ball was seen for 1 s, publish a zero ''Twist'' and reset the derivative state;
+    * publish the ''Twist'' on a topic given by a parameter.
+Suggested starting gains (they are the ones used by the built-in person follower, which is a P-only controller — read its source in ''~/pupperv3-monorepo/ros2_ws/src/person_follower/'' for inspiration):
+<code>
+Kp_angular = 2.0     Kd_angular = 0.15
+Kp_linear  = 1.5     Kd_linear  = 0.1
+target_bbox_area = 0.06
+</code>
+**Test dry first.** Publish to a scratch topic that the mux does not listen to:
+<code bash>
+python ball_follower.py --cmd-topic /ball_following_cmd_vel
+ros2 topic echo /ball_following_cmd_vel
+</code>
+Feed it the test image from Task 1 and verify the signs. Example from our reference run (ball on the left third of the image, box already at target size):
+<code>
+err_yaw=-0.45 area=0.060 -> vx=+0.00 wz=+0.90     # turns left, does not advance
+</code>
+Move the test image around (crop the ball to different corners) and fill in a table:
+^ Ball position in image ^ err_yaw ^ expected wz sign ^ measured wz ^ vx behavior ^
+| left | negative | positive (turn left) | ... | ... |
+| right | positive | negative (turn right) | ... | ... |
+| centered, small box | ~0 | ~0 | ... | forward |
+| centered, huge box | ~0 | ~0 | ... | backward |
+**Then go live.** With the robot standing in an open area, an assistant present, and the joystick in your hand (remember: teleop overrides you at any moment, and 500 ms of joystick silence gives control back to your node):
+<code bash>
+python ball_follower.py --cmd-topic /person_following_cmd_vel
+</code>
+Roll the ball around and tune:
+    * oscillates around the ball heading -> reduce ''Kp_angular'' or increase ''Kd_angular'';
+    * reacts too slowly -> increase ''Kp_angular'';
+    * walks into the ball -> increase ''target_bbox_area'';
+    * jitters when the ball is centered -> increase the deadzone.
+**Checkpoint 2:** Pupper turns towards the ball, walks to it, and stops at a fixed distance. Kick the ball away — Pupper should reacquire and follow.
+===== Task 3 - Let the LLM decide what to do =====
+Following a ball is cute, but hardcoded. In [[.:09|Lab 9]] you built the safe LLM pipeline:
+<code>
+user text -> LLM -> safety filter -> whitelisted robot command
+</code>
+Now you will let an LLM choose //which behavior// the vision pipeline runs. On the lab Pupper there is a **Gemma 4 31B** instance (Google's open-weights model, released April 2026) served with an OpenAI-compatible API:
+<code>
+http://localhost:8111/v1/chat/completions      model name: gemma-4-31b-it
+</code>
+It will be up when you start the lab. Check it:
+<code bash>
+curl http://localhost:8111/v1/models
+</code>
+Instead of Lab 9's "return one command token per line", you will use proper **function calling** (also called tool calling): you describe a set of functions in JSON Schema, and the model returns a structured call — name + arguments — instead of free text. The safety philosophy is unchanged and non-negotiable:
+    * the LLM **never** publishes to ROS 2 topics;
+    * the LLM only //selects// from functions you whitelisted;
+    * your code validates the name and the arguments before doing anything.
+Write ''llm_commander.py'':
+<code python>
+import json, requests
+GEMMA_URL = "http://localhost:8111/v1/chat/completions"
+MODEL = "gemma-4-31b-it"
+TRACKABLE = ["person", "sports ball", "dog", "cat", "backpack", "bottle", "chair", "cup"]
+TOOLS = [
+    {"type": "function", "function": {
+        "name": "follow_object",
+        "description": "Make Pupper walk towards and follow an object visible in its camera.",
+        "parameters": {
+            "type": "object",
+            "properties": {"object_class": {
+                "type": "string", "enum": TRACKABLE,
+                "description": "COCO class of the object to follow"}},
+            "required": ["object_class"]}}},
+    {"type": "function", "function": {
+        "name": "stop",
+        "description": "Stop all motion and stand still.",
+        "parameters": {"type": "object", "properties": {}}}},
+]
+SYSTEM_PROMPT = ("You control a quadruped robot named Pupper. Decide which tool call "
+                 "satisfies the user's request. Only use the provided tools; if the "
+                 "request cannot be satisfied, call stop() and explain briefly.")
+def ask_gemma(user_text):
+    r = requests.post(GEMMA_URL, json={
+        "model": MODEL,
+        "messages": [{"role": "system", "content": SYSTEM_PROMPT},
+                     {"role": "user", "content": user_text}],
+        "tools": TOOLS,
+        "temperature": 0}, timeout=60)
+    r.raise_for_status()
+    return r.json()["choices"][0]["message"]
+def validate_and_dispatch(message):
+    # TODO 1: read message["tool_calls"]; if empty, print the text answer and return
+    # TODO 2: extract name and json.loads(arguments) of the first call
+    # TODO 3: if name == "follow_object" and the class is in TRACKABLE:
+    #           reconfigure your detector to that class and activate the follower
+    #         elif name == "stop": deactivate the follower, publish one zero Twist
+    #         else: REJECT and log - this is your safety filter
+    ...
+</code>
+To make ''follow_object'' actually take effect, generalize Task 1: make ''TARGET_CLASS'' a ROS 2 **parameter** of your detector node (''target_class''), so the commander can switch it at runtime:
+<code bash>
+ros2 param set /ball_detector_node target_class "dog"
+</code>
+(declare the parameter with ''self.declare_parameter'' and read it in the callback, or register an ''add_on_set_parameters_callback'' — both are fine for this lab)
+When the LLM pipeline drives the robot, publish the follower output on ''/llm_cmd_vel'' — that is exactly why the mux has that input, and it has priority over the plain vision input.
+Test with at least these commands and record the results in a table (input, tool call returned, validated?, robot behavior):
+<code>
+Follow the ball.
+Can you chase the dog?
+Follow my backpack.
+Stop right there.
+Follow the ghost in the room.
+Bring me a beer from the fridge.
+</code>
+The last two must end in a rejected call or ''stop()'' — if your pipeline makes the robot move on them, your safety filter has a hole.
+**Checkpoint 3:** demonstrate "follow the ball", a switch to another class by voice or text, and a graceful rejection.
+===== Task 4 - Beyond the 80 classes: open-vocabulary detection =====
+Ask your detector to follow "the red fire extinguisher" and it will shrug: COCO has 80 classes and none of them is a fire extinguisher. What are the options when the object you care about is not in the training set?
+    - **Fine-tune the detector.** Collect and label images of your object, retrain YOLO. Works well, but costs hours and a dataset, and must be repeated for every new object.
+    - **Open-vocabulary detectors.** Models such as YOLO-World, OWL-ViT or Grounding DINO embed //text and image in the same space//, so the "class list" becomes a free-text prompt at inference time. This is the clean production answer, but these models are heavier than ''yolo11n''.
+    - **Use the VLM you already have.** Gemma 4 is **multimodal** — it accepts images. You can simply show it the camera frame and ask, in plain language, where the object is.
+In this task you take option 3, because the infrastructure is already running on ''localhost:8111''. The trick that makes it drop into your existing pipeline: **publish the VLM answer as a ''Detection2DArray'' on the same ''/ball_detections'' topic**. Your PD follower from Task 2 does not care who produced the box.
+<code>
+camera frame (downscaled, base64)
+    -> Gemma 4: "find <description>, answer ONLY with JSON
+       {"found": bool, "box": [x_min, y_min, x_max, y_max]} normalized to [0,1]"
+    -> parse + validate JSON
+    -> Detection2DArray on /ball_detections
+    -> your PD follower (unchanged)
+</code>
+Skeleton for ''vlm_detector.py'':
+<code python>
+import base64, json, cv2
+def frame_to_b64(frame, width=448):
+    h, w = frame.shape[:2]
+    small = cv2.resize(frame, (width, int(h * width / w)))
+    return base64.b64encode(cv2.imencode(".jpg", small,
+                            [cv2.IMWRITE_JPEG_QUALITY, 80])[1]).decode()
+def ask_vlm(jpeg_b64, query):
+    payload = {
+        "model": "gemma-4-31b-it",
+        "messages": [{"role": "user", "content": [
+            {"type": "image_url",
+             "image_url": {"url": f"data:image/jpeg;base64,{jpeg_b64}"}},
+            {"type": "text", "text":
+             f"Find '{query}' in this image. Reply ONLY with JSON: "
+             '{"found": bool, "box": [x_min, y_min, x_max, y_max]} '
+             "with coordinates normalized to [0, 1]."}]}],
+        "temperature": 0,
+        "response_format": {"type": "json_object"},
+    }
+    # TODO: POST to the Gemma server, parse the JSON answer,
+    #       convert the normalized box back to pixel coordinates,
+    #       return it (or None if not found)
+</code>
+Build the node around it: keep the latest camera frame in the callback, query the VLM from a **timer** (a 31B model needs a few seconds per image — do NOT call it from the image callback), and publish the box.
+Things to experiment with and discuss in your report:
+    * **Latency.** Your detections now arrive every ~3-5 s instead of 4 Hz. What does that do to the PD controller? Reduce ''max_linear_vel'' and let the watchdog stop the robot between detections, or hold the last command — try both and compare.
+    * **Prompting.** Does "the red mug" work better than "mug"? What happens with two similar objects in view? Can you ask Gemma to pick the leftmost one?
+    * **Hallucinations.** Ask for an object that is not in the room. Does Gemma ever return ''found: true'' with an invented box? Add a sanity check (box size, aspect ratio) to your validator. This is the same lesson as Lab 9: **never trust raw model output**.
+    * (bonus) **Hybrid pipeline.** Run YOLO for the 80 known classes at 4 Hz and the VLM only to //choose// which YOLO box matches an ambiguous description ("the person wearing red"). Best of both worlds: YOLO's rate, VLM's vocabulary.
+**Checkpoint 4:** Pupper walks to an object of your choice that is NOT a COCO class, commanded in natural language through the full pipeline: text -> LLM tool call -> VLM detection -> PD controller.
+===== Common problems =====
+==== pip refuses to install (externally-managed-environment) ====
+Use the virtual environment from the Setup section. Do not install into the system Python — the ''--break-system-packages'' flag risks breaking the actual robot stack.
+==== Inference is very slow / the robot reacts with a big delay ====
+Use ''yolo11n.pt'' (nano) and ''imgsz=320''. The first inference after loading compiles the model and takes several seconds — that is normal, warm-up. Keep the subscriber queue depth at 1 so you always process the freshest frame instead of building a backlog.
+==== The ball is not detected ====
+Check the confidence threshold (start at 0.35, lower it while debugging), the distance (a far ball is a few pixels — YOLO needs some area), and the lighting. Test on a photograph first; remember the clip-art lesson from Task 1.
+==== My node publishes velocities but the robot does not move ====
+You are publishing to ''/cmd_vel'' directly (forbidden and ignored by design) or to a topic the mux does not subscribe to, or a higher-priority source (the joystick!) is active. Check what the mux sees:
+<code bash>
+ros2 topic echo /cmd_vel_mux/active_source
+</code>
+Also remember the 500 ms timeout: publishing a single ''Twist'' once is not enough, publish at ~10 Hz.
+==== The robot moves but keeps turning away from the ball ====
+Sign error in ''wz''. Positive ''angular.z'' turns the robot counter-clockwise (left). Re-derive the sign from ''err_yaw''.
+==== The VLM answers with prose instead of JSON ====
+Keep ''"response_format": {"type": "json_object"}'' in the request, keep ''ONLY with JSON'' in the prompt, and still wrap the parse in a validator that rejects malformed answers. LLM output is input data, not gospel.
+===== Deliverables =====
+Submit:
+    * ''ball_detector.py'', ''ball_follower.py'', ''llm_commander.py'', ''vlm_detector.py'';
+    * a screenshot of the annotated detection stream in Foxglove;
+    * the sign-verification table from Task 2 and your final PD gains, with one sentence per gain explaining the tuning;
+    * a short video of Pupper following the ball (Task 2 checkpoint);
+    * the command table from Task 3, including the rejected commands;
+    * a short video of the open-vocabulary following from Task 4, plus the object description you used;
+    * answers to the Task 4 discussion points (latency, prompting, hallucinations) — a paragraph each;
+    * one paragraph: when would you choose DETR over YOLO on a robot, and when a VLM over both?
+===== What to remember =====
+<code>
+pixels -> boxes -> errors -> velocities
+</code>
+    * A bounding box and a P(D) controller are enough to walk to an object — relative information beats absolute maps for reactive behaviors.
+    * Detector choice is an engineering tradeoff: YOLO for speed on the edge, DETR for architectural cleanliness, VLMs for open vocabulary at the price of latency.
+    * The LLM/VLM never touches actuators. It emits structured, whitelisted decisions; your code validates them; the controller executes them. This chain — model, filter, controller — is the same pattern from Lab 9, and it scales from "follow the ball" to any embodied AI system you will build after this course.
+===== Resources =====
+    * [[https://cs123-stanford.readthedocs.io/en/latest/schedule/lectures/fall-25/lec-7.html|Stanford CS123, Lecture 7: Computer Vision for Robotics]]
+    * [[https://docs.ultralytics.com/models/yolo11/|Ultralytics YOLOv11 documentation]]
+    * [[https://arxiv.org/abs/2005.12872|DETR: End-to-End Object Detection with Transformers (Carion et al., 2020)]]
+    * [[https://arxiv.org/abs/2401.17270|YOLO-World: Real-Time Open-Vocabulary Object Detection]]
+    * [[https://deepmind.google/models/gemma/gemma-4/|Gemma 4 (Google DeepMind)]]
+    * [[https://docs.ros.org/en/jazzy/p/vision_msgs/|vision_msgs message definitions]]
+    * [[https://pupper-v3-documentation.readthedocs.io/en/latest/|Pupper v3 documentation]]

Labs

Resources

rasb/lab/10.1780590536.txt.gz · Last modified: 2026/06/04 19:28 by ciprian.popescu0411

Show page Old revisions

Media Manager Back to top