Lab 02 - Teletypes & frame buffers

Objectives

• Getting used to image manipulation with PIL
• Learn about Teletypes and Pseudo-Terminal Slaves
• Interact with the frame buffer

Proof of Work

Today we're going to start off by reinventing the wheel, a little bit. The first exercise will have you reimplementing the screenshot functionality in a Python3 script. Before you start gleaming with joy, thinking that there's probably a module / library out there doing just this, know that 1) you're probably right and 2) we won't be using them :). In stead, we'll take the hard road and fetch our pixels right from the frame buffer… Now that I'm thinking about it, when you consider the state of pip and module dependency hell in Python, which one even is the hard road ultimately?

Anyway, in the second half of the lab, we'll do the exact opposite of what we'll have done in the first. In stead of taking a screenshot, our second script will overwrite the frame buffer (and consequently what's being displayed on our screen) with the content of a randomly selected image.

The submission for this lab must contain the two scripts, a screenshot taken with the first script demonstrating that the second script works, and optionally proof that you've completed the feedback form (for the bonus 10p). As usual, check out the moodle assignment. The deadline is set for 23:55 this Sunday evening. Everyone must upload their work, regardless of whether they were present for the lab or not.

01. [40p] Reading the frame buffer

By now, we all know that not all files are backed by a storage device. While some reside on your HDD/SSD, Linux mounts virtual filesystems such as procfs or sysfs. Today we are going to look at a special file called /dev/fb0. This file is an abstraction of the frame buffer used by your video hardware to render your screen. In it, each pixel is represented as a RGBA (Red-Green-Blue-Alpha) value. Creating a copy of this file to persistent storage will effectively take a screenshot. Unfortunately, this screenshot will not be in any known format (e.g.: JPEG, PNG, etc.) A bit of work still needs to be done to properly format it such that the appropriate software (e.g.: gwenview) can interpret and display it.

In this first exercise, we are going to write a Python3 script that does just that: take the content of /dev/fb0 and output an image file in a format of your choosing. To that end, we are going to use the PIL module.

[10p] Task A - Setting up the environment

If you are working on Ubuntu, you may first need to install some prerequisites:

# install developer versions of compression / jpeg handling libraries
$sudo apt update$ sudo apt install zlib1g-dev libjpeg-dev

Next, we are going to set up a virtual environment for us to work in. If you need a reminder on what a venv actually is, refer to this lab.

# this will be our workspace directory
# NOTE: !$is substituted with the last argument of your previous command$ mkdir frame_buffer
$cd !$

# create venv and activate it
$python3 -m venv .venv$ source .venv/bin/activate

# install pillow (an actively maintained fork of PIL)
$pip3 install pillow [20p] Task B - Writing the script To start things off, we are going to use the following skeleton: fb2img.py #!.venv/bin/python3 import argparse # argument parsing import struct # data unpacking from PIL import Image # image processing def main(): # parse cli arguments parser = argparse.ArgumentParser() parser.add_argument('FILE', help='output image file') parser.add_argument('--src', help='data source', default='/dev/fb0', metavar='/dev/fb*') parser.add_argument('--width', help='screen width [px]', type=int, default=1920, metavar=' INT') parser.add_argument('--height', help='screen height [px]', type=int, default=1080, metavar='INT') cfg = parser.parse_args() # TODO 1: read contents of cfg.src (the frame buffer) # TODO 2: split data in groups of 4 bytes # create a new PIL Image object img = Image.new('RGB', (cfg.width, cfg.height)) px = img.load() # set each pixel value for i in range(cfg.width): for j in range(cfg.height): # TODO 3: write each pixel value in px[i,j] as a RGB tuple # NOTE : the four bytes in the groups that you split previously # are in fact in BGRA format; we don't need the Alpha # value but the other three bytes must be revered pass # save image do disk # NOTE: format will be determined from the file's extension img.save(cfg.FILE, None) # clean up PIL Image object img.close() if __name__ == '__main__': main() The script uses the argparse module to interpret some command-line arguments (the --help flag is implied). If you feel like adding something to these, you're free to do so. It's a good idea to make your script as interactive as possible. The parsed arguments (or their default values) are stored as members of the cfg object. If you're unsure how to access them, just print the whole object once to get a feel for what it contains. For now, go through each TODO in the skeleton and fill in the blanks. Next are a few hints that might help you. Feel free to ask for help if you get stuck :) TODO 1 This step is pretty straightforward. Remember that you are working with a binary file! TODO 2 The result from the previous step will be a byte array. In order to split it into an array of 4-byte RGBA sub-arrays, you can use list comprehension (discussed in this lab. If you have other ideas, you're free to do your own thing. TODO 3 While img is a PIL abstraction of the image you wish to create, individual pixels will be accessed via px, an instance of the PixelAccess class in PIL. Each pixel is represented as a tuple of three integers (i.e.: not bytes). To unpack each value from the byte-array corresponding to your pixel, you can use the struct module. Search in the documentation for the format character representing an unsigned char (equivalent to uint8_t). Try using this binary file as source (–src) when testing your script. It's a copy of a 1920×1080 /dev/fb0 that actually contains relevant data. If your script is rendering this correctly and your resolution is 1920×1080 but the image resulting from your /dev/fb0 is still messed up, note that it might have a different (usually smaller) resolution than your screen. Find out if that's the case: $ cat /sys/class/graphics/fb0/virtual_size

Remember that you don't have to get the script right from your first try. If you're not familiar with a module, experiment with it a bit in a python shell. Or better yet, use ipython.

[10p] Task C - Testing the script

Finally, it's time to test the script. Notice how your script takes the --width and --height optional arguments (with values defaulting to 1920 and 1080). If you have another resolution but you're not sure what that is, it's not hard to find out:

$xrandr --prop | grep primary eDP-1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 344mm x 194mm Note that /dev/fb0 has restricted access permissions for non-owners, so you need to run your script with sudo: $ ls -l /dev/fb0
crw-rw---- 1 root video 29, 0 Apr 14 17:45 /dev/fb0

# grant execute permissions if you haven't already
$chmod +x fb2img.py$ sudo ./fb2img.py screenshot.png

Now let's open the screenshot with the default application:

$xdg-open screenshot.png The result may be a bit unexpected, depending on how your X display server is configured. We'll discuss that in the next exercise :) 02. [60p] Writing the frame buffer Continuing from where we left off in Ex. 1, reading /dev/fb0 may or may not yield the screenshot we expected. If it didn't, the culprit here is most likely the X server, the application that is called upon to draw graphic elements (e.g.: windows) on your screen. When active, Xorg (the X server variant that you're most likely to have installed) places /dev/fb0 under a write lock, meaning that all other processes are prohibited from writing to it. This can be verified with lslock (shows file locks) or lsof (shows opened files). However, if your script did not generate the desired screenshot but in stead some boot-time messages (related to filesystems and whatnot), then /dev/fb0 is probably available for writes. So what gives? How does the X server render your GUI without writing to the frame buffer? The answer is that it's using a frame buffer, but not that frame buffer. /dev/fb0 is the most primitive frame buffer that's available. That's the frame buffer used by BIOS or UEFI, way before your OS is even loaded in RAM. That's the slow frame buffer. If you have a GPU, then the X server is using its frame buffer, and that frame buffer takes precedence over /dev/fb0. # check whether Xorg is using the native frame buffer or the GPU frame buffer in /dev/dri$ sudo lsof | grep -e '/dev/fb[0-9]*' -e '/dev/dri/card[0-9]*' | grep Xorg

So then, how do we write the frame buffer? That was the point of the exercise, remember? If the X server is using /dev/fb0, we can't use it since it's locked. If the X server is using /dev/dri/card0, we can use /dev/fb0 but it's pointless. Well, we have a solution for that, but first we need to understand what is a teletype (TTY).

Back in the old days, there were no displays. Not even CRT monitors. When computers started being fast enough to be considered “interactive”, people wanted to have a way to instantly communicate with them. Their solution was to hook said computers up to a teletype writer (see image above), meaning that the terminal output was actually written on paper! In time, the peripherals that we're used to today came to be and the TTY became an abstraction of the old writers (i.e.: unified I/O devices).

As a result, the TTY today can be considered a virtual device that's not hooked to a mechanical keyboard and a roll of paper but might as well be for all we care. Being virtualized, it means that the OS can provide us with as many as we want. And it does. Right now, you are logged in in tty1. The OS provides a basic driver for all TTYs. This driver can echo keys to the screen when you press them, or go to a new line when you hit enter. All the good stuff… But you're too good for that. You wanted a GUI and decided to use the X server. The X server usurped the clean environment that the OS provided and replaced it with colorful pixels. The terminal you're writing in now is not even a real terminal. It's not a TTY. See for yourself:

# print the name of the terminal that's connected to the standard input
$tty /dev/pts/1 That pts there stands for Pseudo-Terminal Slave. A PTS is a userspace program that tries to emulate a TTY, including its driver. On Ubuntu, that program is most likely gnome-terminal. Other examples include Terminator, Alacritty or Kitty. While the terminal emulator tries to recreate the functionality of the original TTY, it relies on Xorg to render each character in it's assigned window, pixel by pixel. And Xorg does this by hiding the true TTY (i.e.: tty1) by directly accessing the frame buffer (either one of them). While Xorg is still alive, we won't be able to accomplish our goal. So is this the solution? Are we killing Xorg? Not necessarily: we can just move to tty2… Xorg is bound to tty1 by default, but the OS can make up as many TTYs as we want. Let's do that: # switch the system to tty2 (also works with <Ctrl + Alt + F2>)$ sudo chvt 2

# tty2 doesn't know about GPUs; it uses /dev/fb0
# Xorg is sleeping since we're not using tty1
$sudo dd if=/dev/urandom of=/dev/fb0 # chaos ensues # clean up with <Ctrl + L> or$ clear

# return to a more familiar sight (<Ctrl + Alt + F1>)
$sudo chvt 1 If you are working on Ubuntu, note that they keep fucking around with the TTY indices: • version < 17.10: Xorg was running in tty7; tty1-6 were free to use. • version >= 17.10: Xorg is running in tty2 and tty1 is used for login with GUI. Now tty3-7 are free to use. You're most likely using a much newer version than 17.10, but if you want to check: $ cat /etc/lsb-release

The reason why we're discussing only tty1-7 is that they are mapped to your Function keys (F1-F7) and can be accessed by hitting <Ctrl + Alt + Fx>. Your system has a lot more TTYs available:

# yes, all virtual TTYs have an entry in /dev
ls -l /dev/tty[0-9]*

We've just confirmed that while in tty2, we can safely (and successfully) write to /dev/fb0. Moreover, once we switch back to tty1, if Xorg uses the GPU and bypasses the native framebuffer, /dev/fb0 will contain the last image rendered while in tty2. If you're running Linux in a VM this is unlikely, but not an issue. So here's the task:

[60p] Task A - Write img2fb.py

This new script will have to do the exact opposite of fb2img.py. The script will take an image of your choosing and write it to the frame buffer (you know which one). Also, it must take the following arguments:

positional arguments:
FILE            input image file

options:
-h, --help      show this help message and exit
--dst /dev/fb*  data destination
--width  INT    screen width      [px]
--height INT    screen height     [px]
--hoff   INT    horizontal offset [px]
--voff   INT    vertical offset   [px]

The result should look something like this:

Here's an easy way to test your script without constantly switching between tty1 and tty2:

# create a copy of /dev/fb0 in persistent storage & change ownership to yourself
$sudo cp /dev/fb0 frame_buffer.bin$ sudo chown student !$# create a backup with the .bak extension$ cp frame_buffer.bin{,.bak}

# test img2fb.py with the persistent copy as destination
$./img2fb.py --dst frame_buffer.bin nyan_cat.jpg # extract an actual image from the modified copy of the framebuffer$ ./fb2img.py --src frame_buffer.bin does_this_look_good.png

# you screwed up: revert changes and try again
\$ cp frame_buffer.bin{.bak,}

03. [10p] Feedback

Please take a minute to fill in the feedback form for this lab.