• Modern software complexity → cyuber security issues;
• Trusted Execution Environments and their application;
• ARM's trusted boot process & the TrustZone architecture;
• Install OP-TEE trusted operating system within ARM CPU's secure domain;
• Build & run trusted applications;

In the cyber security landscape, as you well know, any piece of software may contain various security vulnerabilities (usually, due to its developers' negligence), which may allow a malicious entity to obtain execution privileges on the device.

Though the hacker might gain limited permissions at first (due to the application running as an unprivileged user), a computer bug is never alone in a system and, ultimately, the attacker may be able to circumvent such restrictions by doing (and chaining) additional privilege escalation exploits. The ideal end-game would be to obtain kernel-level execution rights and inject a hidden persistent malware to covertly keep the device under malicious control (e.g., for financial / political gains).

Increased complexity often encountered in modern software is regarded as the primary reason for the apparition of bugs, since having a many number of lines of code makes any verification process considerably harder. Due to this, modern security practices recommend organizing the system's architecture into smaller, standalone components, thus ensuring better security properties and overall good quality of a software (also see: Unix Philosophy).

From here, we can go even further: some parts of a system might be more security-critical than others (cyber-phisical interfaces, databases storing sensitive secrets etc.); modern hardware might allow us to ensure additional protections for these such that, even when the Operating System kernel gets breached, an attacker's access remains confined by the processor and it won't be able to cause system-wide mayhem. Enter Trusted Execution Environments!

A Trusted Execution Environment (TEE) is an isolated execution context whose internal state (memory, CPU registers etc.) is secured and cannot be reached by normal software. This means untrusted applications, and even the highly-privileged Operating System (often considered as being vulnerable to cyber attacks due to its huge complexity) cannot read / write a TEE's protected zones and cannot access secure peripherals (e.g., keypads for sensitive input, protected screens).

Trusted Execution Environments are usually implemented by a combined hardware + software approach: the CPU architecture must be extended to discern normal vs. secure execution and deny requests to secure memory addresses, while some kind of trusted firmware is used to retain an appropriate level of flexibility for a highly secure solution (for increased compatibility with various hardware / peripherals, enforce application-specific rules, implement platform upgradeability - especially in the case of bug disclosures).

Note that a TEE subsystem should follow the minimal complexity rule discussed above. This means that the developers must design a modular architecture and partition its code into trusted (secure) and untrusted counterparts. The trusted application must the only one with access to manipulate sensitive data (e.g., passwords / encryption keys, confidential information, cyber-physical commands etc.); the untrusted components (including the traditional / feature-rich Operating System) will only see such information travel in encrypted & integrity-authenticated (protected from modifications) form and will only serve as an intermmediary for the TAs to communicate with external entities (e.g., human-machine interface peripherals, networks, persistent storage) – practically, it should do anything that does not need to directly process / understand any of it.

Some of the commercially available TEE technologies are:

• Intel Software Guard Extensions (SGX) (for Intel's mainstream CPUs, which was announced as deprecated);
• ARM TrustZone (available for all modern ARM-based embedded / mobile platform and our primary focus for this lab)
• AMD Secure Encrypted Virtualization (for server CPUs / cloud applications);
• Intel Trust Domain Extensions (successor of SGX, but similar to AMD SEV).

On a final note: unfortunately, consumer-friendly TEE solutions for personal computing (i.e., desktops & laptops) remain the domain of ongoing research and no such technologies are available on mainstream (user-centric) CPUs.

Modern ARM CPU architectures feature the TrustZone Security Extensions (starting with armv7 for ARM Cortex-A, expanding to Cortex-Ms after v8), which introduces the necessary execution context separation required for implementing TEEs.

It realized this by using a new CPU state bit (NS - Non-Secure - flag) create an additional privilege level - orthogonal to the original Exception Levels (EL0-2), meaning both Secure and Non-Secure Worlds can have software running in each of these common ELs. An additional Exception Level, EL3 (Firmware / Secure Monitor), was also introduced to securely manage the software transition between the two trust domains.

Furthermore, the ARM AXI (Advanced eXtensible Interface) bus protocol was also extended to tag all memory transactions with a NS flag, which the on-chip modules (e.g., SRAM / DRAM) and other peripherals can check against and allow / deny access to specific resources. Devices using this kind of access control are called TrustZone-aware peripherals. For example, interfaces used for connecting fingerprint readers and NFCs used for banking applications could be configured to block unauthorized access from untrusted applications (modern iPhones smartphones already do this!).

We highlight, in particular, ARM's official TZASC (TrustZone Address Space Controller) module, a SoC (System-on-Chip)-integrated peripheral which filters all requests to the secure memory regions (e.g., Secure SRAM / TZDRAM) mapped the physical address space of a chip. The TZASC gets configured early in the boot process by the ARM Trusted Firmware (executing as BL31) before loading any Normal World components.

Note that the two domains (Secure & Non-Secure Worlds) can communicate to each other via messages (using a buffer in unprotected shared memory) relayed by the Secure Monitor (solely running at the most privileged level: EL3, provided by ARM Trusted Firmware-A for our boards - as we will see at the lab) when the special SMC instruction is invoked (very similar to the traditional userspace to kernel system call mechanism). This may the case either when a normal application wishes to request secure services from its Trusted counterpart (e.g., request to have a piece of data decrypted on behalf of an authorized user) or the reverse: a TA requests a untrusted function typically implemented by the Operating System (filesystem / network / untrusted hardware access - almost anything for which the userspace software or drivers are really complex and should not incur additional overhead to the secure components).

OP-TEE (Open Trusted Execution Environment) is a open-source project which aims to implement a complete Secure Operating System to run inside ARM TrustZone-enabled Secure World and facilitate the development of Trusted Applications (TAs).

OP-TEE's main components are:

• A secure kernel (OP-TEE OS), executing in EL1 privileged mode;
• A set of secure libraries (mainly for cryptographic needs) and scripts to ease the development of Trusted Applications (TA);
• Linux (Normal World) libraries for exchanging messages with a TA;
• Linux kernel TEE driver (mainline) to communicate with the Secure World using shared memory pages;
• A Linux TEE supplicant daemon to provide Rich OS services to the TAs (e.g., network / filesystem access).

These tasks assume you have a proper boot image + base Linux system working (U-Boot SPL + ARM TrustedFirmware-A + Normal U-Boot, Linux Kernel + Buildroot filesystem).

We will continue writing the build scripts (Makefile-based) to augment our firmware boot package with a freshly compiled OP-TEE (BL32), modify ATF / TF-A (BL31) to load OP-TEE and, finally, write our own Trusted Applications and deploy them on our boards (with the help of our previously bootstrapped Linux, ofc.)!

The initial step would be to embed OP-TEE OS into our firmware boot package and re-configure (and recompile) ARM Trusted Firmware-A to load it.

But first, we need to understand the memory organization of our board. This is required since BL31 (TF-A) has to configure the TZASC peripheral to reserve a trusted region inside the main memory (DRAM) for use by our Secure OS & TAs.

In the figure to the right, we use the upper end of the DDR1 memory (36 MB) for the TrustZone OS (BL32). We will need to calculate the addresses and supply them as configuration parameters when compiling the software. TODO: add figure with i.MX8 TZ memory map.

In order to get the actual physical addresses, one needs to check out the official NXP iMX8MQ Reference Manual, Chapter 2.1.2 (Cortex-A53 Memory Map, pg. 20). We note that the first DRAM (DDR1) is mapped at 0x40000000 (exactly 1GB from the beginning the address space) and ends at 0xFFFFFFFF (the difference of the two addresses totals to 3072MB == 3 * 1024MB == 3GB).

But, since our boards only come with 2GB of LPDDR chips installed, we need to recalculate the end of our DRAM memory region: DRAM_END = 0x40000000 + 2*1024^3 == 0xC0000000 (in hex)).

Next, you must do the math to find out the beginning of the TrustZone space: subtract 32 + 4 = 36 MB = 36 * 1024^2 from the end of the RAM!

Finally, we split the TEE space in two: first 32MB are reserved as secure memory (OP-TEE uses the term TZDRAM to describe this region), the last 4MB are used as shared memory between the TEE and the Secure Monitor (thus, it is called SHMEM).

Write those values out as Makefile variables (use the hexadecimal format for the calculation results, no spaces inside the number since space is the shell argument separator!), e.g.:

TEE_TZDRAM_START = 0xbdc00000
TEE_TZDRAM_SIZE = <32MB as number of bytes, hex is accepted!>
TEE_SHMEM_START = <calc TEE_TZDRAM_START + 32MB, resulting value as hex please>
TEE_SHMEM_SIZE = <4MB in bytes please>
TEE_RAM_TOTAL_SIZE = <36MB as number of bytes>

These will be passed as compilation flags to the OP-TEE + TF-A build scripts (also makefile-based!) and will get inserted as C expressions into their code (thus, you can use valid mathematical expressions, but we recommend against it due to arcane makefile ↔ shell argument passing issues).

(oh, the angular brackets, <…>, are NOT PART OF THE Makefile syntax, they are used as commentary placeholder and should be removed, too!)

Now we're ready to run some compilation commands!

Clone the official OP-TEE OS repository inside your working directory's root (where you have all other projects like U-Boot, ATF/TF-A, Linux kernel etc.). Suggested directory name: optee_os (well… git automatically implies it, since it's also the name of the project).

Now cd into OP-TEE's freshly downloaded source dir and compile it using GNU Make! Here's the official build documentation: https://optee.readthedocs.io/en/latest/building/gits/optee_os.html.

Well, not really… since the official instructions lack many of the required configuration flags (especially for our platform) and figuring them out requires some time-consuming source code studying, here are some pointers:

• set the CROSS_COMPILE as in the labs before!
• use the DEBUG, CFG_TEE_BENCHMARK and CFG_TEE_CORE_LOG_LEVEL values from the official example; set the O=… (output directory) to wherever you like;
• the PLATFORM is imx-mx8mqevk; well, not really, but it's the closest one available, we start from here and override some of its configuration defaults, see next vars ;)
• set the CFG_TZDRAM_START, CFG_TZDRAM_SIZE, CFG_TEE_SHMEM_START, CFG_TEE_SHMEM_SIZE configuration flags to the values calculated above (reminder: you can reference another Makefile variable using $(VARIABLE_NAME) syntax);
• finally, you will also need to pass CFG_DDR_SIZE=0x80000000 (yep, that's 2GB, our board's actual installed memory).

After a successful build, check the output (O) directory's core subdirectory for the tee.bin, tee-raw.bin and many other files! We will keep the raw one for later inclusion into the firmware package using mkimage ;)

Remember the TF-A component (BL31) from Lab 1? We need to re-configure + rebuild it (hopefully, you still got the Makefile script)!

This should be straightforward, since you've done it before ;) You need to set the following additional configuration variables to its make invocation:

• SPD=opteed – this is the Secure Payload Dispatcher module, aka: who does ATF's Secure Monitor need to talk with? Our OP-TEE, of course!
• ATF also has to know the memory region where we've put OP-TEE (set the BL32_BASE variable to lower limit, in hexadecimal); we also need to specify its total size: BL32_SIZE (remember: we allocated 36MB, but give it in bytes, either in base 10 or 16 using C integer notation);
• We want ATF to print some debug messages over the first serial peripheral so, finally, set the LOG_LEVEL=40 and IMX_BOOT_UART_BASE=0x30860000 (if you look in iMX8MQ's Memory Map (the Reference Manual), this is the physical address of our Universal Asynchronous Transmitter/Receiver - i.e. our serial communication module!).

Before we can take a look at the fruits of our effort so far, we need to re-build the firmware package with these last two components.

This is easy if you saved your mkimage script (in your Makefile). You will need JUST one additional file copied: tee-raw.bin from optee_os build output directory, core subdirectory (as mentioned in the subtask above!) inside mkimage's build directory, but rename it as tee.bin. The bundled scripts will see that this file exists and add it to the image automatically!

Also make sure to replace bl31.bin with the newly recompiled one from ARM Trusted Firmware-A!

Proceed to load this image over the serial boot protocol using the IMX uuu utility. Check the serial console (the one provided by the board's Micro USB port) for confirmation!

Found header at 0x404089a0
# Now, this is some output from ATF / BL31; never mind the warnings :D
D/TC:0   plat_get_aslr_seed:118 Warning: no ASLR seed
D/TC:0   add_phys_mem:665 VCORE_UNPG_RX_PA type TEE_RAM_RX 0xbdc00000 size 0x000cb000
D/TC:0   add_phys_mem:665 VCORE_UNPG_RW_PA type TEE_RAM_RW 0xbdccb000 size 0x00135000
D/TC:0   add_phys_mem:665 ta_base type TA_RAM 0xbde00000 size 0x01e00000
D/TC:0   add_phys_mem:665 CONSOLE_UART_BASE type IO_NSEC 0x30800000 size 0x00400000
...
# aaaaanddd our main dish, OP-TEE (BL32):
I/TC: OP-TEE version: 3.22.0-12-ga012b9923 (gcc version 13.1.0 (GCC)) #1 Fri Jul 14 19:24:34 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
D/TC:0 0 boot_init_primary_late:1478 Executing at offset 0 with virtual load address 0xbdc00000
...
I/TC: Primary CPU switching to normal world boot
...
# BL33 (Normal U-Boot loads afterwards, as expected)
U-Boot 2022.04-g1f940d6213 (Jul 14 2023 - 22:22:25 +0300)

CPU:   i.MX8MQ rev2.1 1300 MHz (running at 800 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 52C
...

In the second part of our lab, it's time to do some coding (disclaimer: this may sound fun, but it won't quite be like it :(( :P )!