05 - Trusted Execution

Objectives

  • Modern software complexity → cyber security issues;
  • Trusted Execution Environments and their application;
  • ARM's trusted boot process & the TrustZone architecture;
  • Install OP-TEE trusted operating system within ARM CPU's secure domain;
  • Build & run trusted applications;

Contents

Lecture

Before beginning the tasks, please check out the lecture slides & notes here.

Tasks

These tasks assume you have a proper boot image + base Linux system working (U-Boot SPL + ARM TrustedFirmware-A + Normal U-Boot, Linux Kernel + Buildroot filesystem).

We will continue writing the build scripts (Makefile-based) to augment our firmware boot package with a freshly compiled OP-TEE (BL32), modify ATF / TF-A (BL31) to load OP-TEE and, finally, write our own Trusted Applications and deploy them on our boards (with the help of our previously bootstrapped Linux, ofc.)!

01. [Re]Configuring & building OP-TEE + ATF

The initial step would be to embed OP-TEE OS into our firmware boot package and re-configure (and recompile) ARM Trusted Firmware-A to load it.

But first, we need to understand the memory organization of our board. This is required since BL31 (TF-A) has to configure the TZASC peripheral to reserve a trusted region inside the main memory (DRAM) for use by our Secure OS & TAs.

Step 1. Memory / address calculations

In the figure to the right, we use the upper end of the DDR1 memory (36 MB) for the TrustZone OS (BL32). We will need to calculate the addresses and supply them as configuration parameters when compiling the software.

In order to get the actual physical addresses, one needs to check out the official NXP iMX93 Reference Manual's Memory Map. We note that the first DRAM (DDR1) is mapped at 0x80000000 (exactly 1GB from the beginning the address space) and ends at 0xFFFFFFFF (the difference of the two addresses totals 2GB).

Next, you must do the math to find out the beginning of the TrustZone space: subtract 32 + 4 = 36 MB = 36 * 1024^2 from the end of the RAM!

Finally, we split the TEE space in two: first 32MB are reserved as secure memory (OP-TEE uses the term TZDRAM to describe this region), the last 4MB are used as shared memory between the TEE and the Secure Monitor (thus, it is called SHMEM).

Write those values out as Makefile variables (use the hexadecimal format for the calculation results, no spaces inside the number since space is the shell argument separator!), e.g.:

TEE_TZDRAM_START = 0xfdc00000
TEE_TZDRAM_SIZE = <32MB as number of bytes, hex is accepted!>
TEE_SHMEM_START = <calc TEE_TZDRAM_START + 32MB, resulting value as hex please>
TEE_SHMEM_SIZE = <4MB in bytes please>
TEE_RAM_TOTAL_SIZE = <36MB as number of bytes>

These will be passed as compilation flags to the OP-TEE + TF-A build scripts (also makefile-based!) and will get inserted as C expressions into their code (thus, you can use valid mathematical expressions, but we recommend against it due to arcane makefile ↔ shell argument passing issues).

(oh, the angular brackets, <…>, are NOT PART OF THE Makefile syntax, they are used as commentary placeholder and should be removed, too!)

Step 2. Download & compile OP-TEE OS

Now we're ready to run some compilation commands!

Clone the NXP fork imx-optee-os repository inside your working directory's root (where you have all other projects like U-Boot, ATF/TF-A, Linux kernel etc.).

Now cd into OP-TEE's freshly downloaded source dir and compile it using GNU Make! Here's the official build documentation: https://optee.readthedocs.io/en/latest/building/gits/optee_os.html.

Well, not really… since the official instructions lack many of the required configuration flags (especially for our platform) and figuring them out requires some time-consuming source code studying, here are some pointers:

  • set the CROSS_COMPILE as in the labs before!
  • use the DEBUG, CFG_TEE_BENCHMARK and CFG_TEE_CORE_LOG_LEVEL values from the official example; set the O=… (output directory) to wherever you like;
  • the PLATFORM can be found exclusively inside the fork's source code! but there's a catch: the platform prefix must be imx-, continued by model name without the leading i (check some examples here).
  • set the CFG_TZDRAM_START, CFG_TZDRAM_SIZE, CFG_TEE_SHMEM_START, CFG_TEE_SHMEM_SIZE configuration flags to the values calculated above (reminder: you can reference another Makefile variable using $(VARIABLE_NAME) syntax);
  • finally, you will also need to pass CFG_DDR_SIZE=0x80000000 (yep, that's 2GB, our board's actual installed memory).

There is currently a bug inside OP-TEE that makes it unable to boot: it uses the NXP EdgeLock Enclave for TRNG generation, whose firmware does not start (and we weren't able to figure it out since its documentation is “secret” / available using NDA-only).

As workaround, we can disable the ELE RNG driver and enable a software one using the following additional make configuration options: CFG_WITH_SOFTWARE_PRNG=y CFG_IMX_ELE=n. So make sure to also set them!

After a successful build, check the output (O) directory's core subdirectory for the tee.bin, tee-raw.bin and many other files! We will keep the raw one for later inclusion into the firmware package using mkimage ;)

Don't forget to save the build commands inside your Makefile script! You should have seen its usefulness by now ;)

Step 3. Recompile ARM Trusted Firmware-A with added flags

Remember the TF-A component (BL31) from Lab 1? We need to re-configure + rebuild it (hopefully, you still got the Makefile script)!

This should be straightforward, since you've done it before ;) You need to set the following additional configuration variables to its make invocation:

  • SPD=opteed – this is the Secure Payload Dispatcher module, aka: who does ATF's Secure Monitor need to talk with? Our OP-TEE, of course!
  • ATF also has to know the memory region where we've put OP-TEE (set the BL32_BASE variable to lower limit, in hexadecimal); we also need to specify its total size: BL32_SIZE (remember: we allocated 36MB, but give it in bytes, either in base 10 or 16 using C integer notation);
  • We want ATF to print some debug messages over the first serial peripheral so, finally, set the LOG_LEVEL=40 and IMX_BOOT_UART_BASE=0x44380000 (if you look in iMX93's Memory Map (the Reference Manual), this is the physical address of our LPUART1 - i.e. our serial communication module!).
Step 4. Rebuild the firmware image package

Before we can take a look at the fruits of our effort so far, we need to re-build the firmware package with these last two components.

This is easy if you saved your mkimage script (in your Makefile). You will need JUST one additional file copied: tee-raw.bin from optee build output directory, core subdirectory (as mentioned in the subtask above!) inside mkimage's build directory, but rename it as tee.bin. The bundled scripts will see that this file exists and add it to the image automatically!

One more thing: when calling the imx-mkimage's script, set TEE_LOAD_ADDR=<TrustZone DRAM start address>.

The default value is wrong for our SoC.

Also make sure to replace bl31.bin with the newly recompiled one from ARM Trusted Firmware-A!

Proceed to load this image over the serial boot protocol using the IMX uuu utility. Check the serial console (the one provided by the board's DEBUG USB port) for confirmation!

Example UART output

Example UART output

Found header at 0x404089a0
# Now, this is some output from ATF / BL31; never mind the warnings :D
D/TC:0   plat_get_aslr_seed:118 Warning: no ASLR seed
D/TC:0   add_phys_mem:665 VCORE_UNPG_RX_PA type TEE_RAM_RX 0xbdc00000 size 0x000cb000
D/TC:0   add_phys_mem:665 VCORE_UNPG_RW_PA type TEE_RAM_RW 0xbdccb000 size 0x00135000
D/TC:0   add_phys_mem:665 ta_base type TA_RAM 0xbde00000 size 0x01e00000
D/TC:0   add_phys_mem:665 CONSOLE_UART_BASE type IO_NSEC 0x30800000 size 0x00400000
...
# aaaaanddd our main dish, OP-TEE (BL32):
I/TC: OP-TEE version: 3.22.0-12-ga012b9923 (gcc version 13.1.0 (GCC)) #1 Fri Jul 14 19:24:34 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
D/TC:0 0 boot_init_primary_late:1478 Executing at offset 0 with virtual load address 0xbdc00000
...
I/TC: Primary CPU switching to normal world boot
...
# BL33 (Normal U-Boot loads afterwards, as expected)
U-Boot SPL 2024.04-dirty (Aug 06 2025 - 16:19:37 +0300)
SOC: 0xa1009300
LC: 0x2040010
...

02. Writing Trusted Applications

In the second part of our lab, it's time to run some Trusted Applications (TAs)!

We will use the official optee_examples as starting point.

Two questions arise: how can one compile a TA? + how to test it on our board?

Step 5. Not so fast... we forgot about the kernel?

If we wish to communicate with OP-TEE from Linux, we need to configure its driver (don't worry, we don't need to recompile the kernel, it's included in defconfig').

Of course, the intended way to do that is by modifying the Linux Device Tree. Do it as documented here ;)

In case you need a little reminder, the device tree source code is inside linux's arch/arm64/.

We promised that you won't need to recompile the kernel, you just need to recompile its DTB ;) use:

make ARCH=... dtbs

Oh, and re-enable Buildroot if disabled for the previous lab (you backed up your ITS file, hopefully). Then enter its menuconfig and search for PACKAGE_OPTEE_CLIENT. Enable it and rebuild your rootfs!

Afterwards, you need to copy the new .dtb into your staging/ directory and re-build the Linux FIT (e.g., linux.itb). And upload it to your emmc (use u-boot's ums and simply copy the file on the FAT32 boot partition).

Step 6. Building a TA

We can use our workstation / laptop to cross-compile a trusted application!

Read the official instructions here.

Build both the host app and the Trusted Application need to be compiled TOGETHER with the TEE Client Library, and the OPTEE OS exported SDK, respectively.

Make sure to read the examples documentation to see the make variables to set!

Note that, usually, you must build optee-client from source on the host machine the optee_client first. .

But, since we're using Buildroot and have enabled it, you can find it already compiled for the target system at <buildroot-dir>/output/build/optee-client-<version>.

Touugh we need to give a TEEC_EXPORT install path when invoking the TA makefile… Use find for tee_client_api.h and see where it's found (hint: sysroot)!

As for TA_DEV_KIT_DIR of a Trusted Application, it must point to an SDK generated inside BL32 (optee_os) source directory (something like export-ta…'.

Also note that for building Buildroot-targeted applications, your classic aarch64-none-gnu- toolchain won't work since Buildroot uses custom ucLibC by default. But, fortunately, you may find the cross compilation prefix at <buildroot-dir>/output/host/bin/aarch64-buildroot-linux-gnu- (see the difference?)!

Step 7. Signing the TA

Remember the secure boot process? The Secure OS (OP-TEE) will also verify each TA before being able to load it!

Fortunately, OP-TEE comes with a predefined key pair, used to facilitate development / testing (warning: highly insecure since anybody can retrieve that key from the source repository!).

Read on for the TA signing procedure!

Even better: the TA Makefile automatically signs the .ta using the development key. So we won't need to do anything more! Yay…

Step 8. Testing the TA

Our rootfs image already contains the OP-TEE client library.

You just need to copy to the cross-compiled binaries to your boot partition.

For this, put u-boot in USB Mass Storage mode:

u-boot=> ums mmc 0

Do not Ctrl+C yet, leave it running and mount the newly appeared USB device in your PC/VM!

After copying the files, boot Linux.

Now check if tee-supplicant is running… let's mount devtmpfs then start it:

mount -t devtmpfs devtmpfs /dev
tee-supplicant -d

Mount the boot partition and run the TA (you might need to copy it somewhere else and chmod +x)! Then execute the program ;)

Observe the error: OP-TEE cannot find the .ta file inside a trusted memory or REE.

For this, you will need to copy the signed <UUID>.ta file to /lib/optee_armtz/ (create it if it doesn't exist), as (very badly) documented.

ass/labs-2025/05.txt · Last modified: 2025/08/07 22:15 by florin.stancu
CC Attribution-Share Alike 3.0 Unported
www.chimeric.de Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0