OP-TEE on the Raspberry Pi 5

This repository shows how to run OP-TEE on the Raspberry Pi 5.

Caution

This is purely for educational purposes. The Raspberry Pi does not support secure memory, that can be protected from the outside world1.

Outline

OP-TEE supports the RPi 3 and there is a port for the RPi 5 here that I just found after porting it myself.

To load and initialize OP-TEE we use ARM Trusted Firmware that officially supports the RPi 5. There exists a abandoned patch to add OP-TEE support for the RPi 5 on which tutorial is loosely based on here.

To build the image that contains ARM Trusted Firmware with OP-TEE and the Linux system with use buildroot. We will build a simple system that contains everything required to run OP-TEE applications from Linux.

This tutorial will go through all steps of getting, patching and building the whole system to run the OP-TEE hello world application on the RPi 5. It is also loosely based on this tutorial here.

I’m using Ubuntu 24.04 on x86.

Note

To make clear in which directory commands are executed export BASE_DIR to the directory you are working in.

export BASE_DIR=$(pwd)

Toolchain

To cross compile everything for ARM we need compilers. On x86 Linux this is the correct toolchain: https://developer.arm.com/-/media/Files/downloads/gnu/14.3.rel1/binrel/arm-gnu-toolchain-14.3.rel1-x86_64-aarch64-none-linux-gnu.tar.xz. Find all available toolchains here.

Extract the archive and export the path to the binaries.

cd $BASE_DIR
wget https://developer.arm.com/-/media/Files/downloads/gnu/14.3.rel1/binrel/arm-gnu-toolchain-14.3.rel1-x86_64-aarch64-none-linux-gnu.tar.xz
tar -xvf arm-gnu-toolchain-14.3.rel1-x86_64-aarch64-none-linux-gnu.tar.xz
rm arm-gnu-toolchain-14.3.rel1-x86_64-aarch64-none-linux-gnu.tar.xz
mv arm-gnu-toolchain-14.3.rel1-x86_64-aarch64-none-linux-gnu arm-toolchain

Important

Finally we export CROSS_COMPILE so that all following makefiles will use this toolchain.

export CROSS_COMPILE=$(pwd)/arm-toolchain/bin/aarch64-none-linux-gnu-

OP-TEE

We will start by building OP-TEE OS (running in the secure world), the client library and the hello world example application.

OP-TEE OS

Prerequisites

First install the prerequisites to build OP-TEE1.

# apt-get install -y adb acpica-tools autoconf automake bc bison build-essential ccache cpio cscope curl device-tree-compiler e2tools expect flex ftp-upload gdisk git libgnutls28-dev libattr1-dev libcap-ng-dev libfdt-dev libftdi-dev libglib2.0-dev libgmp3-dev libhidapi-dev libmpc-dev libncurses5-dev libpixman-1-dev libslirp-dev libssl-dev libtool libusb-1.0-0-dev make mtools ninja-build python3-cryptography python3-pip python3-pyelftools python3-serial python3-tomli python-is-python3 rsync swig unzip uuid-dev wget xdg-utils xsltproc xterm xz-utils zlib1g-dev

Clone

Clone the OP-TEE repository

cd $BASE_DIR
git clone git@github.com:OP-TEE/optee_os.git
cd optee_os

This tutorial is based on commit c2c23cd49c9f.

git checkout c2c23cd49c9f

Patch

Apply the patches in patches/optee_os.diff.

git apply ../patches/optee_os.diff

The largest difference between the RPi 5 and RPi 3 ports is that the RPi 5 does no longer have a mini UART interface, only PL011s. We therefore have to pl011_init the UART10 at the RPi 5’s debug port at memory address 0x107d001000.

Additionally, it is important to configure $(call force,CFG_AUTO_MAX_PA_BITS,y). Per default, OP-TEE uses only 36 physical address bits to address 32GB of physical memory. However, as you already noticed, the UART address 0x107d001000 is outside this range just barely. This took me quite some time to figure out.

Build

Now build the OP-TEE OS using make.

make PLATFORM=rpi5 \
CFG_ARM64_core=y \
CFG_USER_TA_TARGETS=ta_arm64 \
CFG_DT=y \
CFG_TEE_CORE_DEBUG=y \
CFG_TEE_CORE_LOG_LEVEL=4 \
CFG_CORE_ASLR=n

We build the debug target with hightest log level and disable ASLR because we are not secure anyways.

If successful, the build outputs the file out/arm-plat-rpi5/core/tee-raw.bin, which is the raw binary containing the secure OS.

Important

We also get the “dev-kit” that we need to export to build our example application.

export TA_DEV_KIT_DIR=$(pwd)/out/arm-plat-rpi5/export-ta_arm64

OP-TEE Client

With the OP-TEE OS done, we also need the client library.

Clone

cd $BASE_DIR
git clone https://github.com/OP-TEE/optee_client
cd optee_client

This tutorial is based on commit 4.8.0-rc1.

git checkout 4.8.0-rc1

Build

I was not able to build the library with libteeacl. The compiler and linker could not find libuuid, although it was installed. Libteealc provides helpers for an alternative login method initially used when accessing objects in Op-TEE’s PKCS11 TA2.

make WITH_TEEACL=0 -j

Important

If successful, the build outputs include and library files to out/export/usr/ which we need to export to build our example application.

export TEEC_EXPORT=$(pwd)/out/export/usr

OP-TEE Hello World

OP-TEE has a repository with various example applications.

Clone

cd $BASE_DIR
git clone https://github.com/linaro-swg/optee_examples.git
cd optee_examples

This tutorial is based on commit 4.8.0-rc1.

git checkout 4.8.0-rc1

Build

make

If successful, the build outputs hello_world/host/optee_example_hello_world, the application running in the unsecure world and hello_world/ta/8aaaf200-2450-11e4-abe2-0002a5d5c51b.ta, the application running in the secure world in OP-TEE.

This concludes everything OP-TEE.

ARM Trusted Firmware

The next step is to build the ARM Trusted Firmware that loads and initializes OP-TEE and then starts the Linux kernel. ARM Trusted Firmware officially supports the RPi 5 but not OP-TEE on the RPi 5, see here for more a detailed build and usage description. There is, however, an abandoned patch to add OP-TEE support for the RPi 5 here.

Clone

cd $BASE_DIR
git clone git@github.com:ARM-software/arm-trusted-firmware.git
cd arm-trusted-firmware

This tutorial is based on commit f74d03a12eb0.

git checkout f74d03a12eb0

Patch

Apply the patch in patches/armtfw.diff

git apply ../patches/armtfw.diff

Build

make PLAT=rpi5 SPD=opteed DEBUG=1

If successful, the build outputs build/rpi5/debug/bl31.bin.

Combining ARM Trusted FW and OP-TEE OS

Important

Because the RPi 5 can only load a single binary into the memory, we have to combine the ARM trusted FW and OP-TEE OS binary.

In plat/rpi/common/rpi4_bl31_setup.c:135 the following lines then copy the OP-TEE OS binary to the location defined by BL32_BASE and fills bl32_image_ep_info with the required information to jump to it later.

VERBOSE("rpi: Moving OP-TEE Image to %x\n", BL32_BASE);
memcpy((void*)BL32_BASE, (void*)RPI_OPTEE_IMAGE_BASE,
RPI_OPTEE_IMAGE_SIZE);
bl32_image_ep_info.pc = BL32_BASE;
bl32_image_ep_info.spsr = rpi3_get_spsr_for_bl33_entry();
bl32_image_ep_info.args.arg3 = rpi4_get_dtb_address();
SET_SECURITY_STATE(bl33_image_ep_info.h.attr, SECURE);

To concatinate the OP-TEE OS binary at the correct location we run

cd $BASE_DIR
cp arm-trusted-firmware/build/rpi5/debug/bl31.bin bl31_bl32.bin
dd if=optee_os/out/arm-plat-rpi5/core/tee-raw.bin of=bl31_bl32.bin bs=1024 seek=128

The bl31_bl32.bin will then be the first binary loaded by the VPU and executed by the RPi5 CPU.

Linux OS

We use buildroot to build our system that glues everything together.

Clone

cd $BASE_DIR
git clone git://git.buildroot.net/buildroot
cd buildroot

Patch

Apply the patch in patches/buildroot.diff. It contains changes to the post-image.sh build script and adds an device tree overlay file for the TEE.

git apply ../patches/buildroot.diff

Configuration

Buildroot

Load OPTEE Config

make raspberrypi5_optee_defconfig

This should output configuration written to /home/jonas/Entwicklung/TrustZone/clean/buildroot/.config

Note

The optee default config already sets all the configuration changes below. So nothing must be done here. The Linux kernel must be configured manually!

Root Password

To be able to login over ssh a password for the root user must be set.

System Configuration ==> Root password

I also changed system hostname and banner in this menu.

Shell UART

ARM Trusted FW, OP-TEE OS and the Linux kernel are configures to use UART10 at the debug port. With this configuration, the shell is running on UART0 on GPIO pins 14 and 15. This required two UART to USB adapters but allows to see the shell and logs in to different windows.

System Configuration => Run a getty (login prompt) after boot
"TTY port" = "ttyAMA0"
"baudrate" = "115200"

BR2_TARGET_GENERIC_GETTY_PORT=ttyAMA0
BR2_TARGET_GENERIC_GETTY_BAUDRATE_115200=y

Install DTB Overlay

My system was not booting because per default, no device tree overlays are installed and therefore /overlays/bcm2712d0.dtbo was missing.

Target packages => Hardware handling => Firmware => rpi-firmware
Set boot/config.txt to "board/raspberrypi5/config_optee.txt"
Set boot/cmdline.txt to "board/raspberrypi5/cmdline_optee.txt"

Enable "Install DTB overlays"

BR2_PACKAGE_RPI_FIRMWARE_CONFIG_FILE=board/raspberrypi5/config_optee.txt
BR2_PACKAGE_RPI_FIRMWARE_CMDLINE_FILE=board/raspberrypi5/cmdline_optee.txt
BR2_PACKAGE_RPI_FIRMWARE_INSTALL_DTB_OVERLAYS=y

OP-TEE Client

To support OP-TEE we need the optee-client package on the system.

Target packages => Security
Enable "optee-client"

BR2_PACKAGE_OPTEE_CLIENT=y
BR2_PACKAGE_OPTEE_CLIENT_TEE_FS_PATH=/data/tee
BR2_PACKAGE_OPTEE_CLIENT_RPMB_EMU=y
BR2_PACKAGE_OPTEE_CLIENT_SUPP_PLUGINS=y

Network Access

We want to be able to login to the system through ssh. This requires a few packages

Target packages => Networking Applications
Enable "dhcpcd" and "dropbear"

BR2_PACKAGE_DHCPCD=y
BR2_PACKAGE_DROPBEAR=y

Enable Compression

Filesystem images => ext2/3/4 root filesystem
"Compression method" = "bzip2"

Enable "tar the root filesystem"
"Compression method" = "bzip2"

BR2_TARGET_ROOTFS_EXT2_BZIP2=y
BR2_TARGET_ROOTFS_TAR=y
BR2_TARGET_ROOTFS_TAR_BZIP2=y

Linux

Warning

This is NOT automatically done by the patch.

make linux-menuconfig

Trusted Execution Environment support

We also have to enable the drivers in the Linux kernel to support TEEs.

Device Drivers => Trusted Execution Environment support (almost at the bottom)
Enable OP-TEE

TEE=y
OPTEE=y

OP-TEE Overlay

For the TEE driver to know that there is a TEE in the system we need to add a custom device tree overlay. The overlay is in board/raspberrypi5/overlay/optee.dts

/dts-v1/;
/plugin/;

/ {
compatible = "brcm,bcm2712", "raspberrypi,5-model-b";

fragment@0 {
target-path = "/";
__overlay__ {
optee: optee {
compatible = "linaro,optee-tz";
method = "smc";
};
};
};
};

You can manually compile it with

dtc -@ -I dts -O dtb -o optee.dtbo optee.dts

However, the post-image-optee.sh script automatically compiles it and adds it to the image.

Add OP-TEE Applications

To have the OP-TEE applications directly in the generated image we add them to a rootfs overlay using the script copy_optee_examples.sh.

cd $BASE_DIR
copy_optee_examples.sh
cd buildroot

Build

Now build the SD card image:

Important

bl31_bl32.bin must be in $BASE_DIR

make -j$(nproc)

This will take around 15 minutes, depending on your CPU.

Flash

To flash the disk image onto the SD card use dd.

sudo dd status=progress if=$BASE_DIR/buildroot/output/images/sdcard.img of=/dev/SDCARD
sync

Caution

Pay great attention to write the image to the correct block device!

Debug Connector

To connect the debug connector one can use an adapter cable or the official debug probe. It is a JST SH 1.0mm 3-pin connector and the pinout can be found here.

I was very successful with simply sticking two slightly little thicker cables into the connector. The violet cable is connected to the RX pin of the UART-to-USB converter, the green cable to the TX pin. Ground must also be connected and can be connected with GPIO header.

This is a picture showing all connections.

Boot

Put the SD card into the Raspberry Pi 5 and start it. A full boot log of a successful boot is in full-boot.log.

Run OP-TEE Application

When booting was successful, the login appears on the second UART connection. This is how a successful run of the hello world applications looks like:

Welcome to Raspberry Pi with OPTEE
opteepi login: root
Password:
# optee_example_hello_world
Invoking TA to increment 42
TA incremented value to 43
# poweroff

Appendix

Exports

These are the exports used for the builds

BASE_DIR=/path/to/your/working/directory
CROSS_COMPILE=$BASE_DIR/arm-toolchain/bin/aarch64-none-linux-gnu-
TA_DEV_KIT_DIR=$BASE_DIR/optee_os/out/arm-plat-rpi5/export-ta_arm64
TEEC_EXPORT=$BASE_DIR/optee_client/out/export/usr

No bcm2712d0.dtbo Overlay

This is the error I got when the bcm2712d0.dtbo overlay file was not present. Even though according to the buildroot RPi 5 configuration it is not required:

[    1.500456] bcm2708_fb soc@107c000000:fb: Unable to determine number of FBs. Disabling driver.
[    1.509109] bcm2708_fb soc@107c000000:fb: probe with driver bcm2708_fb failed with error -2
[    1.545988] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[    1.552433] SError Interrupt on CPU0, code 0x00000000be000011 -- SError
[    1.552437] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.12.41-v8-16k #1
[    1.552439] Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
[    1.552440] pstate: 804000c9 (Nzcv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    1.552443] pc : bcm2712_pull_config_set+0x64/0xd8
[    1.552449] lr : bcm2712_pull_config_set+0x48/0xd8
[    1.552451] sp : ffffffc080c73ad0
[    1.552452] x29: ffffffc080c73ad0 x28: 0000000000000000 x27: ffffffd08d6d0118
[    1.552455] x26: ffffffd08d789060 x25: 0000000000000006 x24: ffffff8040982358
[    1.552458] x23: 0000000000000018 x22: ffffff80029ed940 x21: 0000000000000122
[    1.552460] x20: ffffff80029ed880 x19: 0000000000000000 x18: ffffff8040982348
[    1.552462] x17: 0000000000000000 x16: 0000000000000000 x15: ffffffffffffffff
[    1.552464] x14: ffffffc080c73800 x13: ffffff804098234a x12: 3030306330356437
[    1.552466] x11: 0000000000000040 x10: ffffff80026446e0 x9 : ffffffd08cbc3e18
[    1.552468] x8 : ffffff80c0473258 x7 : 0000000000000000 x6 : 0000000000000000
[    1.552470] x5 : ffffff80c0473230 x4 : 0000000000000024 x3 : 000000000000ffff
[    1.552472] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
[    1.552475] Kernel panic - not syncing: Asynchronous SError Interrupt
[    1.552476] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.12.41-v8-16k #1
[    1.552478] Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
[    1.552479] Call trace:
[    1.552480]  dump_backtrace.part.0+0xe0/0x100
[    1.552486]  show_stack+0x20/0x40
[    1.552487]  dump_stack_lvl+0x60/0x80
[    1.552492]  dump_stack+0x18/0x28
[    1.552494]  panic+0x170/0x370
[    1.552497]  nmi_panic+0x90/0x98
[    1.552499]  arm64_serror_panic+0x6c/0x90
[    1.552501]  arm64_is_fatal_ras_serror+0x44/0xc0
[    1.552503]  do_serror+0x64/0x80
[    1.552504]  el1h_64_error_handler+0x30/0x48
[    1.552506]  el1h_64_error+0x64/0x68
[    1.552508]  bcm2712_pull_config_set+0x64/0xd8
[    1.552511]  bcm2712_pinconf_set+0x94/0xe0
[    1.552513]  pinconf_apply_setting+0xd0/0x140
[    1.552515]  pinctrl_commit_state+0x1bc/0x1e8
[    1.552517]  pinctrl_select_state+0x24/0x40
[    1.552519]  pinctrl_bind_pins+0x144/0x160
[    1.552521]  really_probe+0x5c/0x2e0
[    1.552525]  __driver_probe_device+0x80/0x128
[    1.552528]  driver_probe_device+0x44/0x180
[    1.552531]  __driver_attach+0x98/0x1b0
[    1.552534]  bus_for_each_dev+0x84/0xf8
[    1.552536]  driver_attach+0x2c/0x40
[    1.552539]  bus_add_driver+0xec/0x220
[    1.552542]  driver_register+0x70/0x138
[    1.552543]  __platform_driver_register+0x2c/0x40
[    1.552545]  brcmuart_init+0x44/0x70
[    1.552548]  do_one_initcall+0x60/0x2a0
[    1.552550]  kernel_init_freeable+0x22c/0x2a0
[    1.552553]  kernel_init+0x2c/0x150
[    1.552555]  ret_from_fork+0x10/0x20
[    1.552557] SMP: stopping secondary CPUs
[    1.552560] Kernel Offset: 0x100c400000 from 0xffffffc080000000
[    1.552561] PHYS_OFFSET: 0x0
[    1.552562] CPU features: 0x10,00000017,00280928,0200720b
[    1.552564] Memory Limit: none
[    1.827867] ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---

Footnotes

https://trustedfirmware-a.readthedocs.io/en/v2.11/plat/rpi5.html ↩ ↩2 1.

https://github.com/OP-TEE/optee_client/issues/345#issuecomment-1446005817 We don’t need this, so we can disable building libteeacl with WITH_TEEACL=03. ↩ 1.

https://github.com/OP-TEE/optee_client/commit/bbdf665aba39c29a3ce7bd06e4554c62a416ebaa ↩