Investigation of OPC UA PubSub over TSN using Raspberry Pi 4 boards.
This is the project we did for TUM TechChallenge ’21 with our corporate partner Huawei. The project was about investigating the usage of OPC UA PubSub over TSN (Time Sensitive Networking) on Raspberry Pi 4.
The test setup included 2 Huawei TSN switches, 4 Raspberry Pi 4 boards, a PC to simulate heavy traffic and a robotic arm we built as an actuator. Majority of the project was testing different kernel configurations and figuring out the limitations of the RPi.
This guide is the stuff that we learned during this 2 month project. It is heavily influenced by the Kalycito guide https://www.kalycito.com/how-to-run-opc-ua-pubsub-tsn/ . The major difference is that they use Intel i210 Ethernet controller. Raspberry Pi 4 doesn’t have that, which changes lots of things. We reused most of their scripts and methods but tried to adapt them to the Raspberry’s Ethernet adapter and arm64 architecture. One other major difference is them using kernel version 4.19 while we are using 5.4.
This is in no way an exhaustive guide. It barely scratches the surface. There are many things that can be improved upon. We mention some of those things in the future works section.
Steps to reproduce:
1- Flash 2020–08–20-raspios-buster-arm64.zip (https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2020-08-24/2020-08-20-raspios-buster-arm64.zip) to each Raspberry Pi. This will install 64 bit raspbian to the board. Boot the device once to ensure the image is installed correctly.
2- To support TSN, several kernel configurations need to be changed. This was mostly trial and error for us. What we ended up changing is installing a PREEMPT_RT kernel patch (https://github.com/kdoren/linux branch rpi-5.4.81), enabling Netfilter options, QoS options, 8021.q support.
- You can get the kernel files we compiled from https://drive.google.com/file/d/1AOLgMWt15BLk_HZYtj2gnda6duWFZ_fA/view?usp=sharing . Copy the kernel8.img, entirety of the overlays folder and contents of the broadcom folder to the boot folder. Copy all of the modules under lib folder to rootfs /lib/modules to the microSD. Add kernel=kernel8.img to config.txt in /boot folder.
3- If you want to connect to RPi by SSH create a empty file called SSH in /boot folder. Follow https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md to make the wireless connection configuration.
4- Insert the SD card into RPi and boot the device.
5- SSH into the device and type uname -a. If everything until now worked, you should see kernel version 5.4.81-rt with PREEMPT_RT enabled.
6- Give static IPs for eth0 ethernet interfaces. We ended up giving 192.168.20.40 for one RPi, 192.168.20.41 for the other RPi and so on.
7- Append the command isolcpus=1,2,3 in /boot/cmdline.txt . This will isolate the CPUs.
8- Download setup.sh from https://drive.google.com/file/d/1Jfy8kKsQtoX9QaiFu22npGaXk8h2RPNH/view?usp=sharing . Run it inside the RPis. It will install necessary packages, take care of the IRQ balancing, install the correct iproute and linuxptp for kernel 5.4.81-rt and configure ptp. It will also enable necessary 8021q module for VLAN tagging.
9- Reboot the device.
10- In this project, we used basic VLAN tagging and let the TSN switches handle everything but if you want to override the default traffic control schedulers, this will be the step to do it. You can override schedulers by typing the following lines to the terminal.
sudo tc qdisc add dev eth0 parent root mqprio num_tc 3 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@2 hw 0
MQPRIO_NUM=`sudo tc qdisc show | grep mqprio | cut -d ‘:’ -f1 | cut -d ‘ ‘ -f3`
sudo tc qdisc add dev eth0 parent $MQPRIO_NUM:1 etf clockid CLOCK_TAI delta 150000
sudo tc qdisc add dev eth0 parent $MQPRIO_NUM:2 etf clockid CLOCK_TAI delta 150000
Note that ETF schedulers need the offload parameter as well in the normal case. This requires hardware timestamping. Unfortunately Raspberry Pi 4 doesn’t support it.
11- Install the OPC UA library.
sudo apt-get install git cmake cmake-curses-gui build-essential gcc python3-pip qttools5-dev python3-pyqt5 libmbedtls-dev check libsubunit-dev python-sphinx graphviz python-sphinx-rtd-theme
git clone https://github.com/open62541/open62541.git
cd open62541/
git fetch origin pull/3996/head:local_branch && git checkout local_branch
12- Open the pubsub_TSN_loopback.c and pubsub_TSN_publisher.c in examples/pubsub_realtime. Change the sys/io.h to sys/uio.h in both of the files. This is necessary because these headers don’t exist in arm64.
13- Build the library with the correct parameters to support real time PubSub
mkdir build
cd build
cmake -DUA_BUILD_EXAMPLES=ON -DUA_ENABLE_PUBSUB=ON -DUA_ENABLE_PUBSUB_ETH_UADP=ON ..
make -j8
14- Connect the devices to the TSN switches. The subscriber connected to one switch and the publisher connected to the other switch. Connect the two switches from their port 0 (trunk). Connect the PC to the same switch as the publisher.
15- Install iperf3 to the Raspberry Pi (subscriber)
sudo apt-get install iperf3
16- Run iperf server on the subscriber
iperf3 -s
17- Run the iperf3 client on your PC
iperf3 -c raspberry-ip -t600
18- Run the OPC PubSub publisher and subscriber code in their respective boards.
19- 3D print the robot and assemble using the following guide. https://www.myminifactory.com/object/3d-print-rc-servo-robot-arm-128617
20. Connect the robot to the subscriber and the actuator to the publisher.
Future Work:
This project is merely an introduction and feasibility study towards OPC UA over TSN on single board computers. We believe it can be improved upon in many ways.
- The first thing that we wanted to include in this project was having a time chart comparison between the normal case and TSN case. We couldn’t do it due to time constraints but it would be a great addition to this project.
- The biggest improvement would be getting ETF scheduler to run on the Raspberry Pi 4. This is bit of a problem as hardware timestamping is not supported on RPi. It might be possible to run it with software timestamping.
- We believe time synchronization using PTP could be improved by using the sensitive internal clock of the TSN switch. We included configuration of linuxptp for kernel 5.4 in the given setup.sh script but it was not tested thoroughly due to time constraints. A time chart that shows the synchronization of the internal clocks would be a nice addition.
There are many things that can be improved but we hope that this guide will serve as a starting point for those that wish to work on this subject.
