Jekyll2024-01-29T10:14:33+00:00https://mquaresma.github.io/feed.xmlMiguel QuaresmaPh.D researcher in high performance, formally verified cryptography. Endurance athlete focusing on trail running, cycling and triathlons.Docker Primer2020-04-16T21:58:42+00:002020-04-16T21:58:42+00:00https://mquaresma.github.io/devops/containers/docker/2020/04/16/Docker-primer<h1 id="docker-primer">Docker Primer</h1>
<h2 id="backstory">Backstory</h2>
<p>Docker, along with containers, are technologies that I’ve been trying to understand for quite sometime now,
as I’ve had to use them more than once. However, I always managed to just Copy-Paste commands without really
understanding how Docker worked under the hood. This meant that, whenever a problem would arise, I’d have to
Google for the solution and be totally lost on whether the solution was actually the right one for the
problem I was facing. This summer I finally decided to spend sometime reading through some of it’s
documentation and, with the help of <a href="https://www.youtube.com/watch?v=YFl2mCHdv24">Jake Wright’s video</a> I
took some notes that I thought could be useful for anyone in the same situation I was in.</p>
<h2 id="terminology">Terminology</h2>
<p>Here’s some terminology to take into account when dealing with Docker:</p>
<ul>
<li>Image:
<ul>
<li>template describing the desired environment</li>
<li>steps to build the image are specified in a Dockerfile</li>
<li>DockerHub includes a collection of community images that can be customized to suit your needs</li>
</ul>
</li>
<li>Container
<ul>
<li>running instance of a Docker Image</li>
</ul>
</li>
</ul>
<p>For those familiar with Object Oriented Programming, you can look at Images as classes and Containers as
objects. So, with those concepts out of the way, let’s now focus on what Docker really is and how to use
it.</p>
<h2 id="brief-overview-of-docker">Brief overview of Docker</h2>
<p>Containers are frequently confused with Virtual Machines however, while macOS does require a Virtual Machine
instance to be running alongside Docker due to the lack of native support (for containers), containers
running in Linux use the host’s machine kernel instead of creating an additional one, unlike VMs. The main
purpose of Docker containers is to provide a reliable environment with which to develop and deploy
applications. Additionally, containers also provide an isolated environment in which to run applications,
even though this isolation is not as strong as in VMs.</p>
<h2 id="usage">Usage</h2>
<p>As I’ve explained previously, the base of Docker are images as they are used to describe a specific
environment. When you want to spin up a container, you must tell Docker where the image for that container
is. To obtain images you can either create a Dockerfile and build the image from it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker build -t <image_tag> <Dockerfile_file>
</code></pre></div></div>
<p>or grab one from <a href="https://hub.docker.com/search?q=&type=image">Docker Hub</a> using docker pull:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker pull <image_name>
</code></pre></div></div>
<p>After building an image you can spin as many containers as you want from that image</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker run <image_id>
</code></pre></div></div>
<p><strong>Tip:</strong> to launch a container and get an interactive shell use the <code class="language-plaintext highlighter-rouge">-t</code> flag:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker run -it <image_id>
</code></pre></div></div>
<p>Another usefull feature that Docker provides is container orchestration, which facilitates the deployment of
multi-container setups. The usefullness of this feature lies in the fact that most applications nowadays
are multi-layered systems, where each layer runs in a separate environment but must interact with the other
layers. <a href="https://docs.docker.com/compose/">Docker compose</a> is a tool provided by Docker that has commands
for managing the lifecycle of multi-layered applications. The initial configuration is achieved via a YAML
file, docker-compose.yml, which indicates the images (or Dockerfiles) used for the containers, called
services, as well as the volumes, ports and dependencies of each service.</p>Docker Primer Backstory Docker, along with containers, are technologies that I’ve been trying to understand for quite sometime now, as I’ve had to use them more than once. However, I always managed to just Copy-Paste commands without really understanding how Docker worked under the hood. This meant that, whenever a problem would arise, I’d have to Google for the solution and be totally lost on whether the solution was actually the right one for the problem I was facing. This summer I finally decided to spend sometime reading through some of it’s documentation and, with the help of Jake Wright’s video I took some notes that I thought could be useful for anyone in the same situation I was in.Setting up OPTEE and Linux for the Pine A642020-04-11T15:09:42+00:002020-04-11T15:09:42+00:00https://mquaresma.github.io/pine/a64/2020/04/11/Setting-up-OPTEE-and-Linux-for-the-Pine-A64<h1 id="pine-a64-manual-setup">Pine A64 Manual Setup</h1>
<p>The <a href="https://linux-sunxi.org/Pine64">Pine A64</a> is a board running the <a href="https://linux-sunxi.org/A64">A64</a> SoC
and comes in three different variants:</p>
<ul>
<li>Pine A64 512MB</li>
<li>Pine A64+ 1GB</li>
<li>Pine A64+ 2GB (this is the one I have)</li>
</ul>
<p>Allwinner is the manufacturer of the A64 SoC which is a Quad-core Cortex-A53 ARM CPU and a Mali400 MP2 GPU from
ARM. The Cortex A53 implements the AArch64 architecture.</p>
<p>Although Allwinner provides a BSP(Board-Support-Package), the U-Boot bootloader doesn’t compile and the Linux
kernel image is patched (and outdated) so it’s better to compile mainline kernel and U-Boot as support for Pine A64
is present.</p>
<h2 id="boot-flow">Boot Flow</h2>
<p>OP-TEE can be booted on the Pine A64 in combination with Linux. In this setup, the boot flow is as follows:</p>
<p><img src="/assets/ARMv8_BootFlow.png" alt="ARMv8 Bootflow" /></p>
<p>The BootROM code, considered BL1, is loaded from the chips ROM which in turn loads the SPL responsible for
initializing DRAM and loading OP-TEE, ARM-TF and U-Boot to memory. The processor then branches to ARM-TF
which first loads OP-TEE and then hands control over to U-Boot, the normal world bootloader, which will load
the normal world kernel.</p>
<h2 id="firmware">Firmware</h2>
<p>The Pine A64 firmware is made up of four different parts:</p>
<ul>
<li>The on-chip Boot-ROM (BROM), which is the first code to be executed and cannot be modified</li>
<li>A <strong>S</strong>econdary <strong>P</strong>rogram <strong>L</strong>oader (SPL) responsible for initializing DRAM and loading the remaining
firmware parts. This component is part of U-Boot</li>
<li>EL3 runtime firmware (<strong>i.e</strong> ARM TF-A), support for A64 SoC is present in mainline</li>
<li>U-Boot bootloader for loading kernels and data; support for A64 is present in upstream</li>
</ul>
<h2 id="build">Build</h2>
<p>This document takes all the components from mainline/upstream, instead of the ones provided by Sunxi.</p>
<h2 id="tooling">Tooling</h2>
<p>To build the different components (ARM TF-A, U-Boot, Linux), setup a cross compilation toolchain and add the binaries
to the PATH env variable:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export PATH=$PATH:<path to aarch64 binaries>:<path to aarch32 binaries>
export CROSS_COMPILE="ccache aarch64-linux-gnu-"
</code></pre></div></div>
<ul>
<li>U-Boot requires <code class="language-plaintext highlighter-rouge">swig</code>, <code class="language-plaintext highlighter-rouge">python-dev</code> and <code class="language-plaintext highlighter-rouge">device-tree-compiler</code> packages to be present on the system.</li>
<li>Linux kernel modules require <code class="language-plaintext highlighter-rouge">kmod</code> to be installed.</li>
</ul>
<h2 id="arm-tf-a">ARM TF-A</h2>
<p>ARM Trusted Firmware is responsible for the EL3 runtime secure monitor in the Pine A64, switching between the secure
and normal world, S-EL1 and EL1. The following sequence of commands is enough to compile it for the Pine A64:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://review.trustedfirmware.org/TF-A/trusted-firmware-a
cd trusted-firmware-a
git checkout <latest stable release> (e.g. v2.2)
make PLAT=sun50i_a64 SPD=opteed DEBUG=1 bl31
export BL31=$(pwd)/build/sun50i_a64/debug/bl31.bin
</code></pre></div></div>
<h2 id="optee">OPTEE</h2>
<p>OP-TEE is made up of two essential components, the (secure) OS itself, OPTEE OS, and a client
which will run in the normal world and interact with OPTEE OS via the OPTEE driver provided
by the (Linux) kernel.</p>
<h3 id="optee-os">OPTEE OS</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/OP-TEE/optee_os.git -b <latest stable release> (e.g. 3.8.0)
cd optee_os
make CFG_ARM64_CORE=y \
CFG_TEE_LOGLEVEL=4 \
CFG_TEE_CORE_LOG_LEVEL=4 \
CROSS_COMPILE32="ccache arm-linux-gnueabihf-" \
CROSS_COMPILE64="ccache aarch64-linux-gnu-" \
DEBUG=1 \
PLATFORM=sunxi-sun50i_a64
export TEE=$(pwd)/out/arm-plat-sunxi/core/tee-pager_v2.bin
</code></pre></div></div>
<p>The file pointed to by <strong>TEE</strong> is a RAW ARM executable which will be branched to by ARM TF-A.</p>
<h3 id="optee-client">OPTEE Client</h3>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/OP-TEE/optee_client -b <latest stable release> (e.g. 3.8.0)
cd optee_client
make
cd out/export
tar -cfv optee_client.tar.gz usr
</code></pre></div></div>
<p>The resutling tar archive should be placed in the <a href="#rootfs">root partition</a> in the respective subdirectories.</p>
<h2 id="u-boot">U-Boot</h2>
<p>U-Boot is the normal world bootloader, however the repo also includes a SPL which will be loaded by BL1.</p>
<h3 id="modifications">Modifications</h3>
<p>The binary image built for the Pine A64+ board in mainline U-Boot doesn’t load OPTEE, thus the
script used to generate the Device Tree Binary must be patched. The <a href="/assets/mksunxi_fit_atf_optee.sh">patched version</a>
should be copied to <code class="language-plaintext highlighter-rouge">board/sunxi/mksunxi_fit_atf.sh</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://gitlab.denx.de/u-boot/u-boot.git
cd u-boot
git checkout <latest stable release> (e.g. v2020.04-rc3)
make pine64_plus_defconfig
make
export PATH=$PATH:$(pwd)/tools/
</code></pre></div></div>
<p>The SPL, U-Boot plus TF-A and OP-TEE are packaged in <code class="language-plaintext highlighter-rouge">u-boot-sunxi-with-spl.bin</code> which will later be flashed
before the first partition of the SD Card at an 8Kb offset:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dd if=u-boot-sunxi-with-spl.bin of=/dev/sdx bs=1k seek=8
</code></pre></div></div>
<h2 id="linux">Linux</h2>
<h3 id="modifications-1">Modifications</h3>
<p>The mainline kernel provides enought support for a headless setup of the Pine A64 (<strong>e.g.</strong>: serial + ssh),
however in order for the HDMI output to work and the OP-TEE driver to be loaded at boot, some modifications
must be introduced.</p>
<ol>
<li>The default kernel configuration for the Pine A64+ board doesn’t enable the Mali400 GPU driver, so it
must be enabled manually via the menuconfig target when running <code class="language-plaintext highlighter-rouge">make menuconfig</code>, after step <strong>3</strong>.</li>
<li>Modify the DTS (Device Tree Source) file in <code class="language-plaintext highlighter-rouge">arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-plus.dts</code>
to include the following block:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>firmware {
optee {
compatible = "linaro,optee-tz";
method = "smc";
};
};
</code></pre></div> </div>
<p>which enables the kernel to identify and load the OP-TEE at boot time.</p>
</li>
</ol>
<h3 id="steps">Steps</h3>
<ol>
<li>Clone the mainline repo, latest version
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/torvalds/linux.git -b v5.6 --depth=1
cd linux
</code></pre></div> </div>
</li>
<li>
<p>Replace the DT source for the Pine A64, in <code class="language-plaintext highlighter-rouge">arch/arm64/boot/dts</code> with the
<a href="/assets/sun50i-a64-pine64-plus.dts">new one</a></p>
</li>
<li>
<p>Generate a configuration: <code class="language-plaintext highlighter-rouge">make ARCH=arm64 defconfig</code></p>
</li>
<li>
<p>Create the kernel image: <code class="language-plaintext highlighter-rouge">make ARCH=arm64 -j2 Image</code> which is placed in
<code class="language-plaintext highlighter-rouge">arch/arm64/boot/</code></p>
</li>
<li>
<p>Compile the device tree for the Pine A64: <code class="language-plaintext highlighter-rouge">make ARCH=arm64 -j2 dtbs</code>; the
dtb file will be placed in <code class="language-plaintext highlighter-rouge">arch/arm64/boot/dts/</code></p>
</li>
<li>Compile the kernel modules which include the Mali 400 GPU driver needed for HDMI output:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>make -j2 ARCH=arm64 modules
make -j2 ARCH=arm64 INSTALL_MOD_PATH=modules modules modules_install
</code></pre></div> </div>
</li>
<li>Create a tar archive with the kernel modules <code class="language-plaintext highlighter-rouge">tar -czf modules.tar.gz modules</code></li>
</ol>
<h2 id="partitioning-the-sd-card">Partitioning the SD Card</h2>
<p>Partitioning the SD card can be achievied by running the following sequence of commands:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo dd if=/dev/zero of=/dev/sdX bs=1M count=1
blockdev -rereadpt /dev/sdX
cat <<EOT | sfdisk /dev/sdX
2M,32M,c
,,L
EOT
mkfs.vfat /dev/sdX1
mkfs.ext4 /dev/sdX2
</code></pre></div></div>
<h2 id="u-boot-configuration">U-Boot Configuration</h2>
<p>U-Boot is configured via a file, <code class="language-plaintext highlighter-rouge">boot.cmd</code>, which will be wrapped by a U-boot header. To configure U-Boot:</p>
<ol>
<li>Mount the boot partition <code class="language-plaintext highlighter-rouge">/dev/sdX1</code> in <code class="language-plaintext highlighter-rouge">/mnt</code> and create the <code class="language-plaintext highlighter-rouge">boot.cmd</code> file as such:</li>
</ol>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fatload mmc 0 0x46000000 Image
fatload mmc 0 0x49000000 sun50i-a64-pine64-plus.dtb
setenv bootargs console=ttyS0,115200 console=tty1 earlyprintk root=/dev/mmcblk0p2 rootwait panic=10
booti 0x46000000 - 0x49000000
</code></pre></div></div>
<ol>
<li>Copy the kernel <code class="language-plaintext highlighter-rouge">Image</code> and the device tree <code class="language-plaintext highlighter-rouge"><board>.dtb</code> to the boot partition and run: <code class="language-plaintext highlighter-rouge">mkimage -C none
-A arm64 -T script -d boot.cmd boot.scr</code></li>
</ol>
<h2 id="rootfs">Rootfs</h2>
<p>There are several utilities to create a root filesystem, <code class="language-plaintext highlighter-rouge">deboostrap</code> is one such tool. Mount the root
partition <code class="language-plaintext highlighter-rouge">/dev/sdX2</code> in <code class="language-plaintext highlighter-rouge">/mnt</code>:</p>
<ol>
<li>Use <code class="language-plaintext highlighter-rouge">debootstrap</code> to install a Debian base system on the root partition</li>
<li>Chroot into the mounted partition, using qemu: <code class="language-plaintext highlighter-rouge">chroot /mnt /usr/bin/qemu-aarch64-static /bin/sh -i</code></li>
<li>Run <code class="language-plaintext highlighter-rouge">/deboostrap/deboostrap --second-stage</code></li>
</ol>
<h2 id="kernel-modules">Kernel Modules</h2>
<p>With the boot partition mounted on <code class="language-plaintext highlighter-rouge">/mnt</code>:</p>
<ol>
<li>Extract the modules from the tar archive <code class="language-plaintext highlighter-rouge">tar -xf modules.tar.gz</code></li>
<li>Copy the modules to the root filesystem <code class="language-plaintext highlighter-rouge">cp -r modules/lib/ /mnt/lib</code></li>
</ol>Pine A64 Manual Setup The Pine A64 is a board running the A64 SoC and comes in three different variants: