Yocto | Academic OSS ELC

Yocto Project Dev Day Virtual NA 2020

Thu, 02 Jul 2020 00:00:00 +0000

The Yocto Project Developer Day Virtual is a one day presentation and hands-on training event that puts you in direct contact with Yocto Project technical experts and developers. Its primary aim is to show developers how to create, customize, and optimize Linux distributions for embedded devices using the rich features, tools, and content of Yocto Project. Our knowledgeable and engaging instructors will help you better understand topics like build system workflow, working with containers, building applications, optimizing images, hardening your devices, and leveraging tools like devtool. You will also have a chance to network and put your new skills to work.

Program

Keynote & Welcome
Security Hardening
Image Size Optimization
Reproducible Builds
LTS
CI/CD
DevTool Lab
User Space Lab
Kernel Lab
and more…

Agenda

9:00 30 Paul Barker, "Highly-scalable build automation"
9:30 45 Robert Berger, "Containers and micro services", use cases: containers on host and target
10:15 15 --- Morning break ---
10:30 90 Trevor, "Kernel Lab", HANDS ON
12:00 60 --- Lunch ---
1:00 15 BugBoard Presentation, Call to action
1:15 30 Scott, "Security Hardening"
1:45 30 Khem, “Techniques and tools to reduce image size with Yocto Projects"
2:15 30 Konrad, "CI/CD"
2:45 15 --- Afternoon break ---
3:00 60 Rudi, User Space, HANDS ON
4:00 30 Paul Eggelton, "DevTool"
4:30 30 Christopher Clark & TimO, "Virtualization"

Highly scalable build automation

Solutions

Solutions?

Buildbot
GitLab CI
Jenkins

Design

ideally CI should run a single comamnd
Ensure you can run the same scrip tlocally
allo manual build and release manually if your CI system breaks

Pulling in other layers

git submodule
repo
oe-layersetup
kas

examin simple setup

Build environment sstate cahce download cache artifact storage log storage management interface

Scaling up

bigger build machine
NVMe > SATA SSD > SATA HDD
Verify write edurance on SSD
Server/Workstation Hardware

CPU and RAM

Don’t blinly assume more expenseive is better
check for single threaded bottlenecks
- if they dominate go for high clock
- otherwise go for high core count
profile RAM usage

Tips

dedicated machine not VM
if you use docker make sure you look into settings

Benefits of Scaling Out

developer machines can use central sstate cache
maintain download mirror
- license compliance
- protect if upstream sources disappear
reduce single points of failure

Local Scale Out

High speed networking
- 10Gbps between build machines and NAS

Global Scale Out

issues
- Limited network speeds
- Packet drops and errors will happen
- may see errors accessing the cache
server caches over https
update cache over SSH or an API

Distributed Cache and Aritacts Storage

Build your own storage cluster
- Ceph, Gluster, etc
- Minio is also an option with S3 compatbile API

A low cost example

meta-sancloud bsp layer

high level design

GItlab CI
BackBlaze B2
CloudFlare eliminate bandwidth costs

kas

desrived in yaml

gitlab-ci

Build agend

Ryzen 7 3700X
64GB DDR4 EEC
2x 1TB NVMe drives in RAID1 pair

GitLab Runner configuration

gitlab runner has very min configuration
limit concurrent jobs
select the docker job executor
register with gitlab ci

docker

mostly use crops/yocto
sometimes custom

BackBlaze2

No upload cost
cost for download

CloudFlare

downloads from BackBlaze via CloudClare are free
dedicated domain
make sure you disable browser integrity check

Uploading

use rclone
API key stored in gitlab-ci

MOnnthly

Build Agents about $60 per month
BackBlaze B2 approx $2 per month
no long te4rm commitments
Bitbake inc files
- sstate
- download mirro
gitlab-ci
kas config
- kas poky
- kas argo

Summary

requires some sysadmin work in setup and maintenance

2

mosquitto - mqtt browker
telegraf - colect analyze
influxdb
Grafana

mTIG Set ups

mTIG Stack - “classic”
mTIG Stack - “vms”
mTIG Stack - “containers” - microservices
mTIG Stack - “docker-compose”

Showed Dockerfile for mosquitto

volumes for host storage
expose for port acces

yoctozing the setup!

populate sdk
building packages from source
- time consuming task
- normally glang magically pulls in required dependencies
- depency hell

prebuilt binaries

influxdb-prebuilt
this was easier
but
- prebuilt binaries are packaged differently
- how reproducable is this? not at all

package summary

additional work is required
image recipe - easy
mosquitto - easy recipe upstream
influxdb
telegraph

Yocto Person - get docker packages on the target

add packages
configure kernel

Docker on the Target

bitbaking from sources seems to be hard prebuilt containers for the target need docker and friends

docker architecture

Start it up

docker-compose up
- same as before

Put it all together

container standardization

app-container-image-influxdb-prebuilt-oci

we need minimal rootfs without kernel and meta-virtualization
skopeo in host container or host machine
app-container-image-oci.bb
IMAGE_FSTYPE ‘oci’ from meta-virtualization
IMAGE_CONTAINER_NO_DUMMY = “0” -> no kernel

Building container images with OE and Yocto Project Scott Murry 2018

Kernel

upstream linux
linux-stable
vendor kernel
eveil vendor kernel (fraken-kernel)
Install kernel, dtb
installed is highly board specific

Kernel Workflow multiple users

tewaking a kernel/config or writing a kernel module using an existing BSP
board bring-up (creating a BSP)

Kernel Workflow TODO existin BSP chang kernel

many kernels
linux-yocto plus tolling to use your own
linux-yocto has variants
- -dev, - -tiny, -rt

new BSP

obtain kernel
setup cross toolchain - set the processor TUNE
configure kernel and tweak DT . lots of yocto tooling
install kernel modules DTB

Kernel metadata

config fragment

hands on we use devtool
devtool or -c menuconfig -c diffconfig
filename has cfg ext

SCC

series configuration control
ties together config framgent or patches with extra metadata

Kernel Metadata - storage

two places to store
- in-tree
- in separate repository
regardless add the lcoation of metadata to your SRC_URI.
- Make sure to specify “type=kmeta”
its much more sophisticated
can define other kernel types KTYPE and have featured enabled based on kernel type (-tiny,-dev,-rt)
- scc files can include other scc files
- can restrict the includes of other scc
examples of in-tree kmeta:
- meta-odroid
- meta-arm
- meta-xilinx
example of just using *.cfg without *.scc
- meta-raspberrypi

Security Hardening

Intro

Agenda

Security Hardening?
Basic hardening with OE/YP
meta-security
meta-selinux
Updater layers
meta-sca
not an expert

Security Hardening?

Securing a system by reducing its attack surface
Remove unnecessary software/services, users
Control network access, e.g. firewall
Intrusion detection
Remove/improve default passwords/users
Updates to remove vulnerabilities
etc.

Why

Attackers are becoming more aware of Linux devices

Scans of all of IPv4 are a thing, e.g. shodan.io
Customers cannot be relied upon to not attach devices directly to the Internet
uPnP may make device services visible unexpectedly

During ELC there were good discussions

Some presentations this week with more detailed discussion

“Introduction to Embedded Linux Security” - Sergio Prado, Embedded Labworks
https://ossna2020.sched.com/event/c3XR/introduction-to-embedded-linux-security-sergio-prado-embedded-labworks#
“IoT Developer’s Guide to Building Secure IoT Devices” - Yogesh Ojha, Tata Consultancy Services
https://ossna2020.sched.com/event/c3Tl/iot-developers-guide-to-building-secure-iot-devices-yogesh-ojha-tata-consultancy-services#

OWASP vunerability list

OE/YP Hardening

Read the Fine Manual
https://www.yoctoproject.org/docs/3.1.1/dev- manual/dev-manual.html#making-images-more-secure
Provides some useful high-level guidelines
Has some more detailed guidance around disabling debug features, adding users and passwords, and security related compile flags
Mentions meta-security and meta-selinux
Useful, but mostly a starting point

Expanding on the FM

oe-pkgdata-util useful for finding what package files come from

Review kernel configuration

Security options, but also things like hardware RNG, architecture specific address space randomization
Some more ideas in https://www.whonix.org/wiki/Hardened-kernel
Make sure CONFIG_DEVMEM is disabled if at all possible
Typically used to access device registers as a workaround
Somewhat better now with default values of STRICT_DEVMEM and IO_STRICT_DEVMEM, but using/fixing drivers and disabling is safer

It’s common for BSP layers to not enable desired features…

e.g. cgroup, namespace, netfilter, BPF support
These become more visible when using systemd or container runtimes
…and to enable a lot of things you do not need
Usually err on the side of enabling a lot of driver subsystems and drivers
May enable DEBUG options that are problematic
ROOTFS_READ_ONLY feature is worth considering
- Increase difficulty for attackers
- Secondary benefit of also being useful for implementing reset to factory default schemes
May require development effort
- Locally developed applications, or packages from outside oe-core may not work out of the box
- Combining with MAC schemes such as SELinux will require some work (as labelling is done on boot)

meta-security

cve-check.bbclass
SRTool (https://wiki.yoctoproject.org/wiki/SRTool_User_Page) may be useful if you need to set up an issue tracker

meta-security – Compliance

Recipes in meta-security-compliance layer
Lynis (https://cisofy.com/lynis) runtime system auditor
OpenSCAP (https://www.open-scap.org)
- Implementation of Security Content Automation Protocol
- In simple terms, a specification of standardized naming for interaction with tools and databases
- oscap and oscap-daemon tools for checking NIST or other databases for vulnerabilities
These seem likely to be overkill in a lot of embedded usecases
But perhaps still useful in a QA role
meta-security – Runtime Scanners

meta-selinux

meta-sca

https://github.com/priv-kweihmann/meta-sca
Collection of static analysis tools maintained by Konrad Weihmann
Static analysis for C, C++, python, etc.
Classes to enable per package or per image scanning (some limits depending on specific tools)
Significant documentation
Actively maintained

Summary

Updater layers – meta-swupdate
Updater layers – meta-mender
Updater layers – meta-updater
meta-sca - Collection of static analysis tools maintained by Konrad Weihmann

CI/CD

Wrigel the Third

Docker, DockerSwam
Designed for easy setup of small cluters
internal build and in AWS
https://github.com/WindRiver-OpenSourceLabs/ci-scripts

WR setup tool and git-repo

git-repo manages large tree of repos
manifest generation by setup tool
git repo detects changes across entire tree
automatic rebase of local commits when syncing
con: fork with support for bare repos

setup tool and layerindex

dicovery and depenendecy management
bitbake-layers layerindex-fetch is based on some of that

Developer builds

repo detects commits
fork repos and store commits on git server
containerized layerindex uses forks
recreate local buildarea on buildsystem
CONTAINERIZE layerindex???
start_devbuild.sh

developer can’t reproduce build failure on build server
save docker container startup command
create temp ssh key, upload build server
add authorize key entry to run docker on login
build environment perfectly recreated
cleanup key
workspace_login.sh
https://lava.ciplatform.org/

Summary

expose large compuse to developers
build and test patches before commit
build failures inspection
integration with runtime testing

Demo

OERuntimeTestCase
searching layers in devbuild for local commits
yocto autobuilder helper scripts
awe to scale builds
k8s and tekton
yocto self-hosted AMIs
spot instances are a great deal

Questions

try to prefer docker volumes.
host path can be performant
Add tmpfs at the common tmp locations. Some recipes misbehave.
you can do docker run and verify that nothing in overlay has changed.

Techniques and tools to reduce image size with Yocto Projects

Agenda

Introduction
“poky-tiny” Distribution
Image analysis tools
Compiler and toolchain optimizations
C/C++ this time
package refactoring
Package selection

Introduction

Reference Distrubtions

poky-tiny
- small size
- musl
- linux-yocto-tiny
- mdev
- custom init
poky-altconfig
- systemd
- LTSI kernel

“poky-tiny” Distribution

start small and then add
much harder to remove things, often due to dependencies

Packages

musl C Library
- IMAGESIZE = 37668 -> 35532
- this requires changes in application
- changes interface to linux kernel
busybox init system
- IMAGESIZE = 35532 -> 3120
mdev over udev

Image analysis tools

Compiler and toolchain optimizations

Compiler Flags

SELECTED_OPTIMIZATION

TARGET_CFLAGS = "${TARGET_CPPFLAGS} ${SELECTED_OPTIMIZATION}"
…
DEBUG_FLAGS ?= "-g -feliminate-unused-debug-types ${DEBUG_PREFIX_MAP}"
# Disabled until the option works properly -feliminate-dwarf2-dups
FULL_OPTIMIZATION = "-O2 -pipe ${DEBUG_FLAGS}"
DEBUG_OPTIMIZATION = "-Og ${DEBUG_FLAGS} -pipe"
SELECTED_OPTIMIZATION = "${@d.getVar(oe.utils.vartrue('DEBUG_BUILD', 'DEBUG_OPTIMIZATION', 'FULL_OPTIMIZATION', d))}"

LTO

flto
clang support
- thin LTO
- full LTO
- needs gold linker

Package selection

Buildhistory

USER_CLASSES ?= " buildhistory”
BUILDHISTORY_COMMIT ?= "1"

Refine Image

IMAGE_LINGUAS
“en-us”
Use “as needed”, prefer empty
IMAGE_FEATURES
Remove package-management
Remove splash
SSH – Use dropbear instead of OpenSSH
Add ssh-server-dropbear to IMAGE_FEATURES

Build history analysis

installed-package-sizes.txt
dot files for dependencies between packages

Slack Channel

https://elinux.org/images/8/8f/Poky-tiny_and_Beyond.pdf

User Space, HANDS ON

Packing QA

packages-list
- same package multiple times
installed-vs-shipped
already-stripped
ldflags - ELF binary has symbols but no GNU_HASH

Questions

Q: What is the “BACKFILL_CONSIDERED” mean?
- if any packages says it needs sysvint then you should just ignore it
- It’s features that were on by default (usually implictly because there was no knob), so you are acknowledging that “considered” you want it off (at least thats my interpretation)
- Bascially, it was non-configurable, then it became a knob, but they had to default it to “on” to maintain backward compatability
- https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#ref-features-backfill

DevTool

devtool – past presentations

Using Devtool To Streamline Your Yocto Project Workflow - Tim Orling
- https://www.youtube.com/watch?v=CiD7rB35CRE
Yocto Project Extensible SDK: Simplifying the Workflow for Application Developers - Henry Bruce
- https://www.youtube.com/watch?v=d3xanDJuXRA&t=57s

Why devtool exists?

working with sources as hard
- tempting to just edit sources under tmp
- painful (forced builds, mannual patch generations, lost work)

Devtool has two modes

each has slight differences. It’s the only interfaces for eSDK.

eSDK mode

add
build
build-image
build-sdk
configure-help
check-upgrade-status
deploy-target
edit-recipe
export
extract
find-recipe
finish
import
latest-version
menuconfig
modify
package
rename
reset
runqemu
sdk-install
sdk-update
search
status
sync
undeploy-target
update-recipe
upgrade

bitbake mode

add
build
build-image
configure-help
check-upgrade-status
create-workspace
deploy-target
edit-recipe
export
extract
find-recipe
finish
import
latest-version
menuconfig
modify
rename
reset
search
status
sync
undeploy-target
update-recipe
upgrade

recipetool

extra set of tools for working on recipes
logic for creating recipes (used by devtool add)
can also create update bbapends, programmactically set variables in recipes, etc

Future

multiconfig support
recipe modification fixes
smart recipetool

Lab

Host qemu
User root
Hostname localhost
Port 2222
StrictHostKeyChecking no
UserKnownHostsFile /dev/null

SLIRP: advantage is no root access required, disadvantages are minimal documentation, requires SSH knowledge
TAP : advantage is simpler setup, disadvantage is that it requires sudo access

# SLIRP
runqemu slirp nographic serial
devtool deploy-target nano qemu
# TAP
sudo runqemu nographic serial
$ devtool deploy-target nano root@192.168.7.2

Xen

Overview

Dom0 + Control Domain
- Platform stuff
App VMS
- Windows VM
- Linux VM

Xen hypervisor contains

Scheduler
Memory Mgmt
XSM Access Control
author works for open xt

bitbaking Xen for RPI4

layers

xen support in meta-virtualization
rpi 4 suport in meta-raspberrypi
poky with meta-openembedded
https://lists.yoctoproject.org/g/meta-virtualization/message/5495

conf

MACHINE=”raspberrypi4-64"
DISTRO_FEATURES_append = " virtualization xen";
QEMU_TARGETS = "i386 x86_64 aarch64 arm"

build

bitbake xen-image-minimal

RPI4 Boot and verify xen

xl list Testing basic Xen functionality at the console:

# list running VMs
xl list
# show data about the current hypervisor
xl info
# examine Linux’s Xen device nodes
ls -l /dev/xen
# read the contents of the XenStore tree
xenstore-ls
# see the Linux kernel messages relating to Xen
dmesg | grep Xen
# see the Xen boot messages
xl dmesg

Xen patch series

“dynamic layer” for settings taht are specific to Xen-on-rpi4

Xen beyond Pi

Intel x86-64
- wic tool integration
- simple production of bootable
- wic create directdisk-xen -e xen-image-minimal
Can run inside of runqemu!

Xen Questions

Q: What does “directdisk” refer to when it comes to images? How does this differ from the other IMAGE_FS_TYPES?
- will create image suitable for booting x86-64 type of image.

KVM

packagegroup-kvm-host
- qemu, libvirt, libvirtd, virsh
kvm-binary-image-vessel-package
- “vessel” is avoiding the overloaded meaning of “container”
- inherit bin_package

future work

secure boot
virsh (use a tempalte to create XML)
Launch script to auto boot the guest
Insert ssh keys into guest
Acorn is another type 1 hypervisor
yp-dev-day_virtualization
- gitlab instance with CI

KVM Questions

Xen vs KVM
- xen ~ vsphere, kvm ~ vmware/vmplayer
- Xen is type 1, KVM is type 2
  - you boot a “type 1” first before booting anything else
Q: Has anyone used “cloud-init” with Yocto build image?
- A: we used cloud-init extensively in the meta-openstack days/layer

Reproducible Builds and Hash Equivalence in the Yocto Project

Wed, 01 Jul 2020 00:00:00 +0000

As embedded Linux devices get more complex, many questions arise about the process of constructing these systems and how it can be improved. Can these complex systems be built more efficiently to reduce development time? How can developers be assured that these systems can be built reproducibly to ensure software supply chain integrity? Joshua will highlight two new features in the latest release of the Yocto project specifically designed to address these questions. The first feature is hash equivalence which reduces build times by collectively detecting when tasks have rebuilt unnecessarily and preventing rebuilds of downstream tasks. Joshua will explain how this feature works, how to apply it to your own projects, and what to expect when you do. The second feature is build reproducibility testing, which ensures that the project has binary identical outputs from one build to the next. Joshua will describe how these tests work, what they currently cover within the project, and how to enable them for your own projects. Taken together, these features improve the Yocto project’s unique ability to effectively construct complex Linux embedded systems that are cost effective and secure.

Notes

reproducible-builds

RunQueue

MIA

RunQueue with Hash Equivalence

Now each task also has unihash
breaks task hash from it’s upstream
unihash is tashhash by default.
outhash is calculated. Then stored in hashserver

Trickle down effect

all downstream tasks will still have taskhash updated.
if it’s a trivial task we would like not to have to rebuild the downstream tasks.
if the outhash is the same even if the taskhash changed, then the hashserver will set unihash to a certain value
- Now this unihash propagates down to downstream tasks.

Recipe Updates

Which updates work well with this hash equivalence server?

Library updates & CVE Fixes
Native tool updates

Output Hash Calc

Checksum of all files and metadata that goes into an sstate object
signature files can be found in depsig

Hash Equivalence Server

Reference impl in bitbake
- BB_HASHSERVE = "auto"
can be shared between multiple clients
- Specify server with BB_HASHSERVER
- Only makes sense if you are sharing SSTATE_CACHE

Debugging Hash Equiv.

Can configure logging in local.conf
- BB_LOGCONFG = "logconfig.json"

Future Improvements

Read-only hash server for CI centered workflows
- “read-only” flag for the client
- “read-mostly” flag for the client
better hash equivalence database introspection tools

Reproducable Builds

https://reproducible-builds.org/

Imrpove Hash Equiv. effectiveness
Code archival
Software supply chain
1. verify toolchain is not comprising your target

Build FLow

tool builds almost all the tools it needs
compile native tools from source

Reproducibility QA Test

reproducible build
- available since 3.0 (zeus) release

QA Test

Does an ‘A’ and ‘B’ build
- A uses sstate
- B from scratch
test images
- core-image-minimal
- core-image-full-cmdline
- core-image-sato
test packages formats
- deb
- ipk

What causes reproducibility issues?

build path differences
timestamp differences
host differences
- consider using docker container to reduce issues

diffoscope

report non-reproducible packages in browsable HTML
supports diff inside of archive files
- packages (deb, ipks)
- archives (tar, tar.xz, etc)

Future Imrpovements

NOTE: exttool-sdk provides all the tools required to rebuild

test more packages
- SDK images
- sato images
- world images
test rpm packages
test final rootfs images
test other deployed objectstest native tools
test more architectures
- Currently only tests x86-64
- AArch64 would be the highest priority
reproducible-builds.org recomended these things
- Use DisorderFS to test for file system ordering non-reproducibility
- Use libfaketime to do proper timestamp testing
  - psuedo library needs adjustments
  - provides fake timestamps

Conclusions

Community

Freenode IRC
- #yocto
- #oe
Monthly virtual planning meeting
- First Tuesday of each month at 8:00 AM Pacific Time
Bug triage
- Thursday at 7:30

Contact

JPEWhacker@gmail
Joshua.Watt@Garmin.com

Questions

QA with Speaker

restoring from sstate is called set-scene?
- Yes, bitbake actually executes a task called do_populate_sysroot_setscene in place of of do_populate_sysroot to do the actual restoring from setsecene
Is the hash server part of the sstate cache?
- No. It depends on the sstate cache contents (as I’ll cover in a bit), but it’s a separate entity.
Will the uni-hash always stay as the first ‘equivalent’ task-hask?
- The hash equivalence server currently uses the oldest reported hash as the “canonical” unihash, but the decision is internal to the server, so other things could be done.
What if a recipe doesn’t change it all but only a referenced file (using SRC_URI) which is copied by the recipe into the roots?
- hash equivalence is based on the output of sstate tasks, so if the file changes sstate output, then it won’t be equivalent, but if the state output is the same, than it will be equivalent.
This is true for any of the things that go into a tasks hash (files, variables, etc).
- how does hash equivalence work in a multiconfig scenario when building for both arm32 and aarch64?
Can you elaborate?
- Can the path for your own tests be in a location outside of the poky repo?
Yes. In fact, OE’s QA test core will look in those paths in any layer, so you can simply drop the file in that path relative to your own layer root and it will find it.
- Can you restate the reproducobilty project that recommend DisorderFS and Libfaketime? reproducible-builds.org, and it’s on my references slide

My Questions

Can you comment on ‘universal/’ and on ‘uninative’ with respect to sstate?
- uninative provides consistent native tools. allows you to use the same tools
- really “light” container

Notes

Paths encoded in dwarf symbols in debug symbols becomes problems

Building Bare Metal Toolchains, Crosstool-ng and Yocto Project

Tue, 30 Jun 2020 00:00:00 +0000

Recently I was tasked to create a bare metal toolchain to create software for a variety of embedded processor architectures and configurations. Crosstool-ng is often used to create these toolchains, but Yocto Project SDK builder is capable of doing this as well. This presentation will compare both crosstool-ng and the Yocto Project for this task, include my experiences working with both tools, include Yocto Project configuration information and give the audience an understanding when they may want to use one tool vs the other.

Agenda

crosstool-ng
Yocto Project SDK builder
Experience
Yocto Project Configuration
Recommendation
Questions

crosstool-ng

crosstool-ng is versatile cross toolchain generator
good at reproducable toolchain
`ct-ng list-samples
Runtime relocatable
- can install in one location and mount/run from many
binaries are very specific to the host environments they were built for
- determine which operating system the user is using
- builds on the oldest distro
- Can build from Cygwin
- expects you to have cygwin cross compiler
Configuration is similar to kernel config
Config process will create ctx files
xilinx typically distributed “multilib” versions

Yocto Project SDK builder

Yocto Project

yocto project is not linux specific
- FreeRTOS
- OpenAMP
- Bare Metal
Outputs include
- Run-time images
- SDK / eSDK

This talk is focusing on SDK

Yocto Project SDK

target SDK
- matches operating system runtime environment
defined SDK
- SDK where each component is define to be include in SDK
SDK can be multilib enabled
- multilib are built independently of each other
- slower, but safer approach foor complex configuration
self extracting installation file
built to isolate sdk environment from the host system
- SDK includes its own glibc, and some runtime components
- Don’t need your own cygwin and other environment
installation and runtime locations must be the same
automatic runtime relocation is not supported
- note - this is supported by the plain ‘sdk’

example

oe-init-build-env build-baremeta-tc
DISTRO=xilinx-standalone MACHINE=microblaze-tc bitbake meta-toolchain

slower than crosstool-ng
- more steps to prevent multilib contamination

SDK - distro conf - xilinx-standalone.conf

How to create SDK config (and distro) ?

LIBC_DEPENDENCIES
- newlib-static dev
- libgloss-staticdev
install static library
- clear defaults for distro feature backfill
- virtual runtime init manager
- preffered virtual kernel

# xilinx-standalone.conf
SDK_VERSION = ""xilinx-standalone

SDK - machine conf - microblaze-tc.conf

multilib
- MULTILIB_GLOBAL_VARIANTS
- MULTILIBS += “multilib:libmble”
- tons more
default tune
- DEFAULTTUNE = "microblaze"
- AVAILTUNES
- BASE_LIB
- TUNE_FEATURES
- PACKAGES_EXTRA_ARCH

Experience - part 1

Changes from crosstool-NG

binutils
- set diffferent defaults to match prior toolchain
GCC
- different default to match prior toolchain
- restore some previously disabled newlib options (sysroots)
- only build SDK GCC once, create symlinks for other multilibs (gnu-toolchain-canadian.bb)
newlib/libgloss
- adjust defaults
- workaround issues with multilibs conflicted
- workaround issues where libgloss and newlib dependency wasn’t multilib aware

Compare

cross
- MIA
yocto
- linux sample confs
- no baremetal confs
- not relocatable
- full feature toolchain
- support mingw

Experience - part 2

Xilinx Bare Meta Toolchain

transitions from crosstool-ng to Yocto Project SDK to unify toolchain source and testing
transition was not painless from development perspective
- 2-3 weeks for first few weeks
- took 3 months to iterate the get it stable
lots of questions about multilib configuration
why was this transition and maintence was hard
- people didn’t understand why certain settings were set
- old version of crosstool-ng, people were not keeping up on configuration
initial goals was compatability with fromer crosstool-ng and custom ARM toolchain
- pseudo runtime relocation capable

Bare Metal Multilibs

aarch32-tc.conf
aarch32
- standard ARM (A profile) 32bit instruction set
- multilib config based on GNU/ARM defaults
- 17 multilibs defined
  - custom defined to match GNU/ARM settings
first we got it working
Now we can focus on fine tuning
Custom arm-rm-tc.conf
ARM R/M
- readltime
- microcontroller
aarch64
- aarch64-tc.conf
Microblaze (microblaze-tc.conf)
- common microblaze isntruction set permulation
- custom config

Bare Meta Toolchain COnfiguration

binutils
- disable: gold as LD, gprof, shared
- enable: enable-lto, static, multilib
gcc
newlib/libgloss
- recipes-core
- xilinx baaremeta implements device specific items using libxil, but we use libgloss to provide the framework
- flags
  - enable-newlib-io-c99-format
  - long-long
  - io-flow
  - long-double
  - disable-newlib-supplied-syscall
- libgloss multilib
  - “${MLPREFIX}newlib”

This was one of the major issues:

disable ‘disable-initfini-array’

Lessons

more multilib - longer initial project parse time
microblaze parse time is nearly an hour (48 multilibs)
- devlopment workaround, build for only one multilib
mingw32 builds are sequential with linux version, using shared sstatic-cache
roughly 15 minutes on the same machine for mingw parts

Recommendations

for quick toolchain - use Crosstool-ng
Yocto Project makes sense if you need Cygwin
- less important since windows is moving to linux
Yocto Project take more time and effort
- easy way for common source and configuration switchs
- simplify defect handling and propagation of fixes for systems

Questions

Vivado vitis baremetal tools chians are built with this
How did you converge configuration
- query gcc
- check logs for what arguments were passed
- print config
which release was this against.
- 3.0 zeus
multilib vs multiconfig
- this was largely due to legacy reasons
- one gcc binary to execute

Author

kernel-crashing.org
author: fray

Developing, Building and Testing Your Baremetal Applications Using The Yocto Project and OpenEmbedded Infrastructure

Tue, 30 Jun 2020 00:00:00 +0000

Multiarchitecture SoCs are more widely used, usually containing a large architecture where it can run a full operating system such as Linux, and one or more small architectures designed to run an RTOS or a baremetal application,requiring less hardware resources,leveraging suitable tasks from one to another creating a more efficient product.

While the Yocto Project is well known for its capabilities on creating customized Embedded Linux Distributions,bitbake is also capable of building a toolchain to create Baremetal applications or an RTOS.

In this presentation Alejandro will show how to create baremetal applications using the same flow currently used in OpenEmbedded to develop Linux applications, using recipes, classes and user configuration files, he will show how users from the community can benefit from a single flow, showcasing how to run these applications on QEMU and create tests for them to be automatically tested using the OpenEmbedded infrastructure along with their corresponding Linux distributions.

Anyone from the community interested in learning about baremetal programming or about the Yocto Project might find the contents of this presentation interesting.

Notes

Multiarchitecture Embedded Devices

Azure Sphere
- Secured MCU
- Secured OS
- Cloud Security
- MT3620 Dev Board Diagram

OE Runtime Testing Infra

Testing a linux OS (poky)
TEST_TARGET
- QEMU
- Hardware (simpleremote)
  - Send commands via SSH (TEST_SERVER_IP)
TEST_SUITES
- auto
- cases: ping ssh scp parselogs
- meta/lib/oeqa/runtime/cases

TESTIMAGE_BOOT_PATTERNS in OE

testimage.bbclass
- TESTIMAGE_BOOT_PATTERNS
- TESTIMAGE_BOOT_PATTERNS[send_login_user]
qemurunner.py
- default_boot_pattern
- accepted_patterns

Testimageflow

H: Start booting
T: Login
H: ROOT\n
T: Prints out root username
H: SEND_COMMAND
T: Check for another prompt
H: READ

Baremetal Toolchain on the Yocto Project

MACHINE = "qemuarm64"
TCLIBC = "baremetal"

Then build and run in qemu

bitbake baremetal-helloworld
runqemu

Other tagets

bitbake meta-toolchain
bitbake baremetal-helloworld -c populate_sdk

TCLIBC="newlib" - little more than baremetal

Simple Baremetal HelloWorld on Aarch64

QEMU source code (virt)
UART, RAM, Kernel Address
ARM Developers Manual - UART PL011

What files do you need?

Start Code
Linker Script
Your source code

Example for aarch64

hello_baremetal_aarch64.c
startup_aarch64.s
linkerscript_aarch64.ld

Creating test for FreeRTOS

Overriding Patterns in local.conf
- TEST_SUITES = “freertos_echo”
defined new test cases
- meta-freertos
- [freertos_echo.py]](https://github.com/aehs29/meta-freertos/blob/de4c8530b9bb31b9bbf08124a21a3ffc47d0121f/lib/oeqa/runtime/cases/freertos_echo.py#L1)
These tests run in do_testimage

How to create FreeRTOS test

This is a simple example of using OEQA test framework with a baremetal application.

He then created a very basic C application that emulated the basic prompts for Linux.
Hacky, but a simple way to get an initial test in with a framework focused on Linux testing.
You could possibly get it working for libgloss

Questions

Q: Can a testimage.bb depend on something
A: tests can depend on other recipes. Works with the build system itself.

Yocto | Academic OSS ELC

Yocto Project Dev Day Virtual NA 2020

Program

Agenda

Highly scalable build automation

Solutions

A low cost example

2

Put it all together

Kernel

Kernel Workflow multiple users

Kernel metadata

Security Hardening

Intro

Why

OWASP vunerability list

OE/YP Hardening

Expanding on the FM

meta-security

meta-selinux

meta-sca

Summary

CI/CD

Wrigel the Third

Developer builds

Build Failure Login

Summary

Demo

Questions

Techniques and tools to reduce image size with Yocto Projects

Introduction

“poky-tiny” Distribution

Image analysis tools

Compiler and toolchain optimizations

Compiler Flags

LTO

Package selection

Buildhistory

Refine Image

Slack Channel

User Space, HANDS ON

Packing QA

Questions

DevTool

devtool – past presentations

Why devtool exists?

Devtool has two modes

More on devtool

recipetool

Future

Lab

sidebar – SLIRP versus TUN/TAP

Xen

Overview

bitbaking Xen for RPI4

RPI4 Boot and verify xen

Xen beyond Pi

Xen Questions

KVM

future work

KVM Questions

Reproducible Builds and Hash Equivalence in the Yocto Project

Notes

RunQueue

RunQueue with Hash Equivalence

Recipe Updates

Reproducable Builds

Build FLow

Reproducibility QA Test

Future Imrpovements

Conclusions

Community

Contact

Questions

Building Bare Metal Toolchains, Crosstool-ng and Yocto Project

Agenda

crosstool-ng

Yocto Project SDK builder

Yocto Project

Yocto Project SDK