Embedded Linux Conference (ELC) | Academic OSS ELC

Reproducible Builds and Hash Equivalence in the Yocto Project

Wed, 01 Jul 2020 00:00:00 +0000

As embedded Linux devices get more complex, many questions arise about the process of constructing these systems and how it can be improved. Can these complex systems be built more efficiently to reduce development time? How can developers be assured that these systems can be built reproducibly to ensure software supply chain integrity? Joshua will highlight two new features in the latest release of the Yocto project specifically designed to address these questions. The first feature is hash equivalence which reduces build times by collectively detecting when tasks have rebuilt unnecessarily and preventing rebuilds of downstream tasks. Joshua will explain how this feature works, how to apply it to your own projects, and what to expect when you do. The second feature is build reproducibility testing, which ensures that the project has binary identical outputs from one build to the next. Joshua will describe how these tests work, what they currently cover within the project, and how to enable them for your own projects. Taken together, these features improve the Yocto project’s unique ability to effectively construct complex Linux embedded systems that are cost effective and secure.

Notes

reproducible-builds

RunQueue

MIA

RunQueue with Hash Equivalence

Now each task also has unihash
breaks task hash from it’s upstream
unihash is tashhash by default.
outhash is calculated. Then stored in hashserver

Trickle down effect

all downstream tasks will still have taskhash updated.
if it’s a trivial task we would like not to have to rebuild the downstream tasks.
if the outhash is the same even if the taskhash changed, then the hashserver will set unihash to a certain value
- Now this unihash propagates down to downstream tasks.

Recipe Updates

Which updates work well with this hash equivalence server?

Library updates & CVE Fixes
Native tool updates

Output Hash Calc

Checksum of all files and metadata that goes into an sstate object
signature files can be found in depsig

Hash Equivalence Server

Reference impl in bitbake
- BB_HASHSERVE = "auto"
can be shared between multiple clients
- Specify server with BB_HASHSERVER
- Only makes sense if you are sharing SSTATE_CACHE

Debugging Hash Equiv.

Can configure logging in local.conf
- BB_LOGCONFG = "logconfig.json"

Future Improvements

Read-only hash server for CI centered workflows
- “read-only” flag for the client
- “read-mostly” flag for the client
better hash equivalence database introspection tools

Reproducable Builds

https://reproducible-builds.org/

Imrpove Hash Equiv. effectiveness
Code archival
Software supply chain
1. verify toolchain is not comprising your target

Build FLow

tool builds almost all the tools it needs
compile native tools from source

Reproducibility QA Test

reproducible build
- available since 3.0 (zeus) release

QA Test

Does an ‘A’ and ‘B’ build
- A uses sstate
- B from scratch
test images
- core-image-minimal
- core-image-full-cmdline
- core-image-sato
test packages formats
- deb
- ipk

What causes reproducibility issues?

build path differences
timestamp differences
host differences
- consider using docker container to reduce issues

diffoscope

report non-reproducible packages in browsable HTML
supports diff inside of archive files
- packages (deb, ipks)
- archives (tar, tar.xz, etc)

Future Imrpovements

NOTE: exttool-sdk provides all the tools required to rebuild

test more packages
- SDK images
- sato images
- world images
test rpm packages
test final rootfs images
test other deployed objectstest native tools
test more architectures
- Currently only tests x86-64
- AArch64 would be the highest priority
reproducible-builds.org recomended these things
- Use DisorderFS to test for file system ordering non-reproducibility
- Use libfaketime to do proper timestamp testing
  - psuedo library needs adjustments
  - provides fake timestamps

Conclusions

Community

Freenode IRC
- #yocto
- #oe
Monthly virtual planning meeting
- First Tuesday of each month at 8:00 AM Pacific Time
Bug triage
- Thursday at 7:30

Contact

JPEWhacker@gmail
Joshua.Watt@Garmin.com

Questions

QA with Speaker

restoring from sstate is called set-scene?
- Yes, bitbake actually executes a task called do_populate_sysroot_setscene in place of of do_populate_sysroot to do the actual restoring from setsecene
Is the hash server part of the sstate cache?
- No. It depends on the sstate cache contents (as I’ll cover in a bit), but it’s a separate entity.
Will the uni-hash always stay as the first ‘equivalent’ task-hask?
- The hash equivalence server currently uses the oldest reported hash as the “canonical” unihash, but the decision is internal to the server, so other things could be done.
What if a recipe doesn’t change it all but only a referenced file (using SRC_URI) which is copied by the recipe into the roots?
- hash equivalence is based on the output of sstate tasks, so if the file changes sstate output, then it won’t be equivalent, but if the state output is the same, than it will be equivalent.
This is true for any of the things that go into a tasks hash (files, variables, etc).
- how does hash equivalence work in a multiconfig scenario when building for both arm32 and aarch64?
Can you elaborate?
- Can the path for your own tests be in a location outside of the poky repo?
Yes. In fact, OE’s QA test core will look in those paths in any layer, so you can simply drop the file in that path relative to your own layer root and it will find it.
- Can you restate the reproducobilty project that recommend DisorderFS and Libfaketime? reproducible-builds.org, and it’s on my references slide

My Questions

Can you comment on ‘universal/’ and on ‘uninative’ with respect to sstate?
- uninative provides consistent native tools. allows you to use the same tools
- really “light” container

Notes

Paths encoded in dwarf symbols in debug symbols becomes problems

Finding Sources of Latency on your Linux System

Tue, 30 Jun 2020 00:00:00 +0000

In today’s computer systems the level of complexity has risen such that when a task or response to an event takes longer than expected, it is not easy knowing what the culprit is. The Linux operating system contains several utilities that allows a user to see where things may be held up. This talk will cover many of these utilities and briefly explain how to use them. From the hardware latency detector to the latency tracers. It will also discuss the new synthetic event interface that allows users to create a histogram on the time it takes any two events to occur.

I have previously discussed the latency tracers but they are soon to be obsolete, and the new synthetic event interface will be replacing them. There is still a lot more development going on with the synthetic events and this talk will briefly go over some of the new features that are yet to arrive.

Notes

Latency from Hardware
- System Mangement Interrupt (SMI)
Cache miss
branch prediction
Hyper-threading
Hardware Latency detector (HWLAT_TRACER)
fedora 31 not debian
tracefs

Measuring latency from interrupts

Linux Stateless Video Decoder Support

Tue, 30 Jun 2020 00:00:00 +0000

While it has been under development for years, the support for video CODEC accelerators has gain a lot of traction in past year. A formal specification has now been merge into Linux Media subsystem and staging control APIs and drivers now exist. This allow for blob free HW accelerated decoding on popular SoC like Allwinner, i.MX8 and Rockchip.

In this talk, Nicolas will give an overview of the decoding process using such hardware accelerators along with an overview of the user space API and how it’s used within multimedia frameworks. Nicolas will also explain the relation between this and accelerators attached to modern GPUs. This presentation would not be complete without mentioning the development of FFMPEG and GStreamer native support and their major role in the development of the the new Open Source drivers.

This talk is addressed to multimedia enthusiasts and developers curious about video decoding and the upstream effort effort to make that available to users.

Notes

State-full Decoder

Begining of Linux Codecs

What is state-full?

stateful is a blackbox
blackbox gets bitstream

V4L2 M2M

output -> VPU -> capture

V4L2 output queue is used for the bitstream
capture queue is used for the decoded pictures
capability to seek and end stream

Pro/cons

pro: minimal per codec code needed
con: require a firmware
con: harder to multiplex

CODA Driver

Enabling iMX51 and iMX6
reversed enginerring from imx binary blobs

Begining of State-Less

2015

Rockchip
new type of CODEC hardware
Rockchip VDPU

What is State-Less

no processor
expose “accelarators”
1. inputs:
- reference
- bitstream
- parameters
1. accerator
1. outputs
- pictures
- may not be in order

What are GPU?

Crafting command stream is HW specific
only implement in userspace drivers (mesa)
commands are schedule by the GPU driver
multiple GPU hardware in the same application remains tedius

This is not what google did!

V4L2 M2M + Request (2015)

Instead, extend stateless: V4L2 M2M + Request

Added components

CODEC Controls
Request
Topology
Request API
- way to queue parameters, controls, and bitstream buffers
- added to media controller API

H.264

NALU Sequence
- SPS, PPS, IDR Slice, P Slice, B Slice
- P previous decoded pictures
- both past and future references
Annex B NALU
- Start-Code
- HDR, Paylod
AVCc NALU
- Size
- HDR, Payload

Decoding Process

MIA… Missed one slide
Fill SPS/ PPS, Decode Parameters, Slice parameters
Modify reference list
Decode the slide/frame
Do DPB management as per spec
- Display Picture management
Output frames that could be re-ordered

V4L2

Allocate a Request (an FD)
set per frame
queue a v4l2_buffer
queue the request
poll the request FD for completion

Continue in time

MediaTek VPU (State-full)

tiled output only (requires HW converter)

Qualcomm Venus

Stateful
lots of CODECS

Upstreaming Stalled

Could not settle on the Request/Job API
Low knowledge of CODEC decoing process by the linux-media maintainers
only one hardware to test the API design (Only rockchip)
No formal specific (not that state-full CODEC had one either)

Allwinner VPU support Kickstarter by Bootlin (2018)

Request API is finalized
MPEG2 Support in Staging
H.264 support was progressing (but only sliced based)
reverse engineered from binary userspace
VAAPI userspace drivers

2019

formal specification was merged
RK3288 driver was mainlined
H264, VP8, HEVC uAPI added as staging control API
RK3288 driver was renamed?

The Hantro Driver

I.MX8M Quad, using Hantro G1/G2
Registry compatible with the RK3288
Give email and you get specification

Hantro Company

Hantro Visibility Better
on2 technology
Google
- VP8 and VP9 royalty free
VeriSilicon

Other story of API rename

stmmac is an ethernet driver, that was thought to be STM design
later found to be DesignWare design, shared across numerous SoC
Since API was released it was unable to be renamed

Testing and Fixing

LibreELEC - just enough OS for KODI
- FFMPEG support
- bug fixing
- interlaced Content Support

2020

RK3399 JPEG, MPEG2, H.264, and VP9 support
Start having base class for Stateless
- DXVA2 and NVDEC support
VA V4L2 Request driver was abandoned

demo

linux kernel 5.7rc2
gstreamer has tool called ‘gst-build’ built on mesin
- allow you to run plugin
- gst-play-1.0
video layers are an overlay
prerolling.
three different streams
one two iMx8

Questions

performance stateless vs state-full
- have to wake up processor more
- more control over queuing
converting them to state-less is complex

Robot Operating System (ROS) 2 - How Open Source Software and Linux is Powering the Next Generation of Robotics

Tue, 30 Jun 2020 00:00:00 +0000

Robot Operating System (ROS) 2 - How Open Source Software and Linux is Powering the Next Generation of Robotics

Katherine Scott, Open Robotics

Robot Operating System (ROS) is a set of mature free and open source software packages used by the robotics community to program robots. Recently ROS had its first major version upgrade in a decade to ROS 2. This new version introduces a number of changes and new features, many of which simplify the development process and allow users to build virtual robots without ever touching a piece of hardware. ROS 2 is the version of ROS that supports Python 3 out of the box and has new additions that allow for high-fidelity simulations. Simulation makes learning advanced robotics topics more accessible, as you no longer need to outfit a robotics lab and you can simulate cutting edge hardware for free. In this talk we’ll cover the state of the open source robotics ecosystem and cover the changes made in ROS 2 to improve its simplicity, stability, safety, and security. These features, along with enhanced embedded hardware support, are helping ROS become the de facto standard for the developers of autonomous vehicles, drones, industrial, and agricultural robots. This talk will give the Linux community and update on ROS ecosystem and a starting point for learning how to use it.

Notes

ROS is a lyer on top of Linux for robots
metrics.ros.org

Windows

works on Windows
vscode

Intro

Software before times

research grade
proprietary
works on Windows 95
system integrators
vendors
bespoke
hardware controllers
Feed forwards

Catalyst

2004 competition for autonomous vehical

Input to innovation

flexible robots
IPO
FOSS
Willow Garage - 2007 Poof FOOS for robots

Three projects results

opencv
pcl
ROS

Two Robots born

Willow: Shared World Class Research
2013 Willow Folders but OS lives on

Ethos of Global Research Community

Small, Simple, Composable Utilities
Don’t Reivent the Wheel
Non-Exclusive (polyglot, packagin infrastructure)
Inclusive (in addition to, not in place of)
Freedom

Out of Willow was born:

OpenRobotics
ROS Industrial

ROS Philosophy

Federation over Centralation

Plus list from above.

Small, Simple, Composable Utilities
Don’t Reivent the Wheel
Non-Exclusive (polyglot, packagin infrastructure)
Inclusive (in addition to, not in place of)
Freedom

Order of topics

Don’t Reinvent

Dont Reinvent

Multiprocess Control

Components
- Sensors
- Control
- actuators
nodes are process encapsulation between languages
Imagine
- C++ Python programs all start concurently
- with hardware
- lots of config
- control flow based on config
- What would your shell script file look like?
- ROS nodes and launch files take care of this
- Tooling to start them.

Topics

pub/sub bus
ROS nodes communicate over topics
Cross language serialation primitives that predate protobuff, rabbitMQ, etc.
Standard messages allow
- plug and play
- loose coupling
Online introspection
quality of service capabilities
build as an abstraction
- DDS implementation, ensures security, robustness
- implementation of pub/sub is pushed off to other implementation

RVIZ

Robots make lots of data
- much of it is live
- with 3D context
- annotation
RVIZ is your GUI

Bags

standard serialzation library for transport also works for disk storage
- MIA
bags are amazing
- ML/CV/DL
- unit tests
- collaborators
- Black Box
collect on robot and transfer elsewhere for analysis
De-facto standard for sharing data
- Ford Autonomous Vehical Data
http://wiki.ros.org/ROS/Tutorials/Recording%20and%20playing%20back%20data

Services/Actions

Services: synchronous behavior API
Action: async behavior
- Handles request negotation
- with preemption
- with status callbacks
Callback driven API
build robot behvaiors, fast

Geometry/Kinematics

Hard robot problems
- where is everything
- where am I
- how to get A to B
- I can see X, how to graph it
- Many more
URDF: Universal Robot Description Format
- Defines 3D Geometry of a Robot
- Works with your CAD format
- gives a robot an idea of its shape and size
TF2 Transform 2
- transform coordinates from different frames
- API driven, minimal math

Forward/Inverse Kinematics

MoveIt

SLAM/Navigation

Path planning

Simulation

Robots are expensive, dangerous, slow
sharing a robot is hard
testing is hard
what if you don’t need robot?
What if you could have robot VM?

Simulation - Gazebo

build a virtual robot
simulate Physics, robot, and sensors
Plays well with ROS2
Allows yo build test debug from your desk

ROS: Getting to Plug and Play Hardware

Hardware Ready
Bridge between ROS 2 DDS layer and ROS

Community not just Code

ros.org
discourse.ros.org
answers.ros.org

ROS 2 Foxy

6th Ros 2 Release
First major LTS
Start now 3 year of patches and supports

I want to learn!

DARPA SubT Simulator has great tutorial
Autoware has Autonomy class
- Prepackaged docker container to run simulation
- awf-course
ROS 2 Docs
TurtleBot3

Contact

rosorg
kscottz
org.org
openrobics

Questions

Q: Gazebo included?
A: No
Q: I work on embedded projects that run on custom hardware for fancy PTZ cameras. I would love to convience our Python app developers how/if ROS could be leveraged. Where do you think I should start? Are their any consultants that could help analysis the specific use case?
A: This is something we are missing. She views robots broadly. MicroROS. Developing concepts for hardware. hardware interface framework.
Start with simulation

Using MIPI DSI as Main Display Interface

Tue, 30 Jun 2020 00:00:00 +0000

Using MIPI DSI as Main Display Interface

Marcel Ziswiler, Toradex AG

The MIPI Display Serial Interface (DSI) is the de-facto standard display interface featured by modern higher-end SoCs. Lacking the long-term availability of discrete MIPI DSI display panels most embedded systems rely on bridge chips converting to more common display interfaces like parallel RGB, LVDS, (e)DP or HDMI. Our generic system concept relies on DSI adapter boards integrating various such bridge chips. After introducing the Linux DSI subsystem this talk concentrates on the auto-detection of such DSI adapters based on parametrisation stored in EEPROMs. The U-Boot boot loader reads the EEPROM contents and chooses the applicable device tree overlay to be applied. The concept of DDC/EDID with hot-plug detect vs. a custom model-specific parametrisation is discussed. The talk continues covering the integration aspect of such DSI bridge chips within the Linux DRM stack. Various limitations of the DSI subsystem and possible solutions are discussed. The DSI bridge chip ecosystem is covered and we look into possible reasons only very few bridge chips are actually supported in mainline so far. The talk concludes with a live demo of our DSI auto-detection implementation.

Tordadex - swiss. Embedded. Computing

Intro

MIPI
U-Boot FIT Image
LIve Demo

MIPI Display Serial Interface (MIPI DSI)

Spec by MIPI Alliance
Low power (LP) mode or high speed (HS) mode
- HS is ussually one direction
MIPI DSI Spec initial version May 2006
MIPI DSI-2 Spec supports Ultra high definition (4k and 8k)
Physical Layer
- 1.0 Gbps/lane (1.01), 1.5, 2.5, 4.5 Gbps/lane (D-PHY 2.0)
- MIPI DCS - MIPI Display Command Set
- Incorportates Display Stream Compression (DSC)
  - Standard from VESA
- de-facto standard display interface feature by modern high-end SoC

Verdin products

Verdin MIPI DSI Display Adapter System Design

generic system concept
display adapter boards integrating various bridge chips
DSI Mezzanine Connector
- MIPI DSI: 1 clk + 4 data
- GPIO ( BLK1_EN, Touch Interrupt)
2 x I2C bridge chip + DDC/EDID
PWM backlight
I2S for optional audio
Generic system control signals
- PWR_EN_MOCI
- SLEEP_MOCI#
- RESET_MOCI#

Verdin iMX8M Mini

NXP i.MX 8M Mini SoC
Single display controller, LCDIF
MIPI DSI output w/ 4 data
MIPI D-PHY 1.2 max data per lane 1.5 GPS
Resoution up to 1920x180p60 and 1800x1200p60

Verdin DSI to HDMI Adapter

Lontium Se
HDMI V1.4 1080p, 8bit RGB up to 60Hz
ST HDMI2C1-14HD ESD prtect and signal conditiong
type A standard HDMI connector

Verdin DSI to LVDS Adatper

SN65DS184 MIPI DSI to dual link LVDS bridge
signle/dual-lane LVDS up to 1920x1200/1377x768
LVDS and touch connector compatible with Capacitive Touch Display 10.1” LVDS

Intro Linux DSI Subsystem

DRM MIPI DSI Core common logic helpers to deal with MIPI DSI peripherals

DRM Panel COre - Generic LVDS Panel (panel-lvds.c)
DRM Bridge Core TI SN65DS184 DSI to LVDS Bridge - (ti-sn65dsi84.c)
MIPI DSI Core - Northwest Logic MPI DSI Host COntroller (nwi-dsi.c)
DRM CORE - Display Engine Driver

DSI bridge chip integration

How does DSI bridge chip integrate with that?

DRM bridge
DRM connector
I2C device
MIPI DSI device

DSI Bridge Chip Integration Pitfalls

availability of super secret data sheets
- chinese vendors hide their data sheets
ancient downstream or bare skeleton drivers only
lots of hard-coded parmeters
link bring-up sequences not well documented
- lots of trial and error
Drivider frequency limitations on either controller side, bridge side or both
- may require running outside of recommended range

Bridge Chips in our current Adapters

How about Bridge Chips used in our designs current Adapters?

Lontium LT8912B MIPI DSI to HDMI bridge
Adopted driver from Rockchip Linux on Github
- Forward ported to later DRM API
- Reword driver to be proper I2C device
- full register set from Lontium psuedo code driver
- imrpoved regmap integration
- I2C sub addresses
- added regular I2C based DDC/EDID handling
- Added GPIO based hot-plug detection
  - Needed for I2C GPIO expander support
  - hot plug detect GPIO handling crashed GPIO exapdnder
- bus format was not properly set
not pretty but it works
Further clean-up and upstreaming pending
Texas Instruments SN65DSI84 MIPI DSI to dual-link LVDS bridge
Downstream driver taken from a patch in CompLab Yocto Meta Layer on Github

DSI Bridge Chip Ecosystem

Vendors still reluctant to maining drivers
Few mainline supported bridge chips
Few examples to copy from
procedure of actual silicon may be difficult
conformance of MIPI DSI host IP vs bridge chip silicon

Bridge Chips supported in Mainline

driver/gpu/drm/bridge
Differentiate between
- SoC internal IP
- discrete external bridge chips
- directly connected panels
SoC internal IP
- NXP i.MX 8 series
discrete external bridge chips
- ADV7533
- Parade PS8640 MIPI DSI to eDP converter
- SN65DSI86 DSI to eDP Bridge
- TC358764 DSI/LVDS Bridge
- TC358768AXBG/TC358778XBG MIPI DSI bridge chips
directly connected panels
- Rasperry Pi 7In touch. Toshiba TC358762 DSI to DPI aka parallel RGB bridge

Auto-Detection of DSI Adapter Base on EEPROM Contents

idea 1
- regular device tree: setting bridge stat to disable vs okay
- device graph: linking endpoint and remote-endpoint nodes?
- full flexbility requires device tree overlays
idea 2
- storing device tree overlay in EEPROM
- simple DTBO may be below 1kb more complex ones quickly more than 2kb
compromise. Store configuration block

U-Boot read EEPROM Contents and Selecting Applicatble Device Tree Overlay

Generalized ConfigBlock Handling from NAND/eMMC to EEPROM
table with product ID to device treee overlay file name mapping
Distroboot script to apply device tree overlays based on auto-detection as well as overlays.txt file
HDMI may do hot-plug detect
DDC/EDID
MIA…

DT Overlays

Verdin DSI

Verdin DSI to HDMI Adapter
- fragment for I2C bus
- fragment for I2C bus where you have bridge chip on
Verdin DSI to LVDS Adapter

Uboot FIT Image

board specfici device trees
display adapter specific device tree overlays
FIT image allows conveninet packing of Linux kernel binary toegether with various device trees, device tree overlays and/or ramdisk
booted as…
fdt_module is deduced form the EEPROM on the module. fdt_board from one on the carrier board. display_adapter_dtbo from one of the display adapters

mkimage -l tezi.itb
...

Device Tree Overlay Pitfalls

complex device tree overlays may require symbols
DTC_FLAGS='-@'
if reference device nodes with hex addresses.
- CASE SENSITIVE!!!
- use lower case!!!
troubleshooting
- Add dtc on target. opkg install dtc
- Use dtc -l fs on target and dump /proc/device-tree

Conclusion

Demo - DSI Auto-Detection Demo

Questions

Q: Used overlay stuff in u-boot
Q: Is Low power signaling Single ended?
Q: Can the end pane llike LVDS support partial update? How will the bridge propogate the presence/absence of particular DCS feature to Linux DRM driver subsytem?
Q: did you consider doing the overlays in linux kernel?
Q: Why is DRM required with DSI?
Q: Why are the vendors reluctant with regards to mainlining (or even docs)?
Q: Keep the conversation going. Please visit the #2-track-embedded-linux on our Slack Workspace after the session ends