Date: February 23, 2021
This document describes the testing Linaro will carry out when making a release of the Linaro OpenEmbedded RPB release for Qualcomm™ Robotics RB5 platform.
Linaro provide releases for the following platforms from Qualcomm:
Board | Board Revision |
---|---|
RB5 | DVT |
Linaro uses LAVA test definition syntaxt to create and manage automated and manual tests. To learn more about LAVA test definition, you can refer to LAVA Documentation, in particular: Writing a Lava-Test Test Definition 1.0ΒΆ.
Linaro test cases are archived in the upstream Linaro Test Definitions repository here: https://github.com/Linaro/test-definitions/.
Linaro runs automated testing in the LAVA lab.
This section describes the automated tests used from the Linaro Test Definitions repository.
Run basic smoke tests.
Test definition: automated/linux/smoke/smoke.yaml
Test scope: functional
Parameter | Value |
---|---|
TESTS | pwd, uname -a, ip a, vmstat, lsblk |
SKIP_INSTALL | False |
Basic USB test for Linaro Ubuntu images. The test examines all available USB devices and prints supported USB protocols and speed.
Test definition: automated/linux/usb-smoke/usb-smoke-test.yaml
Test scope: functional
Parameter | Value |
---|---|
SKIP_INSTALL | false |
Simple test that checks whether gcc compilation works
Test definition: automated/linux/toolchain-smoke/toolchain-smoke.yaml
Test scope: functional
Parameter | Value |
---|---|
STATIC | False |
SKIP_INSTALL | False |
Device tree test to check the folder structure. The test verifies that device-tree is available and contains model name of the board.
Test definition: automated/linux/device-tree/device-tree.yaml
Test scope: functional
Collect the information in /proc/meminfo after boot.
Test definition: automated/linux/meminfo/meminfo.yaml
Test scope: functional
dd write/read speed test.
Test definition: automated/linux/dd-wr-speed/dd-wr-speed.yaml
Test scope: performance
Parameter | Value |
---|---|
PARTITION | |
FS_TYPE | |
ITERATION | 5 |
SKIP_INSTALL | false |
SysBench is a modular, cross-platform and multi-threaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load. Current features allow to test fileio, cpu, memory, threads, mutex and oltp.
Test definition: automated/linux/sysbench/sysbench.yaml
Test scope: performance
Parameter | Value |
---|---|
NUM_THREADS | 4 |
TESTS | cpu memory threads mutex |
SKIP_INSTALL | false |
Use ‘hdparm -t’ to perform timings of block device reads for benchmark and comparison purposes.
Test definition: automated/linux/device-read-perf/device-read-perf.yaml
Test scope: performance
Parameter | Value |
---|---|
DEVICE | |
SKIP_INSTALL | False |
Hackbench is both a benchmark and a stress test for the Linux kernel scheduler. It’s main job is to create a specified number of pairs of schedulable entities (either threads or traditional processes) which communicate via either sockets or pipes and time how long it takes for each pair to send data back and forth.
Test definition: automated/linux/hackbench/hackbench.yaml
Test scope: performance
Parameter | Value |
---|---|
ITERATION | 1000 |
TARGET | host |
DATASIZE | 100 |
LOOPS | 100 |
GRPS | 10 |
FDS | 20 |
PIPE | false |
THREADS | false |
Linpack benchmark calculates fLoating point operations per second(FLOPS).
Test definition: automated/linux/linpack/linpack.yaml
Test scope: performance
Parameter | Value |
---|---|
ARRAY_SIZE | 200 |
Use openssl speed command to test the performance of cryptographic algorithms
Test definition: automated/linux/openssl/openssl-speed.yaml
Test scope: performance
Parameter | Value |
---|---|
SKIP_INSTALL | False |
STREAM benchmark is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s). This test case run STREAM on a single processor.
Test definition: automated/linux/stream/stream-uniprocessor.yaml
Test scope: performance
pi_stress checks Priority Inheritence Mutexes and their ability to avoid Priority Inversion from occuring by running groups of threads that cause Priority Inversions.
Test definition: automated/linux/pi-stress/pi-stress.yaml
Test scope: functional
Parameter | Value |
---|---|
DURATION | 5m |
MLOCKALL | false |
RR | false |
BACKGROUND_CMD |
Test power management (PM-QA). Currently, the test runs cpufreq, cpuidle, cpuhotplug, thermal and cputopology by default and individual test can be run by setting TESTS parameter in test job definition too. Test case documentation available in wiki: https://wiki.linaro.org/WorkingGroups/PowerManagement/Resources/TestSuite/PmQaSpecification
Test definition: automated/linux/pm-qa/pm-qa.yaml
Test scope: functional
Parameter | Value |
---|---|
TESTS | cpufreq cpuidle cpuhotplug cputopology |
RELEASE | pm-qa-v0.5.2 |
SKIP_INSTALL | false |
Docker smoke test. When docker not pre-installed, docker-ce will be installed from upstream on supported distributions.
Test definition: automated/linux/docker/docker.yaml
Test scope: functional
Parameter | Value |
---|---|
IMAGE | aarch64/ubuntu |
Basic WLAN smoke tests
Test definition: automated/linux/wlan-smoke/wlan-smoke.yaml
Test scope: functional
Parameter | Value |
---|---|
DEVICE | wlp1s0 |
BOOT | enabled |
Basic HCI smoke tests
Test definition: automated/linux/hci-smoke/hci-smoke.yaml
Test scope: functional
Parameter | Value |
---|---|
BOOT | auto |
DEVICE | hci0 |
Run bootrr https://github.com/andersson/bootrr
Test definition: automated/linux/bootrr/bootrr.yaml
Test scope: functional
Parameter | Value |
---|---|
BOARD | qcom,qrb5165-rb5 |
SKIP_INSTALL | False |
Glmark2 test for Linux is an OpenGL 2.0 and ES 2.0 benchmark. The Glmark2 test suite is used to measure different aspects of OpenGL (ES) 2.0 performance like video, graphics and display.
Test definition: automated/linux/glmark2/glmark2.yaml
Test scope: performance
Parameter | Value |
---|---|
DISPLAY | :0 |
GLMARK_BIN | glmark2 |
Piglit is an open-source test suite for OpenGL implementations. For more information: https://piglit.freedesktop.org/
Test definition: automated/linux/piglit/piglit.yaml
Test scope: functional
Parameter | Value |
---|---|
DISPLAY | :0 |
PIGLIT_SOURCE_DIR | /usr/lib64/piglit |
OPTIONS | -x streaming-texture-leak -x glx |
SUITE | tests/quick.py |
USE_XVFB | False |
IGNORE_TESTS_REPO | |
IGNORE_TESTS_BRANCH | master |
IGNORE_TESTS_FILE |
A suite of tools to run integration tests for Gstreamer, For more information: https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-validate/html/
Test definition: automated/linux/gst-validate/gst-validate.yaml
Test scope: functional
Parameter | Value |
---|---|
MAIN_DIR | /gst-validate |
OPTIONS | -nd -v -j 1 |
GST_INTEGRATION_SUITES | http://testdata.validation.linaro.org/gst-validate/gst-integration-testsuites_1.12.tar.gz |
GST_IGNORE_TESTS_REPO | |
GST_IGNORE_TESTS_BRANCH | master |
GST_IGNORE_TESTS_FILE |
OpenEmbedded Package tests support (ptest) The pacakge tests from OpenEmbedded enables the support to run package unittests into the target device. For more information: https://wiki.yoctoproject.org/wiki/Ptest
Test definition: automated/linux/ptest/ptest.yaml
Test scope: functional
Parameter | Value |
---|---|
TESTS | |
EXCLUDE |
Run stress_ng, stress_oom and stress_network tests
Test definition: automated/linux/24h-stress-test/24h-stress-test.yaml
Test scope: functional
Parameter | Value |
---|---|
TEST | stress_ng |
DURATION | 28800 |
INTERFACE | eth0 |
LINK | http://192.168.3.1/testdata/stress-network.img |
MD5 | e5c834fbdaa6bfd8eac5eb9404eefdd4 |
Run stress_ng, stress_oom and stress_network tests
Test definition: automated/linux/24h-stress-test/24h-stress-test.yaml
Test scope: functional
Parameter | Value |
---|---|
TEST | stress_ng |
DURATION | 28800 |
INTERFACE | eth0 |
LINK | http://192.168.3.1/testdata/stress-network.img |
MD5 | e5c834fbdaa6bfd8eac5eb9404eefdd4 |
Run stress_ng, stress_oom and stress_network tests
Test definition: automated/linux/24h-stress-test/24h-stress-test.yaml
Test scope: functional
Parameter | Value |
---|---|
TEST | stress_ng |
DURATION | 28800 |
INTERFACE | eth0 |
LINK | http://192.168.3.1/testdata/stress-network.img |
MD5 | e5c834fbdaa6bfd8eac5eb9404eefdd4 |
v4l2 Compliance test suite
Test definition: automated/linux/v4l2/v4l2-compliance.yaml
Test scope: functional
Parameter | Value |
---|---|
VIDEO_DEVICE | /dev/video0 |
VIDEO_DRIVER |
v4l2 Compliance test suite
Test definition: automated/linux/v4l2/v4l2-compliance.yaml
Test scope: functional
Parameter | Value |
---|---|
VIDEO_DEVICE | /dev/video0 |
VIDEO_DRIVER |
The drawElements Quality Program (deqp) Testing allows to test OpenGL API’s, uses OpenGL ES CTS from https://github.com/KhronosGroup/VK-GL-CTS.
Test definition: automated/linux/deqp-runner/deqp-runner.yaml
Test scope: functional
Parameter | Value |
---|---|
DISPLAY | :0 |
EGL_PLATFORM | surfaceless |
DEQP_BIN | deqp-gles2 |
DEQP_CASES | |
DEQP_FAIL | |
DEQP_EXCLUDE | |
DEQP_RUNNER_OPTIONS | —compact-display false —shuffle false —allow-flakes true |
DEQP_RUNNER_JOBS | |
DEQP_OPTIONS | —deqp-surface-width=256 —deqp-surface-height=256 —deqp-surface-type=pbuffer —deqp-gl-config-name=rgba8888d24s8ms0 —deqp-visibility=hidden —deqp-shadercache=disable |
In addition Linaro runs additional manual tests, when they cannot be automated using the Linaro LAVA lab.
This section describes the manual tests used from the Linaro Test Definitions repository.
Test if the serial connection to the board works
Test definition: manual/generic/serial-console.yaml
Test scope: functional
Test if HDMI video output works well
Test definition: manual/generic/primary-video-out-hdmi.yaml
Test scope: functional
Test input switching from Monitor.
Test definition: manual/generic/hdmi-input-switch.yaml
Test scope: functional
Check if switching HDMI resolution on monitor works from Terminal.
Test definition: manual/generic/linux/hdmi-resolution-switching-cmd.yaml
Test scope: functional
Check if switching HDMI resolution on monitor works.
Test definition: manual/generic/linux/hdmi-resolution-switching.yaml
Test scope: functional
Basic test for USB keyboard functionality
Test definition: manual/generic/linux/usb-keyboard.yaml
Test scope: functional
Basic test for USB keyboard functionality
Test definition: manual/generic/linux/usb-storage.yaml
Test scope: functional
Test if USB camera works on Linux
Test definition: manual/generic/linux/usb-camera.yaml
Test scope: functional
Test if WiFi works well from UI
Test definition: manual/generic/linux/wifi.yaml
Test scope: functional
Test if Bluetooth scan, pairing and file transfer works.
Test definition: manual/generic/linux/bluetooth-scan.yaml
Test scope: functional
Test if Bluetooth scan, pairing and file transfer works.
Test definition: manual/generic/linux/bluetooth-pair.yaml
Test scope: functional
Test if Bluetooth audio works well
Test definition: manual/generic/linux/bluetooth-audio.yaml
Test scope: functional