Test Plan for Linaro OpenEmbedded RPB release for Qualcomm™ Robotics RB5 platform

Date: April 10, 2021

Executive Summary

This document describes the testing Linaro will carry out when making a release of the Linaro OpenEmbedded RPB release for Qualcomm™ Robotics RB5 platform.

Hardware Platforms

Linaro provide releases for the following platforms from Qualcomm:

Tests definitions

Linaro uses LAVA test definition syntaxt to create and manage automated and manual tests. To learn more about LAVA test definition, you can refer to LAVA Documentation, in particular: Writing a Lava-Test Test Definition 1.0¶.

Linaro test cases are archived in the upstream Linaro Test Definitions repository here: https://github.com/Linaro/test-definitions/.

Automated Tests

Linaro runs automated testing in the LAVA lab.

This section describes the automated tests used from the Linaro Test Definitions repository.

1. smoke-tests-basic

Run basic smoke tests.

Test definition: automated/linux/smoke/smoke.yaml
Test scope: functional

2. usb-test-basic

Basic USB test for Linaro Ubuntu images. The test examines all available USB devices and prints supported USB protocols and speed.

Test definition: automated/linux/usb-smoke/usb-smoke-test.yaml
Test scope: functional

3. toolchain-smoke

Simple test that checks whether gcc compilation works

Test definition: automated/linux/toolchain-smoke/toolchain-smoke.yaml
Test scope: functional

4. device-tree

Device tree test to check the folder structure. The test verifies that device-tree is available and contains model name of the board.

Test definition: automated/linux/device-tree/device-tree.yaml
Test scope: functional

5. meminfo

Collect the information in /proc/meminfo after boot.

Test definition: automated/linux/meminfo/meminfo.yaml
Test scope: functional

6. dd-speed-test

dd write/read speed test.

Test definition: automated/linux/dd-wr-speed/dd-wr-speed.yaml
Test scope: performance

7. sysbench

SysBench is a modular, cross-platform and multi-threaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load. Current features allow to test fileio, cpu, memory, threads, mutex and oltp.

Test definition: automated/linux/sysbench/sysbench.yaml
Test scope: performance

8. device-read-perf

Use ‘hdparm -t’ to perform timings of block device reads for benchmark and comparison purposes.

Test definition: automated/linux/device-read-perf/device-read-perf.yaml
Test scope: performance

9. hackbench

Hackbench is both a benchmark and a stress test for the Linux kernel scheduler. It’s main job is to create a specified number of pairs of schedulable entities (either threads or traditional processes) which communicate via either sockets or pipes and time how long it takes for each pair to send data back and forth.

Test definition: automated/linux/hackbench/hackbench.yaml
Test scope: performance

10. linpack

Linpack benchmark calculates fLoating point operations per second(FLOPS).

Test definition: automated/linux/linpack/linpack.yaml
Test scope: performance

11. openssl-speed

Use openssl speed command to test the performance of cryptographic algorithms

Test definition: automated/linux/openssl/openssl-speed.yaml
Test scope: performance

12. stream-uniprocessor

STREAM benchmark is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s). This test case run STREAM on a single processor.

Test definition: automated/linux/stream/stream-uniprocessor.yaml
Test scope: performance

13. pi-stress

pi_stress checks Priority Inheritence Mutexes and their ability to avoid Priority Inversion from occuring by running groups of threads that cause Priority Inversions.

Test definition: automated/linux/pi-stress/pi-stress.yaml
Test scope: functional

14. pm-qa

Test power management (PM-QA). Currently, the test runs cpufreq, cpuidle, cpuhotplug, thermal and cputopology by default and individual test can be run by setting TESTS parameter in test job definition too. Test case documentation available in wiki: https://wiki.linaro.org/WorkingGroups/PowerManagement/Resources/TestSuite/PmQaSpecification

Test definition: automated/linux/pm-qa/pm-qa.yaml
Test scope: functional

15. docker

Docker smoke test. When docker not pre-installed, docker-ce will be installed from upstream on supported distributions.

Test definition: automated/linux/docker/docker.yaml
Test scope: functional

16. wlan-smoke

Basic WLAN smoke tests

Test definition: automated/linux/wlan-smoke/wlan-smoke.yaml
Test scope: functional

17. hci-smoke

Basic HCI smoke tests

Test definition: automated/linux/hci-smoke/hci-smoke.yaml
Test scope: functional

18. bootrr

Run bootrr https://github.com/andersson/bootrr

Test definition: automated/linux/bootrr/bootrr.yaml
Test scope: functional

19. glmark2

Glmark2 test for Linux is an OpenGL 2.0 and ES 2.0 benchmark. The Glmark2 test suite is used to measure different aspects of OpenGL (ES) 2.0 performance like video, graphics and display.

Test definition: automated/linux/glmark2/glmark2.yaml
Test scope: performance

20. piglit

Piglit is an open-source test suite for OpenGL implementations. For more information: https://piglit.freedesktop.org/

Test definition: automated/linux/piglit/piglit.yaml
Test scope: functional

21. gst-validate

A suite of tools to run integration tests for Gstreamer, For more information: https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-validate/html/

Test definition: automated/linux/gst-validate/gst-validate.yaml
Test scope: functional

22. ptest

OpenEmbedded Package tests support (ptest) The pacakge tests from OpenEmbedded enables the support to run package unittests into the target device. For more information: https://wiki.yoctoproject.org/wiki/Ptest

Test definition: automated/linux/ptest/ptest.yaml
Test scope: functional

23. 24h-stress-test-ng

Run stress_ng, stress_oom and stress_network tests

Test definition: automated/linux/24h-stress-test/24h-stress-test.yaml
Test scope: functional

24. 24h-stress-test-oom

Run stress_ng, stress_oom and stress_network tests

Test definition: automated/linux/24h-stress-test/24h-stress-test.yaml
Test scope: functional

25. 24h-stress-test-network

Run stress_ng, stress_oom and stress_network tests

Test definition: automated/linux/24h-stress-test/24h-stress-test.yaml
Test scope: functional

26. v4l2-compliance-encoder

v4l2 Compliance test suite

Test definition: automated/linux/v4l2/v4l2-compliance.yaml
Test scope: functional

27. v4l2-compliance-decoder

v4l2 Compliance test suite

Test definition: automated/linux/v4l2/v4l2-compliance.yaml
Test scope: functional

28. deqp-runner

The drawElements Quality Program (deqp) Testing allows to test OpenGL API’s, uses OpenGL ES CTS from https://github.com/KhronosGroup/VK-GL-CTS.

Test definition: automated/linux/deqp-runner/deqp-runner.yaml
Test scope: functional

Manual Tests

In addition Linaro runs additional manual tests, when they cannot be automated using the Linaro LAVA lab.

This section describes the manual tests used from the Linaro Test Definitions repository.

1. serial-console

Test if the serial connection to the board works

Test definition: manual/generic/serial-console.yaml
Test scope: functional

2. primary-video-out-hdmi

Test if HDMI video output works well

Test definition: manual/generic/primary-video-out-hdmi.yaml
Test scope: functional

3. hdmi-input-switching

Test input switching from Monitor.

Test definition: manual/generic/hdmi-input-switch.yaml
Test scope: functional

4. hdmi-resolution-switching-cmd

Check if switching HDMI resolution on monitor works from Terminal.

Test definition: manual/generic/linux/hdmi-resolution-switching-cmd.yaml
Test scope: functional

5. hdmi-resolution-switching

Check if switching HDMI resolution on monitor works.

Test definition: manual/generic/linux/hdmi-resolution-switching.yaml
Test scope: functional

6. usb-keyboard

Basic test for USB keyboard functionality

Test definition: manual/generic/linux/usb-keyboard.yaml
Test scope: functional

7. usb-storage

Basic test for USB keyboard functionality

Test definition: manual/generic/linux/usb-storage.yaml
Test scope: functional

8. usb-camera

Test if USB camera works on Linux

Test definition: manual/generic/linux/usb-camera.yaml
Test scope: functional

9. linux-wifi-ui

Test if WiFi works well from UI

Test definition: manual/generic/linux/wifi.yaml
Test scope: functional

10. linux-bluetooth-scan

Test if Bluetooth scan, pairing and file transfer works.

Test definition: manual/generic/linux/bluetooth-scan.yaml
Test scope: functional

11. linux-bluetooth-scan

Test if Bluetooth scan, pairing and file transfer works.

Test definition: manual/generic/linux/bluetooth-pair.yaml
Test scope: functional

12. linux-bluetooth-audio

Test if Bluetooth audio works well

Test definition: manual/generic/linux/bluetooth-audio.yaml
Test scope: functional