Cucumber Tests with Zephyr
Reliable firmware is a prerequisite for every successful embedded product; however, conventional test methods usually depend on physical prototypes that are expensive and slow to obtain. Cucumber resolves this bottleneck by converting functional requirements written in plain language into fully automated test cases. When these tests are executed on the Zephyr real-time operating system (RTOS) in a virtual hardware environment, we can verify system behaviour long before the first circuit board is produced.
Adopting this methodology at bytes at work offers three concrete advantages for our clients:
- Transparent quality assurance: Because each test reads like an acceptance criterion, product owners, quality managers and other stakeholders can review and extend them without specialist training.
- Faster time-to-market: Continuous, hardware-independent testing allows development to proceed around the clock, enabling safe, rapid delivery of new features.
To illustrate these advantages in practice, the remainder of this article walks through a minimal demonstration application: an LED that changes its blink-rate when a proximity sensor is activated. Using Cucumber, we can express the expected behaviour in plain-language statements such as:
When I trigger the proximity sensor Then the LED is off for 900 milliseconds Then the LED is on for 900 milliseconds
These sentences double as executable test cases, executed on Zephyr in a fully virtual environment. The code for this blog post can be found on GitHub .
Setting up the Zephyr application
Let’s start with a Zephyr example application .
west init -m https://github.com/zephyrproject-rtos/example-application --mr v4.1.0 my-workspace cd my-workspace west update
This application has a simple functionality: Through a proximity sensor (modeled through a GPIO), the blink period of a LED can be modified.
Now let’s write a human-readable integration test for it!
Emulating the hardware
To be able to run this application without a board (e.g. in CI), we need to compile it for the native_sim target (our host compiler). Let’s try that:
cd example-application west build -b native_sim/native/64 app
We get the following error message:
error: ‘__device_dts_ord_DT_N_NODELABEL_example_sensor_ORD’ undeclared
Our application uses the “example” sensor, which the native_sim board does not support. Let’s add it as a board overlay:
# app/boards/native_sim_64.overlay
/ {
example_sensor: example-sensor {
compatible = "zephyr,example-sensor";
input-gpios = <&gpio0 1 GPIO_ACTIVE_HIGH>;
};
blink_led: blink-led {
compatible = "blink-gpio-led";
led-gpios = <&gpio0 2 GPIO_ACTIVE_HIGH>;
blink-period-ms = <1000>;
};
};We can now run the application on our computer:
$ west build -t run *** Booting Zephyr OS build v4.1.0 *** Zephyr Example Application 1.0.0 Use the sensor to change LED blinking period
Sadly, we don’t have have proximity sensors and LEDs on our computer, so we are not able to interact with the application. For testing purposes, we need to add some kind of interface to the application to be able to interact with it.
This is where cucumber-cpp comes in. This is a C++ library that opens a tcp server that can be used to interact with the application via the cucumber wire protocol.
Installing Cucumber
First, install the required dependencies:
apt-get install \ libasio-dev \ libgtest-dev \ libtclap-dev \ nlohmann-json3-dev \ ruby-dev
Then install the cucumber ruby gem:
gem install \ cucumber:9.2.1 \ cucumber-wire:7.0.0
The cucumber-cpp library will be added as a dependency to west.yml:
– name: cucumber url: https://github.com/cucumber/cucumber-cpp path: modules/cucumber revision: c5c420097c820583a1ff26f29ca1d6438d00fece
It can be installed by simply running
west update
Setting up Cucumber-cpp
Cucumber-cpp is not (yet) a zephyr module, so we need some custom CMake magic to build it together with our application:
# app/CMakeLists.txt
if (CONFIG_BOARD_NATIVE_SIM_NATIVE_64)
set(CUKE_ENABLE_GTEST ON)
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../../modules/cucumber cucumber)
endif()Cucumber-cpp is written in C++, so Zephyr needs to be configured to support the C++ language:
# app/boards/native_sim_64.conf CONFIG_CPP=y CONFIG_REQUIRES_FULL_LIBCPP=y CONFIG_CPP_EXCEPTIONS=y CONFIG_STD_CPP20=y CONFIG_CPP_RTTI=y
# app/CMakeLists.txt target_link_libraries(app PRIVATE cucumber-cpp-nomain)
Let’s add a cucumber server in a separate thread that runs next to the application:
void cucumber_server_thread(void *, void *, void *)
{
const std::string IP = "127.0.0.1";
const unsigned short PORT = 3900;
{
::cucumber::internal::CukeEngineImpl cukeEngine;
::cucumber::internal::JsonWireMessageCodec wireCodec;
::cucumber::internal::WireProtocolHandler protocolHandler(wireCodec, cukeEngine);
::cucumber::internal::TCPSocketServer tcpServer(&protocolHandler);
tcpServer.listen(asio::ip::tcp::endpoint(asio::ip::address::from_string(IP),
PORT));
std::clog << "Listening on " << tcpServer.listenEndpoint() << std::endl;
tcpServer.acceptOnce();
}
}When we now run the application, we see that the tcp server was started:
$ west build -t run *** Booting Zephyr OS build v4.1.0 *** Zephyr Example Application 1.0.0 Use the sensor to change LED blinking period Listening on 127.0.0.1:3900
Interfacing with the application
To interface with the application, we add cucumber “steps”. The steps are defined using regexes and can interface with the application by calling the usual Zephyr functions. The interface with the GPIOs is handled through gpio_emul , which is compiled automatically when using native_sim.
We will first add a step to trigger the proximity sensor. Special care has to be taken to respect the timing of the main thread.
const gpio_dt_spec proximity_gpio = GPIO_DT_SPEC_GET(DT_NODELABEL(example_sensor), input_gpios);
void set_proximity_sensor(int enabled)
{
gpio_emul_input_set(proximity_gpio.port, proximity_gpio.pin, enabled);
}
void wait_for_main_tick(void)
{
k_sleep(K_TIMEOUT_ABS_MS(ROUND_UP(k_uptime_get() + 1, MAIN_LOOP_PERIOD_MS + 1)));
}
WHEN("^I trigger the proximity sensor$")
{
set_proximity_sensor(1);
wait_for_main_tick();
set_proximity_sensor(0);
}Similarly, we can add steps to check the value of the LEDs.
const gpio_dt_spec led_gpio = GPIO_DT_SPEC_GET(DT_NODELABEL(blink_led), led_gpios);
int get_led_state(void)
{
return gpio_emul_output_get(led_gpio.port, led_gpio.pin);
}
void check_led_state(bool state, std::uint64_t duration_ms)
{
k_msleep(TIME_SLACK_MS - 1);
ASSERT_EQ(state, get_led_state());
k_msleep(duration_ms - 2 * TIME_SLACK_MS - 1);
ASSERT_EQ(state, get_led_state());
k_msleep(TIME_SLACK_MS - 1);
}
THEN("^the LED is on for (\d+) milliseconds?$")
{
REGEX_PARAM(std::uint64_t, ms);
check_led_state(1, ms);
}
THEN("^the LED is off for (\d+) milliseconds?$")
{
REGEX_PARAM(std::uint64_t, ms);
check_led_state(0, ms);
}This is enough to write our first cucumber test!
Adding a test case
In our test case, we will verify that the application reacts correctly. We will use the steps we just defined:
# app/features/blink.feature Feature: Blink the LED Scenario: Blink slow when proximity sensor is activated When I trigger the proximity sensor Then the LED is off for 900 milliseconds Then the LED is on for 900 milliseconds Then the LED is off for 900 milliseconds Then the LED is on for 900 milliseconds Scenario: Blink faster when proximity sensor is activated again When I trigger the proximity sensor Then the LED is off for 800 milliseconds Then the LED is on for 800 milliseconds Then the LED is off for 800 milliseconds Then the LED is on for 800 milliseconds
To tell cucumber how to connect to the Zephyr application, we need two additional files:
# app/features/step_definitions/cucumber.wire host: localhost port: 3900 # app/features/support/wire.rb require 'cucumber/wire'
Now we can run our tests:
$ west build -t run & $ cd app $ cucumber Feature: Blink the LED Scenario: Blink slow when proximity sensor is activated # features/blink.feature:3 When I trigger the proximity sensor # steps.cpp:41 Then the LED is off for 900 milliseconds # steps.cpp:54 Then the LED is on for 900 milliseconds # steps.cpp:48 Then the LED is off for 900 milliseconds # steps.cpp:54 Then the LED is on for 900 milliseconds # steps.cpp:48 Scenario: Blink faster when proximity sensor is activated again #features/blink.feature:10 When I trigger the proximity sensor # steps.cpp:41 Then the LED is off for 800 milliseconds # steps.cpp:54 Then the LED is on for 800 milliseconds # steps.cpp:48 Then the LED is off for 800 milliseconds # steps.cpp:54 Then the LED is on for 800 milliseconds # steps.cpp:48 2 scenarios (2 passed) 10 steps (10 passed)
Integrating Cucumber into Zephyr’s test runner
Zephyr has an integrated test runner called twister . It is used to automatically run tests cases across different configurations. Let’s integrate our Cucumber test into it.
Running a separate test binary next to the emulated board is made easy through twister’s pytest harness . We will add a small pytest that runs the cucumber binary we previously started manually:
# app/pytest/test_cucumber.py
def test_srs(dut):
cucumber = subprocess.run(
[
"cucumber",
"--strict",
"--publish-quiet",
],
cwd=TEST_DIR,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
logging.info(f"Cucumber returned with code {cucumber.returncode}.")
logging.info(f"Cucumber output:\n\n{cucumber.stdout}")
assert cucumber.returncode == 0The test can then be added to sample.yaml for twister to pick it up.
# app/sample.yaml tests: app.cucumber: build_only: false platform_allow: native_sim/native/64 harness: pytest
Now we can run the test using twister:
$ west twister -T app -v -s app.cucumber INFO - 1/1 native_sim/native/64 app.cucumber PASSED INFO - 1 of 1 executed test cases passed (100.00%) on 1 out of total 949 platforms (0.11%).
Continuous integration (CI) is already set up for the Zephyr example application, so the Cucumber test now is automatically run in CI.
Conclusion
In this blog post, we demonstrated how to use Cucumber tests with Zephyr. Cucumber tests allow us to use plain language to write test cases which are also understood by non-programmers.
We used it to test the main thread of a sample application. Similarly, the demonstrated concepts can also be applied to test individual components of an application that is written in a more testable way.