The w-iLab.t testbed provides a set of tools to support benchmarking and repeatable experiments in general. Currently, these tools can be used separately or in conjunction to create a complete benchmarking workflow.

Repeatable Wi-Fi experiments

Repeatability is a strong concern when considering wireless experiments. Through the iPlatform concept the testbed provides the repeatable execution of scripts on the iNodes. Using iPlatforms, a user defines a remote mount of the w-iLab.t fileserver on each node, where an executable file, start_mount_script, will be executed after node booting.

By design choice there is no strict synchronization present in the execution of these start scripts, but the code for using the shared directories for synchronization is available for download here. By using this or a user chosen method is is possible to schedule and forget your benchmarks, and analyse them afterwards.

For more information on using the iNodes for your experiments, please see the detailed iNode documentation.

Creating a repeatable environment

Part of the CREW benchmarking goals is the creation of repeatable environments. On the w-iLab.t testbed we currently can provide a repeatable home environment that can be customized to a user's needs. Variations of this and future environments are to be released as well. To see a demonstration of this environment, please see the advanced tutorial.

To use the Home Environment in your experiments, download the iPlatform scripts here, unzip them in an iPlatform directory of your choosing and give start_mount_code executable permissions. Before running, please take a look at the variables file, containing all adjustable parameters for the experiment. The file is annotated, but the most important variables are listed in the following table

We are currently transitioning to a new experimentation control framework for w-iLab.t (OMF), where the experiments themselves can be parametrized, allowing a more generic approach to defining an environment. When available, a detailed explanation to this new approach will also be available here.

Benchmarking Wireless Sensor Networks

The w-iLab.t testbed provides different facilities for a WSN protocol developer to benchmark their own code. The only requirement is that your developed code is compatible with the telosb mote. Any WSN code can be benchmarked using our repeatable environments if the variables that need to be varied are exposed as global variables in your WSN code (see how to change global variables at schedule time). However, a benchmarking API is provided that takes care of the repeatable execution of your WSN code and reports all logged data in a standardized format to the w-iLab.t database for quick visualization.

This API is implemented as TinyOS modules that should be included in your compiled TinyOS image, or for the IDRA framework, which is a networking focused modular development framework. The IDRA API is closely supported and follows the latest features of IDRA, and can be downloaded here. To learn more about IDRA, its purpose and how to configure it, please visit the official website, idraproject.net. The TinyOS modules are currently being updated to support the same features and will be available soon.

To schedule benchmarks using the provided API w-iLab.t uses a BNF syntax to define parameters or parameter traversals. More information on the BNF system is available here.

The full benchmarking API is given in the following table. for IDRA, these variables are available as Benchmarking_P.<parameter_name>

Benchmarking analysis

The final step in the benchmarking process, is also supported by the w-iLab.t testbed. When using the WSN API all the logs from a benchmark are automatically inserted into a separate database table using a fixed format. This table is not restricted to WSN results, but then the data has to be inserted following the described logging format below. This is however the only requirement to use the provided analysis tools

Following events can currently be logged, according to the benchmarking API: node purpose, boot time, send attempt, send success, send failure, radio sleep duration, total sent packets, total received packets, debug statistics (total debug msgs/fails)

All log data is processed using sql instructions and presented in a barchart, linechart or a 2d map of a testbed using the analyser and visualiser tools. For more information how to use and configure these tools can be found here.

Following metrics and visualisations are implemented and available for download here

• Reliability: calculated on application level, each packet that is sent by a SUT should be received by the destined SUT to reach 100% reliability for the sending SUT. Available as analyser and visualiser tool.
• Packets sent/received: The total amount of packets sent or received by the radio adapter, also available as packets sent/received per second. Available as analyser and visualiser tool.
• Radio sleep percentage: The fraction of the benchmark that the radio adapter spends sleeping, this is the primary energy efficiency metric for WSN and similar networks of embedded devices. Available as analyser and visualiser tool.
• Wifi throughput: Visualises the network wide wifi throughput, as logged by the environment. Can only be used when using one of the repeatable environments. Available as analyser.
• Application level events: the amount of network wide events visualised over time, includes packet sending, receiving, errors and boot times. Available as analyser.