Basic information

TUD coordinates the test activities that are related with the EASY-C testbed. Cellular use cases and CR-related field trials are provided by the Dresden test bed, which is supervised by TUD.

The TUD contribution is based on the EASY-C campus infrastructure, i.e, the EASY-C outdoor lab test bed which is directly operated by the Vodafone Chair research team. An LTE-like cellular infrastructure is used where relevant network parameters are measured such as frame error rates, outage events, throughput or latency. One base station at rooftop level will be used which serves multiple UEs. This BS resides at the faculty of electrical engineering and information technology, TUD. Stationary and mobile user equipment are used. Below are depicted, from left to right: mobile test UE, base station equipment, UE lab equipment.

EASY-C test equipment.

External users of the TUD test bed need to install their own equipment at the TUD test site. A predefined test setup is used which provides well defined and reproducible EASY-C LTE traffic – good for the CREW cognitive radio benchmarking initiative. The LTE network parameters are constantly monitored and recorded. The CR transceivers are then activated where the LTE performance parameters are compared for the non-CR and CR case. Hence, it will be possible to benchmark the impact of various CR schemes on a cellular infrastructure through a well-defined set of reproducible test setups.

Another possibility for external users is to connect via Remote Desktop to the TUD indoor test bed to perform experiments with a fixed setup of one eNB, one UE, National Instruments USRP 2920 and one Signalion HaLo device.

Please click on the thumbnail below to get an overview picture of the hardware available in LTE advanced testbed.

Usage of the testbed

Two experimentation setups are available: the indoor lab and the outdoor lab.

In order to conduct experiments in the LTE+ testbed, participants are required to bring their spectrum sensing and/or secondary system hardware to Dresden, if the experiment cannot be performed by a USRP or HaLo device. In order to pre-evaluate certain theories and algorithms, a testbed reference signal in the form of baseband I/Q samples can be provided. Further, during the experiments it is possible to dump transmitted and received signals in the same format. This allows for offline post-processing of the signals, evaluation of the signals and replay in other testbeds.

It is important to distinguish if a downlink (DL) or an uplink (UL) experiment is desired.

In uplink experiments, it is possible to serve up to 4 UEs. The UEs use 1 antenna for transmission, while the eNBs can receive with 1 or 2 antennas. The resolution for scheduling a transmission is 1 ms, which corresponds to 1 TTI (transmission time interval). Scheduling can be done for a total duration of several minutes. The number of occupied PRBs is either 10, 20, 30 or 40 (cf. Table 1). QPSK, 16QAM and 64QAM modulation are supported.

In downlink, up to 4 UEs and up to 4 eNBs can be used simultaneously. The eNBs can transmit with up to 2 antennas and the UEs can receive with up to 2 antennas, thus up to 2 streams per UE can be sent. Time resolution is 1 ms corresponding to 1 TTI (same as UL). The number of occupied PRBs can be 12, 24, 36 or 48 (cf. Table 2).

The evaluation of an experiment happens via dumps of the received signals at the UEs / eNBs. While in the UL, signal dumps can be recorded for all eNBs in synch, the dumping process needs to be initiated manually and out of synch in the DL.

The signal dumps contain the received time samples as well as additional control information. Further processing in Matlab allows derivation of indicators like SINR, BLE, etc. in semi-realtime/offline.

The performance evaluation of experiments can be performed in real-time as well as semi real-time and offline. Real-time measures include

• Received Signal Strength Indicator (RSSI),
• Reference Signal Receive Power (RSRP),
• Path loss, and
• Channel Quality Indicator (CQI; derived from SINR).

In semi real-time, additionally QAM constellations and block error rate (BLER) can be monitored via file dump of I/Q samples and Matlab post processing. Further performance measures could be obtained in offline processing from those file dumps.

Please click on the thumbnail below to get an overview picture of the usage overview in LTE advanced testbed. 

Deviations from LTE release 8

As the eNB and UE provide only minimal LTE release 8 (Rel 8) PHY/MAC functionality, it is particularly important to note that the DL frame structure and control channels differ slightly. Differences include:

• PDCCH is always in the second OFDM-symbol (position is variable according to Rel. 8),
• PHICH (HARQ Indicator Channel) is not in the first OFDM symbol and has a different structure/content,
• PCFICH (Control Format Indicator Channel) is not supported, and
• PBCH (Broadcast Channel) is not supported.

Further, the OFDM scheme is used in the uplink. Also, 5 MHz and 10 MHz mode are not supported, thus the testbed operates in 20 MHz mode only.

An overview of the uplink and downlink processing chains can be seen below:

Uplink and Downlink Processing Chain

Instructions for external participants

Before the experiment:

• Contact the testbed staff, make sure the hardware is compatible (frequencies) and the testbed supports all features necessary for the intended experiment
• If necessary for the experiment and after the inital contact with the research staff: Get an account to access the network and computers. Remote desktop access over internet is also possible.
• Make sure there will be enough testbed hardware available (indoor/outdoor?)
• If reasonable, ask for reference signal files to check compatibility with external hardware
• Get familiar using the spectrum analyzer R&S FSQ
• Prepare UE/eNB configuration files or ask testbed staff to do it

During the experiment:

• Carefully check the setup
• Use a terminator when there is no antenna/cable plugged
• Check if all cables are ok
• Make sure you are using the latest version of the config tool to configure the hardware
• Keep a record of the config files you use as you are changing parameters
• Check the signal on the spectrum analyzer, several things can be validated that way
• If in nothing else seems to work reboot and reconfigure the UE/eNB prototype hardware
• When in doubt, ask the testbed staff

After the experiment:          

• Put everything back where you took it from

Example experiment

Detection of occupied frequency bands is the foundation for applications of dynamic spectrum access (DSA). In order to convince network operators that DSA is feasible in cellular frequencies, it has to be shown that a reliable detection of their primary signals is possible. In this section, we present an experimental validation of an algorithm and hardware, which can detect the presence of a Long Term Evolution (LTE) signal. In contrast to the classical mono antenna approach, an array of antennas is used, which allows to enhance the detection capabilities, particularly when besides the useful signal there is also interference.

Objectives of experiments

• Reliable sensing in real environment
• Performance gain of multi-antenna vs mono-antenna
• Effectiveness of primary and secondary synchonisation criteria
• Parametrization of sensing algorithm (detection threshhold)

Setup

The multi-antenna LTE sensing platform allows to acquire LTE (I, Q) data and to process them using advanced antenna processing algorithms. As described in the pictures below, the multi-antenna demonstrator is made of:

• A set of 5 antennas,
• A 4-channel receiver,
• A 4-channel acquisition board,
• A GPS system for positioning,
• A laptop for data storage and off-line multi-antenna signal processing.

Multi-antenna sensing platform block diagram

Filtering and gain control are applied to the signal in the multi-channel receiver unit, the multichannel acquisition board is used to convert the signals to digital domain and a control computer handles processing and evaluation of the digital samples. The multi-antenna LTE sensing platform is validated with lab tests by measuring sensitivity and co-channel interference rejection with real LTE eNBs. A hardware array simulator consisting of splitters, coupling modules and a set of cables of particular lengths is employed to virtually create a multiantenna, mono-path propagation channel with two directions of arrival.

Multi-antenna sensing platform

The main characteristics of the multi-channel receiver are: frequency bands - 1920-1980 MHz / 2110-2170 MHz); bandwidth - 5 MHz; Output intermediate frequency - 19.2 MHz; Noise factor (at maximal gain) <7 dB; Rx gain - 0 to 30dB (1 dB step); Frequency step - 200 kHz;  number of Rx channels – 4; Gain dispersion<1 dB; Phase dispersion<6°; Frequency stability <10^-7; Selectivity at ±5 MHz >50dB.

The main characteristics of the multi-channel acquisition are: Resolution - 12 bit; internal quartz clock - 15.36 MHz; Number of channels - 4; -3 dB bandwidth >25 MHz; Memory - 8MSamples (i.e. 2MSamples per channel).

Sensitivity tests

For sensitivity measurements, the platform depicted below is used. The level of the BTS is gradually lowered in order to estimate the sensitivity level when using one or four channels for detection processing.

Lab test platform for sensitivity measurements

Interference rejection tests

For interference rejection measurements, the platform described in the picture below is used. The second BTS is considered as the interfering BTS in the following. Its bandwidth was set to 20MHz in order to be able to highly load the OFDM sent symbols. The level of the first BTS is gradually lowered while the level of the second one does not change. The 80% detection limit level is searched when using one or four channels for detection processing.

Lab test platform for interference rejection measurements

Results

LTE detection sensitivity level for an 80% detection rate

The LTE detection sensitivity performance is summarized in the table above. It is slightly higher to what can be expected (6 dB with 4 antennas). This might be due to a lower sensitivity of the first channel compared to the other three.

LTE rejection capacity for an 80% detection rate

We can see that, with four antennas, the rejection gain is equal to 32 dB. 

For further details refer to: Nicola Michailow, David Depierre and Gerhard Fettweis: “Multi-Antenna Detection for Dynamic Spectrum Access: A Proof of Concept”,  QoSMOS Workshop at IEICE 2012

https://mns.ifn.et.tu-dresden.de/Lists/nPublications/Attachments/895/main.pdf

Basic tutorial

This tutorial explains how to set up a basic transmission.
Download this Zip archive with all necessary default configuration files.

Setup the eNB

• Power the hardware
  • Sorbas eNB Simulator
  • Radio Unit
  • eNB control computer
• Configure the eNB
  • Open the SimpleProxy 1.3.1 tool
  • Click Load Settings and select CREW_DL_config_default_1eNB_2UEs.xml or CREW_UL_config_default_1eNB_2UEs.xml
  • Click Reset to reset the eNB
  • Wait for eNB broadcast message to appear in the logging box below
  • Click Config to send the configuration to the eNB
  • Check logging box for errors

Setup the UE

• Power the hardware
  • Sorbas Test UE
  • Radio Unit
  • UE control computer
• Configure the UE
  • Open the TestUE Config tool
  • Select the System Config tab
  • Click Load settings and select CREW_DL_config_default_UE_id=0 or CREW_UL_config_default_UE_id=0
  • Click DL config and wait for UE response
  • Click UL config and wait for UE response
• Trace
  • Open the TestUE Trace tool
  • Select the Display (4) tab
  • Click Enable
  • Click Run

The system is now running. Check spectrum on R&S FSQ.

Record IQ data dumps

• Dump at the eNB
  • Open the SimpleProxy 1.3.1 tool
  • Click Freq && Dump tab
  • Click Start Dump
• Dump at the UE
  • Browse into the directory of the UE_dump_tool
  • Click start.bat
• Process IQ dumps
  • Extract IQ samples and AGC values with dumpDemux.m script
  • Perform further processing in Matlab

Advanced tutorial

Besides the manual, gui-based control for test bed related programs and devices, script controlled measurements are also possible. This approach developed at the Vodafone chair is called TestMan. The basic idea is to provide a common interface to exchange data, commands and status messages between different application, running on the same or distributed systems and written in different languages.

TestMan is based on the Microsofts .NET framework, which similar to java, is platform independent. So even a Linux computer can make use of .NET programs if the Mono project is used. In three important languages Matlab, Labview and Python it is possible to use dynamic link libraries (DLL). Therefore TestMan is a DLL which can be loaded into the specific application or script and it makes sure that data is transferred over the network from one Application to another, respectively to a group of applications.

TestMan uses two different network techniques to exchange information’s: For SNMP like status messages and commands UDP multicast is used whereas TCP peer to peer connection comes into place for transferring bigger data.

To distinguish between different applications a type and an ID are introduced for every application. So it is possible to group similar applications together and the TestMan DLL filters out messages which are intended for other applications.

UDP packets can be thrown away by network elements like routers and switches. For a status messages this is not always a problem, but definitely when a command is send. To mitigate this problem TestMan uses 4-way handshaking for commands as stated in the following figure.

The following picture shows an example how an OFDM-Transmitter can be implemented using TestMan.

Are more detailed example code will be published soon.

Configuration

Dresden’s LTE/LTE+ like testbed was set up in 2008 as part of the Easy-C project (www.easy-c.com).

The signal processing hardware includes:

• Sorbas602 eNodeB Simulators with ZF Interface to a Sorbas Radio Unit ("ZF Interconnect”)
• Sorbas202 Test UEs  with ZF Interface to a Sorbas Radio Unit ("ZF Interconnect”)
• Sorbas472 Radio Units, by Signalion: EUTRAN band VII (2.5 - 2.57 GHz and 2.62 - 2.69 GHz) as well as close to band I (1.98-2 GHz and 2.17-2.19 GHz), 20MHz bandwidth, Tx power approx. 15dBm (indoor) and approx. 30dBm (outdoor), supports up to two Tx and two Rx channels.

They were supplied by Signalion (www.signalion.com). The eNBs and UEs are connected through IF interconnects at 70MHz with the radio unit frontend. The hardware supports up to two Tx and two Rx channels for MIMO capability. The testbed operates in EUTRAN band VII (DL @ 2670-2690 MHz / UL @ 2550-2570 MHz) with fixed bandwidth of 20MHz and in FDD mode.

The LTE testbed at TUD has been upgraded with a new spectrum license in the 2.1 GHz band (1980 MHz to 2000 MHz and 2170 MHz to 2190 MHz). This step was necessary to ensure the continuous operation of the LTE testbed when the spectrum license for 2.6 GHz is withdrawn due to commercial use of the corresponding frequencies in Germany. Along with the license, several nodes have been equipped with 2.1 GHz frontends. Note that only the RF parts have been replaced, while LTE eNB and UE baseband processing remains unaffected due to the modular structure of the equipment. Further note that as long as the 2.6 GHz license is not withdrawn, those frequencies are still available for experimentation.

The operation of the new equipment has been successfully tested. The internal US5 experiment “LTE Multi-Antenna Sensing” has been conducted in the 2.1 GHz frequency range.

Base station (eNB) and mobile terminal (UE) nodes each are connected to a host PC and configured with text files in XML format. The host computer also manages measurements of the received signals and stores them in dumps. At the eNBs, a GPS unit is used for synchronization, while the UEs employ GPS for position tracking. Additionally, UEs can be powered by a mobile power supply if necessary. 

Configuration of a baste station node (eNB)

Configuration of a mobile terminal node (UE)

All UEs in the testbed, as well as the indoor eNBs are equipped with Kathrein 800 10431 omnidirectional antennas. The antennas of the outdoor eNBs are sectorized and of type Kathrein 800 10551. You can find detailed information about these antennas here:

http://www.kathrein-scala.com/catalog/80010431.pdf
http://www.kathrein.de/en/mcs/catalogues/download/99811214.pdf

Other testbed equipment includes six batteries that can power an individual UE for around 2-4 hours, GPS receivers for time synchronization, various cables, attenuators and splitters. Measurement equipment includes spectrum analyzers Rohde & Schwarz FSH4, Rohde & Schwarz FSQ8 and Rohde & Schwarz TSMW. For more details click on following links:

R&S FSH4 data sheet:

http://www.test-italy.com/occasioni/2013/rs_fsh8-18/FSH_dat-sw_en.pdf

R&S FSQ8 data sheet:

http://cdn.rohde-schwarz.com/dl_downloads/dl_common_library/dl_brochures_and_datasheets/pdf_1/FSQ_dat-sw_en.pdf

R&S TSMW operating manual:

http://www.rohde-schwarz.de/file_9446/TSMW_Operating_Manual.pdf

R&S TSMW software manual: 

http://www.rohde-schwarz.de/file_11255/TSMW_Interface_and_Programming_Manual.pdf

Indoor and outdoor setups

For the CREW project, two experimentation setups are available.

The indoor lab features 5 eNBs and 4 UEs. While the hardware itself is stationary, the Tx and Rx antennas can be positioned anywhere in the lab room. Further, four additional UEs are mounted on studio racks/carts and can be moved within the building. The approximate transmit power is 15 dBm.

The outdoor lab consists of two base station sectors that are fixed on two opposing corners of the faculty building, approximately 150 m apart. In addition to the mobile indoor UEs from setup 1, three rickshaw UEs are available for outdoor experiments in the vicinity of the building. There are also 6 batteries which can supply an UE for around 2-4 hours. The transmit power is approximately 30 dBm.

Outdoor setup

Indoor setup

Secondary System

The Signalion Hardware-in-the-Loop (HaLo) is a platform designated to simplifying the transition from simulation to implementation. To support cognitive radio setups that consider a primary/secondary user configuration, the LTE testbed has been extended by a HaLo node that can take the role of the secondary user. On the HaLo device, a novel modulation scheme called Generalized Frequency Division Multiplexing (GFDM) is now available.

This enables experimenters to consider experiment setups in LTE testbed, where the LTE system can act as a monitored primary system, while the GFDM system can run as an interfering secondary system.

HaLo Concept

The HaLo consists of a wireless transceiver that can operate in the 2.6 GHz frequency band. The concept is such, that complex valued data samples are transmitted to the device’s memory via USB from a control computer. The samples can be either read from a previously recorded file or generated on the fly e.g. by a Matlab script. The signal is transmitted over the air and received in a similar way. The device digitizes the signal and stores complex valued samples to an internal memory before they are fed back via USB to the control computer.

The HaLo setup

Note that due to limitations in the HaLo’s internal memory real-time operation is not possible.

GFDM Theory

The transmission scheme chosen to be implemented on the HaLo device to act as a secondary system in the testbed is a novel, non-orthogonal and flexible modulation scheme called GFDM. The concept is such that a multicarrier signal is transmitted, quite similar to the well know and established OFDM scheme, however one of the differences is in the pulse shaping of the individual subcarriers. This step allows shaping of transmissions and produces a signal with particularly low out-of-band radiation, which is a very desirable property in cognitive radio. For further details please refer to:

 https://mns.ifn.et.tu-dresden.de/Lists/nPublications/Attachments/826/Michailov_N_VTCfall12.pdf

GFDM transmitter and receiver block diagram

Requesting a user account

To access the TKN instance of the TWIST web interface you need to have registered an account. If you are not yet registered, go to the TWIST web interface where you should see the following welcome page:

Make sure that your browser has cookies enabled and click on "New account". In the form fill in your name, email address and choose a username (at least 6 characters) and a password. Make sure you confirm the password and answer the spam control question. Then press the "Request" button; if you filled in the form correctly you will see a new page saying "Successful account request". Now go to the TWIST terms of use page. Copy and paste the content of this page into an email, add the requested information (the nature of the intended experiments, etc.) and send this email to the TWIST administrator (email address is given on the same webpage). Please also make sure that you explain your relationship to the CREW project. The last step in obtaining an account is in the responsibility of the administrator, and you will be notified by email when your account has been activated. If there are any problems, please contact Mikolaj Chwalisz.

Running a simple experiment

Installing a node image

In this section we install the TinyOS 2 Oscilloscope application on a set of Tmote Sky nodes in the TKN TWIST testbed. The Oscilloscope application is described in the TinyOS 2 tutorial 5. After you have compiled the application with make telosb open a web browser and access the TWIST web interface. Press the "Login" button, enter your username and password and then click on "Sign in". If you have not yet registered a TWIST user account take a look at this tutorial page.

You will see a welcome page where you have three options: manage and update your account settings ("My Info"), schedule and control jobs in the testbed ("Jobs") or logout ("Logout"). Click on "Jobs" and you will see a list of scheduled jobs, i.e. the currently active jobs as well as pending future jobs. Take a close look at the list and find a time period for which Tmote/TelosB nodes are not reserved by someone else. Then click on "Add" and you will see the Job Management page as follows:

Under "Platforms" select Tmote; then choose a "Start/End date" and "Start/End time" such that the time interval is not overlapping with other jobs, which you checked in the previous step. You cannot make a real mistake here, because the system will automatically check for and not permit jobs that are overlapping in time if they use the same mote platform. However, different platforms (eyesIFX vs. Tmote) may be used concurrently. In the field "Description" enter a short note on what you plan to do in your job, such as "Testing the T2 Oscilloscope application", then click on "Add". If the time interval that you entered was accepted you will be taken back to the list of scheduled jobs, otherwise you get an error message and need to adapt the values.

The list of "Scheduled jobs" should now include your job. Your entry is likely to have gray background colour, meaning that it is registered but not yet active. The current system time is always shown in the upper right corner of the page and once your job becomes active -- its start time is shown in the column "Start" -- the background colour of your entry will turn yellow (you need to click the reload button of your browser).

When your job is active apply a tick mark at the left side of the entry and press the "Control" button at the bottom (the "Edit" button would be used to change the time of your job and with the "Delete" button you can remove your job).

Hint: When your job is active (during a experiment) you can still extend its "End time" by clicking on "Edit" on the "Jobs" page, provided that the new "End time" does not overlap with other registered jobs.

After you have clicked the "Control" button you will see the page for controlling your active job as shown in this figure:

This page is divided into the list of Tmote node IDs available in the testbed ("Available reserved resources"), a section for submitting up to three different program images to be programmed on a subset of the nodes ("Job configuration") and a set of buttons (on the bottom, not shown in Figure 3) to perform some actions, such as installing the image(s) on the nodes.

For the TinyOS 2 Oscilloscope application we want to install the Oscilloscope program image on some Tmote nodes, and one node will need to act as gateway and will be programmed with the TinyOS 2 BaseStation application (see TinyOS 2 tutorial 5). Because we will install two different application images, in the "Job configuration" field we will use two of the three "Control group" sections: the "Control group 1" section for the Oscilloscope application and the "Control group 2" section for the BaseStation application.

In the "Control group 1" section, enter in the "Node list" field a whitespace-separated list of the node IDs on which the the Oscilloscope is to be programmed, let's say 10 11 12. For convenience you can copy & paste from the list of IDs shown on top in the "Available reserved resources" list.

Then click on the "browse..." button next to the "Image" field just below the "Node list" field. Select the Oscilloscope image, which is the main.exe in your local tinyos-2.x/apps/Oscilloscope/build/telosb (you must have compiled the Oscillocope application with "make telosb" before). The "SF Baudrate" and "SF Version" fields control whether a SerialForwarder will be started for all nodes in the respective "Node list". Since we only need a SerialForwarder for the BaseStation application, we don't change the values (leaving it "None", "TinyOS 2.x"). Finally, "Channel" is the IEEE 802.15.4 channel to be used by the Tmote Sky radio CC2420 (if you change the channel for the Oscilloscope application, make sure that you do the same for the BaseStation application). In fact, the value of the CC2420_DEF_CHANNEL symbol inside your progam image will be replaced by the value of the "channel" field and thus, if your application includes the TinyOS 2 CC2420 radio stack, you can still modify the default radio channel after you have compiled the image.

Hint: The node ID is another symbol that is modified for each node individually before programming the image. It is accessible via TOS_NODE_ID in a TinyOS application.

We use the "Control group 2" section for installing the BaseStation program image on another node. In the "Node list" field enter 13 (or whichever node ID you want to use for the BaseStation application) and under "Image" click "browse..." and select the main.exe from your local tinyos-2.x/apps/BaseStation/build/telosb folder (you must have compiled the BaseStation application with "make telosb" before). Because we want to later establish a serial connection to the BaseStation node, select the pull-down menu under the "SF Baudrate" field and choose a serial baudrate. Whenever this field has a value other than None a SerialForwarder will be started for all nodes in the respective "Node list". The default baud rate for the "TelosA", ",TelosB" and "Tmote" platforms is 115200 baud.

Hint: You can change the baud rate for a telos node by modifying tinyos-2.x/tos/platforms/telosa/TelosSerialP.nc (this file is included by telosa, telosb and tmote platform). Make sure you recompile your application after changing the file.

The "SF Version" field defines the version of the Serial Forwarder protocol. Because we are using a TinyOS 2 applications select "2" (for a TinyOS 1 application you would select "1"). If the "SF Baudrate" field is None then the "SF Version" is ignored. Finally, make sure you select the same "Channel" as the one for the Oscilloscope application. Your configuration should now look like the one shown the next figure:

To actually program the images on the nodes scroll down, press the "Install" button and wait. After not much longer than 1 minute you should see a page with the "Execution log". Check for possible errors (any line "Could not find symbol [...] ignoring symbol" is only telling you that the respective symbol was not found/changed in the application image) and scroll down to the bottom where you can find a summary of the "Install" operation. Here you can also see that a SerialForwarder has been started for node 13:

To forward SF e.g. for node 13 use: ssh -nNxTL 9013:localhost:9013 twistextern@www.twist.tu-berlin.de

In the next section we will establish an ssh tunnel to the TWIST server and connect to the SerialForwarder of the BaseStation node. The remainder of this section summarizes the fields and options for controlling an active job over the web interface.

The following table describes the fields in the "Job configuration" section:

Field	Meaning
Node list	Whitespace separated list of node IDs on which the image will be programmed
Image	The image to be programmed on the nodes in the "Node list"
SF Baudrate	Whether a SerialForwarder is started for each of the nodes in "Node list" and what baudrate it will use
SF version	The version of the SerialForwarder: use 1 for TinyOS 1.x and 2 for TinyOS 2.x
Channel	The CC2420 radio channel

The following table describes the buttons on the bottom of the "Controlling active job" page:

Button	Meaning
Install	Installs the image(s) on the node(s) specified in the above "Job Configuration" section; SerialForwarders will be started (if selected) and nodes are powered on
Erase	Programs the TinyOS Null application on the selected set of nodes
Reset	Resets (powers off & on) the selected set of nodes
Power On	Cuts the USB power supply for the selected nodes
Power Off	Enables the USB power supply for the selected nodes
Start SF	Starts a SerialForwarder for the selected nodes
Stop SF	Stops the SerialForwarder for the selected nodes
Start Tracing	Stores the serial data output from the nodes in a trace file
Stop Tracing	Stops storing data in a trace file

By pressing the "Start Tracing" button the serial data output from all nodes are automatically stored to a trace file. This file can be accessed via the job control page by pressing the "Traces" button (with your job checked). If you want to use automatic tracing then it is recommended that during install you select the correct "SF Baudrate" and "SF Version". After the install process, you can then simply click on "Start Tracing" without having to manually start the serial forwards.

Exchanging Data via the Serial Connection

Through the previously described "Install" operation a SerialForwarder for the BaseStation node was started. In order for your tinyos-2.x/apps/Oscilloscope/java/Oscilloscope.java client to connect to this SerialForwarder, you first need to establish an SSH Tunnel to forward the port of the SerialForwarder to your machine. At the very end of the execution log you find the syntax for this SSH command (type it into a shell):

ssh -nNxTL 9013:localhost:9013 twistextern@www.twist.tu-berlin.de

Once you have forwarded the port you can access the remote SerialForwarder like a local one. However, when you start your client application make sure that it attaches to the correct port as specified in the SSH Tunnel (the above command forwards the remote port to your local port 9013). For example, to start the JAVA Oscilloscope client you would first need to set the MOTECOM environment variable as follows:

export MOTECOM=sf@localhost:9013

Now you can start the Oscilloscope GUI by typing:

.\run

in the tinyos-2.x/apps/Oscilloscope/java directory as described in TinyOS 2 tutorial 5.

You should now see an Oscilloscope GUI like the one described in the TinyOS tutorial.

Using cURL for automated control

cURL is a command line tool that can, among other things, transfer files and POST web forms via HTTPS. It can thus be used to automate sequences of operations on the testbed, such as installing an image or powering a node off. Before you can actually control your job you need to authenticate via cURL (Step 1) and find you job ID (Step 2). Afterwards you can control your job (Step 3) and download traces (Step 4) associated with your job ID. The following steps list the relevant cURL commands.

Step 1: Authenticate

Use the following format to authenticate and store the secure cookie for the future requests (replace YOUR_USER_NAME and YOUR_PASSWORD with your username and password, respectively):

curl -L -k --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt -d 'username=YOUR_USER_NAME' -d 'password=YOUR_PASSWORD' -d 'commit=Sign in' https://www.twist.tu-berlin.de:8000/__login__

Note that all data fields have to be URL encoded either implicitly using --data-urlencode or explicitly (in case you have special characters in username/password)

Step 2: Find the job_id

You need to know the job_id before you can use curl to control it. This can also be done by fetching and parsing the jobs page with cURL, maybe passing the output through "tidy"

curl -L -k --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt https://www.twist.tu-berlin.de:8000/jobs | tidy

Step 3: Control

The following is a list of examples on how to control a job. Make sure that you replace the job_id and node IDs.

• Erase - For job_id 346, erase nodes 12 and 13:

  curl -k --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt -F __nevow_form__=controlJob -F job_id=346 -F ctrl.grp1.nodes="12 13" -F erase=Erase https://www.twist.tu-berlin.de:8000/jobs/control

• Install - For job_id 346, install TestSerialBandwidth on nodes 12 and 13 and start serial forwarders:

  curl -k --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt -F __nevow_form__=controlJob -F job_id=346 -F ctrl.grp1.nodes="12 13" -F ctrl.grp1.image=@/home/hanjo/tos/tinyos-2.x/apps/tests/TestSerialBandwidth/build/telosb/main.exe -F ctrl.grp1.sfversion=2 -F ctrl.grp1.sfspeed=115200 -F install=Install https://www.twist.tu-berlin.de:8000/jobs/control

• Power Off - For job_id 346, power off nodes 12 and 13:

  curl -k --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt -F __nevow_form__=controlJob -F job_id=346 -F ctrl.grp1.nodes="12 13" -F 'power_off=Power Off' https://www.twist.tu-berlin.de:8000/jobs/control

• Power On - For job_id 346, power on nodes 12 and 13:

  curl -k --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt -F __nevow_form__=controlJob -F job_id=346 -F ctrl.grp1.nodes="12 13" -F 'power_on=Power On' https://www.twist.tu-berlin.de:8000/jobs/control

• Start Tracing - For job_id 346, start tracing on nodes 12 and 13:

  curl -k --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt -F __nevow_form__=controlJob -F job_id=346 -F ctrl.grp1.nodes="12 13" -F 'start_tracing=Start Tracing' https://www.twist.tu-berlin.de:8000/jobs/control

• Stop Tracing - For job_id 346, stop tracing on nodes 12 and 13:

  curl -k --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt -F __nevow_form__=controlJob -F job_id=346 -F ctrl.grp1.nodes="12 13" -F 'stop_tracing=Stop Tracing' https://www.twist.tu-berlin.de:8000/jobs/control

Step 4: Collect data

To collect the specific trace file from archived job 336

curl -g -k --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt -d 'job_id=339' -d 'trace_name=trace_20080507_114824.0.txt.gz' -o trace_20080507_114824.0.txt.gz https://www.twist.tu-berlin.de:8000/jobs/archive/traces/download

Experiment Definition Tool

The experiment definition tool is used to define, configure, and make changes to wireless experiments that are to be conducted on different testbeds. Before explaining the experiment definition in detail, an experimenter need to be familiar with the concept of experiment resource grouping. This concept is taken from the cOntrol and Management Framework (OMF) developed by the collaborative effort of NICTA and Winlab [1].

The experiment resource grouping concept treats an experiment as a collection of resources aggregated into groups. Resources can be any of these but not limited to WI-FI nodes, sensor nodes, spectrum sensing devices, and software-defined radios. With in a group, multiple nodes run different applications that were predefined in an application pool. And a single application as well defines a number of optional measurement points. In figure 1, we show a simple experiment scenario where two WI-FI nodes (node 1 and node 3) sends TCP and UDP packets towards two receiver WI-FI nodes (node 1 and node 3). In real life this could mean, for example, Node1 performing a file transfer to Node2 while at the same time listening to a high quality (192kbps) radio channel from Node3 and Node2 is watching a 10Mpbs movie streamed from Node3.

Figure 1. simple experiment scenario using three WI-FI nodes.

The experiment shown above, also called Iperf Sender Receiver (ISR) experiment , is realized using the iperf measurement tool which is a commonly used in network testing application. The iperf tool has a number of output formats instrumented for different uses. For example when iperf is used for TCP streaming, throughput information is displayed and when used for UDP streaming, jitter and packet loss information are displayed to the end user. Therefore an application can instrument more than one output to the user and in the context of OMF, they are refereed to as measurement points (MP). For this experiment scenario, transfer, jitter, and loss measurement points are used. A graphical view of experiment resource grouping is shown in figure 2.

Figure 2. Experiment resource grouping of the experiment scenario. Note that Node3 does not have measurement points since it only streams UDP packets.

Next we look at how experiment is defined in the experiment definition tool. The ISR experiment, defined previously, is used here to illustrate tool's usage. From an experimenter point of view, there are two ways of defining an experiment. These are defining new experiment and defining from a saved configuration file.

Defining new experiments

Click the Start New Configuration link to start defining a new experiment. The first stage of experiment definition is experiment abstract. Here the experimenter can give a high level information  about the experiment such as project name, experiment name, title, author, contact information, summary, and duration. Figure 3 shows the experiment abstract of the ISR experiment.

Figure 3. Experiment abstract definition of ISR experiment.

The next step is application definition. Here we create the application pool containing all applications that will be used in the experiment. Figure 4 shows the iperf application from ISR experiment application pool.

Figure 4. iperf application defined inside the application pool.

The last step in experiment definition tool is binding applications from the pool to different nodes of the wireless testbed. It involves platform specific node selection from testbed layout, interface configuration of group of nodes, and finally binding applications to nodes. Application binding involves a number of steps which are application selection, optional output instrumentation, input parameter configuration, and application timeline definition. Figure 5 shows the node configuration section (only shown for group Node1) of the ISR experiment.

Figure 5. Experiment node configuration of ISR experiment.

Finally after finish configuring the experiment definition, we save it to a configuration package (name it ISR) composed of three files. These are XML configuration file (ISR.xml), OEDL (OMF Experiment Description Language) file (ISR.rb), and a network simulation file (ISR.ns) containing all configuration setting to be used inside emulab framework [2]. We explain the content of the XML configuration file in this section but the rest two files are explained on a separate page.

In CREW, XML is used as a configuration format for experiment descriptions and the XML configuration of the experiment definition in particular is a subset of the CREW common data format. An overview of the XML configuration of the ISR experiment is shown in figure 6.

Figure 6. Excerpt from ISR experiment XML configuration file

Defining an experiment from a saved configuration file

For the experimenter, defining everything from scratch might be time consuming. An experimenter can use an existing configuration file or customize it according to his need. Reconfiguration is done first by downloading the configuration file (look CREW repository, background environments section in CREW portal) and then loading it on the "Load/Configure Experiment" section of the experiment definition tool. Finally, start reconfiguration and save it after finish modification.

[1]. Thierry Rakotoarivelo, Guillaume Jourjon, Maximilian Ott, and Ivan Seskar, ”OMF: A Control and Management Framework for Networking Testbeds.”

[2]. http://www.emulab.net/

Experiment Execution Tool

Recall at the end of CREW experiment definition tool, we end up with a tar package containing three different files. The first file, ISR.xml, is an  XML experiment description file for the ISR experiment. The second file generated, ISR.rb, is an experiment description of the type OEDL (OMF Experiment Description Language) and the last file generated, ISR.ns, is a network simulation file containing all configuration settings to be used inside emulab framework. Current implementation of the CREW experiment execution tool only works on top of OMF (cOntrol and Management Framework) testbeds. But it is designed to be versatile and work with different testbeds having their own management and control framework. The tool interacts with the testbed using a specific API designed for the testbed. Thus working on a different testbed with a different framework only relies on the availability of interfacing APIs and minor change on the framework itself.

Coming back to what was left at the end of the experiment definition tool, we start this section by using the two files generated (i.e. ISR.rb and ISR.ns). Details of the ISR.rb file and OEDL language are described on a separate page. However, to have a deeper understanding of the language details, one can study OEDL 5.4.

We start by definining the experiment topology on the emulab framework [1] using the NS file ISR.ns. Tutorial about creating your first experiment on the emulab framework and explanation of the NS file follows this page. After finish defining your topology and swapping in your experiment in the emulab framework, start the experiment execution tool. The experiment execution tool allows experimenters load an experiment description, configure parameters, schedule, and finally start an experiment. Figure 1 shows the front view of the experiment execution tool after the OEDL file (i.e. ISR.rb) is loaded.

Figure 1. Experiment execution tool at glance.

After loading the file, four different sections are automatically are populated and each section performs a specific task.

• Parameter Optimization Section configures single/multi dimensional optimizer that either maximizes or minimizes an objective performance.
• Performance Visualization Section configures parameters to be visualized during experiment execution.
• Interference Estimation Section configures the pre/post interference estimation of experiments and detect unwanted wireless interference that could influence the experiment.
• Experiment Rounds section configures the number of identical runs an experiment is executed.

Recall the scenario of ISR experiment where Node 1 performs a file transfer to Node 2 while at the same time listening to a high quality (192kbps) radio channel from Node 3 and Node 2 is watching a 10Mbps movie stream from Node 3.

Now starting from this basic configuration, let us say we want to know how large the video bandwidth from Node 3 to Node 2 can increase so that we see a high definition movie at the highest possible quality. This is an optimization problem and next we see how to deal with such a problem using the experiment execution tool.

Steps to follow

1. Select IVuC optimizer from the Parameter Optimization section and click the text on the left side to reveal its content.
2. Select Node3_UDP2_bandwidth as design space parameter located at the end of the Tune list.
3. Start from 10,000,000 bps using step size 2,000,000 bps and reduction rate of 0.5. The optimizer either stops once the step size is reduced to 100,000 bps or when search parameter exceeds 20,000,000 bps.
4. In the objective variables subsection, configure two variables to calculate the sending and receiving bandwidth using the aggregation AVG.[ Note: Click the plus icon on the right hand corner to add extra variables]
5. Define your objective function as a ratio of the two variables (i.e. (8*x1)/x2) defined previously and select the condition less than with a stopping criteria 0.95 (i.e. (8*x1)/x2 < 0.95 => Stop when 8*x1 < 0.95*x2). [ Note: The reason 8 is multiplied to x1 variable is because iperf reports bandwidth in byte/sec].

Figure 2 shows the configuration details of parameter optimization section.

Figure 2. IVuC optimizer configuration for the ISR experiment.

The next thing we do is configure the performance visualization section where we define parameters to be visualized during experiment execution. For this exercise, we visualize the UDP and TCP bandwidth performance at Node2 coming from Node3 and Node1 respectively. As the wireless medium is shared among all Nodes, the UDP bandwidth stream from Node3 is affected by the TCP bandwidth stream from Node1 and visualizing these two parameters reveal this fact. Figure 3 shows the configuration detail of performance visualization section.

Figure 3. Performance visualization configuration for the ISR experiment

After that, we can enable interference estimation and check for possible pre and post experiment interferences. This is one way of detecting outlier experiments as there is a higher possibility for experiments to be interfered if the environment before and after their execution is interfered. For now, we skip this section. Similar to the experiment definition tool, execution configuration settings can be saved to a file. Click the button  to update the execution configuration into the OEDL file and later reloading the file starts the experiment execution page pre configured. Finally, set the number of experiment rounds to one and click on the button .

Once the experiment has started, the first thing we see is a log screen populated with debug information. The debug information contains high level information about the experiment such as defined properties, experiment ID, events triggered, and a lot more. The log screen view being an important interaction tool to the experimenter, there is also a second view called the graph screen view. To switch to the graph screen view, click the notebook icon  from the experimentation screen. The graph screen view displays parameters that were defined in the performance visualization section (see above) as a function of time. For the ISR experiment, the UDP and TCP bandwidth plot as a function of time from node3 and node1 are displayed respectively. Figure 4 shows the graph screen view of the ISR experiment.

Figure 4. A glance at the graph screen view from experimentation screen.

From the above figure, we note a couple of things. Top on the left, there is the graph icon which triggers the debug information view up on clicking. Next is the experiment status (i.e. Scheduled, Running, Finished) indicator. The label check box turns the label ON and OFF. Execution of SQL statements is allowed after the experiment has stopped running. SQL statements are written in the log file viewer by which switching to this view, writing your SQL statements and pressing the icon  executes your statements. Finally the UDP and TCP bandwidth as a function of time are plotted and it also indicates an equal bandwidth share between both applications. Not shown on the figure but also found on the experimentation screen are parameter settings, objective value, correlation matrix table, and experiment run count.

Following the whole experimentation process, we see the ObjFx. Value (i.e. (8*x1)/x2) starts around 1 and decreases beyond 0.95 which triggers the second experimentation cycle. The optimization process carries on and stops when step size of Node3_UDP2_bandwidth reaches 100,000 bps. The whole process takes around 11 minutes under normal condition. The IVuC optimizer locates an optimal bandwidth of 14.125 Mbps UDP traffic from Node3 to Node2. Therefore the highest bit rate a movie can be streamed is at 14.125 Mbps.

Finally, the experimenter has the option to save experiment traces and later perform post processing on the stored data. CREW also incorporates a post processing tool to facilitate benchmarking and result analysis. The aim of this tool is to make comparison of different experiments through graph plots and performance scores. A performance score can be an objective or a subjective indicator calculated from a number of experiment metrics. The experiments themselves doesn't need to be conducted on a single testbed or with in a specific time bound, as long as experiment scenarios (aka experiment meta data) fit in the same context. Thus an experiment once conducted can be re-executed at a latter time in the same testbed or executed on a different testbed and conduct perform comparison of the two solutions.

For this tutorial save the experiment (i.e. press the Save experiment result button) by giving a name (e.g. Solution_one) and start post processing on the saved data using the benchmarking and result analysis tool. Please be reminded that only selected experiment runs are saved to a file and you should tick the check box next to each experiment run that you want to save.

[1] http://www.emulab.net/

List of optimizers explained

Experiment optimization is the heart of the CREW experiment execution tool. The efficiency of an experiment execution tool mostly depends on how it optimizes the different experiments to come and how fast it converges to the optimum. However, due a variety of problems in the real world, coming up with a single optimizer solution is almost impossible. The normal way of operation is to categorize similar problems into groups and apply unique optimizers to each one of them. To this end, the experiment execution tool defines a couple of optimizers which are fine tuned to the needs of most experimenters. Thus this paper explains the working principle of each optimizer supported in the experiment execution tool.

Step Size Reduction until Condition (SSRuC) optimizer

SSRuC is a single parameter optimizer aimed at problems which show local optimum or local minimum in the vicinity of the search parameter. Such kinds of problems are approximately described using two monotonically increasing and decreasing functions from either side of the optimum point. Figure 1 explains the problem graphically.

Figure 1. An example showing two local optimas with monotonic functions on either side of the optimum points

SSRuC tackles such problems using the incremental search algorithm approach [1]. Five parameters are passed to the optimizer. These are the starting, ending, step size of the search parameter, the step size reduction rate and the step size limit used as a stopping criteria by the optimizer. The optimizer starts by dividing the search parameter width into fixed intervals and performs unique experiment at each interval. For each experiment, measurement results are collected and performance parameters are computed. Next, a local maximum or local minimum is selected from performance parameters depending on the optimization context. If the optimization context is "maximization", we take the highest score value whereas for "minimization" context, we take the lowest score value. After that, a second experimentation cycle starts this time with a smaller search parameter width and step size. The experimentation cycle continues until the search parameter step size lowers below the limit. Figure 2 shows the different steps involved.

Figure 2. The different steps involved in an SSRuC optimizer over the search parameter width A₁ to A₅

Figure 2 shows the SSRuC optimizer in a three level experimentation cycle. In the first cycle, five unique experiments are conducted out of which the fourth experiment (experiment A₄ ) is selected. The second experimentation cycle works in the neighborhood of A₄ with a reduced step size ?₂ . This cycle again conducts 5 unique experiments out of which the second experiment (experiment B₂ ) is selected. The last experimentation cycle finally conducts 5 unique experiments from which the third experiment (experiment C₃) is selected and treated as the optimized value of the parameter.

Increase Value until Condition (IVuC)

IVuC optimizer is designed to solve problems which show either increasing or decreasing performance along the design parameter. A typical example is described in the experiment execution tool where video bandwidth parameter was optimized for a three node Wi-Fi experiment scenario. Datagram error rate was set as a performance parameter and the highest bandwidth was searched limited to 10% datagram error rate and below.

The algorithm used by IVuC optimizer is similar to the SSRuC optimizer in such a way that both rely on incremental searching. However, the main difference between the two is that the SSRuC optimizer performs a complete experimentation cycle before locating the local optimum value whereas the IVuC optimizer performs a local optimum performance check after the end of each experiment. Later on, both approaches refine their searching parameter range and restart the optimization process to further tune the search parameter. Figure 3 shows the different steps involved in IVuC optimizer.

Figure 3. The different steps involved in an IVuC optimizer over the search parameter width A₁ to A₅

From figure 3, we see the three experimentation cycles each with five, three, and four unique experiments respectively. At the end of each experimentation cycle, performance parameter drops below threshold and that triggers the next experimentation cycle. At the end of the third experimentation cycle, a prospective step size ?₄ was checked and found below threshold, making C₄ the optimal solution.

SUrrogate MOdelling (SUMO)

Unlike SSRuC and IVuC, SUMO optimizer works on multiple design parameters and multiple design objectives. It is targeted to achieve accurate models of a computationally intensive problem using reduced datasets. SUMO manages the optimization process starting from a given dataset (i.e. initial samples + outputs) and generates a surrogate model. The surrogate model approximates the dataset over the continuous design space range. Next it predicts the next design space element from the constructed Surrogate model to further meet the optimization’s objective. Depending on the user’s configuration, the optimization process iterates until conditions are met.

The SUMO optimizer is made availabe as a MATLAB toolbox and works as a complete optimization tool. It bundles both the control and optimization functions together where the control function sitting at the highest level manages the optimization process with specific user inputs. Figure 4 shows SUMO toolbox in a nutshell highlighting the control and optimization functions together.

Figure 4.  Out of the box SUMO toolbox in a nutshell view

In the context of CREW benchmarking tools, the aim is to use SUMO toolbox as a standalone optimizer and put it inside the experimentation framework. This means starting from out of the box SUMO toolbox, the loop is broken, the control function is removed and clear input/output interfaces are created to interact with the controlling framework. Figure 5 shows how modified SUMO toolbox is integrated in the wireless testbed.

Figure 5. Integration of SUMO toolbox in a wireless testbed

The testbed management framework in the above figure controls the optimization procdess and starts by executing a configuration file. The controller pases configurations and control commands to the wireless nodes and measurement results are send back to the controller. After executing a number of experiments, the controller starts the SUMO toolbox supplying the experiment dataset which has been executed so far. The SUMO toolbox creates a surrogate model from the dataset and returns the next sample point to the controller. The controller again executes a new experiment with the newest sample point and generates a new dataset. Next the controller calls the SUMO toolbox again sending the dataset (i.e. one more added). The SUMO toolbox creates a more accurate surrogate model with the addition of one dataset. It sends back a new sample point to the controller and the optimization continues until a condition is met. It should be understood, however, that operation of the customized SUMO toolbox has not changed at all except addition of a number of blocks.

Having said about its operation, an example of wireless press conference optimization using customized SUMO toolbox is located on this link.

[1]. Jaan Kiusalaas, ”Numerical Methods in Engineering with MATLAB” Cambridge University Press, 01 Aug 2005, pp 144-146.

OEDL explained

This section provides a detailed description of the experiment description file, ISR.rb, that was created during the hands on tutorial on experiment definition tool. It mainly focus on introducing the OEDL language, the specific language constructs used inside ISR.rb file, how it is mapped to the XML experiment description file, and finally a few words on the tools OEDL language capability.

OEDL (OMF Experiment Description Language) is a Ruby based language along with its specific commands and statements. As a new user, it is not must to know the Ruby programming language. And with a simple introduction on the language, one can start writing a fully functional Experiment Description (ED) file. Any ED file written in OEDL is composed of two parts

1. Resource Requirements and Configuration: this part enumerates the different resources that are required by the experiment, and describes the different configuration to be applied.
2. Task Description: this part is essentially a state-machine, which enumerates the different tasks to be performed using the requested resources.

This way of looking an ED file is a generic approach and basis for learning the OEDL language. But specific to this tutorial we take a different approach and further divide content of an ED file into three sections.

Application Definition section
This section of ED file, all application resources taking part in the experiment are defined. Each application defines specific properties like input parameters, measurement points, and application specific binaries. Figure 1 shows an excerpt of the application definition from ISR.rb file.

Figure 1. excerpt of ISR.rb application definition

From figure 1, we see a number of OEDL specific constructs. The defApplication command is used to define a new application. Such defined application can be used in any group of nodes as required. The  defProperty command  defines an experiment property or an application property. Experiment property is a variable definition that can be used anywhere inside the OEDL code. For example node1_UDP_server and node1_UDP_udp are two property definitions inside ISR.rb file. Application property on the other hand is defined inside an application and can only be accessed after the application is added. For example interval, port, udp, and others are application properties of iperf program. Next is the defMeasurement command used to define a single Measurement Point (MP) inside an application definition. Finally the defMetric command defines different output formats for the given MP.

Group Definition section

In the group definition, a group of nodes are defined and combined with the applications defined earlier. Figure 2 shows an excerpt of ISR.rb file group definition only for Node1.

Figure 2. excerpt from iperf group definition in ISR.rb file

In this section, a number of OEDL specific constructs are used. The first one is defGroup which is used to define group of nodes. Here only nodeC1 from the testbed is defined inside the group Node1.  addApplication is the second command used and it adds application into the group of nodes from the application pool. Since it is possible to add a number of identical applications with in a single group, it is good practice to give unique IDs to each added application. For the Node1 group, two iperf applications are added with IDs TCP and UDP. setProperty command is used to set values to different input parameters of the application added. Finally node specific interface configuration is handled by resource path constructs. For the Node1 group mode, type, channel, essid, tx_power, and ip configurations are set accordingly.

Timeline Definition section
The timeline definition defines the starting and stopping moments of each defined applications with in each group of nodes. Figure 3 shows an excerpt of ISR.rb file timeline definition.

Figure 3. excerpt from iperf timeline definition in ISR.rb file

Inside OEDL language, events play a huge role. An Event is a physical occurrence of a certain condition and an event handler performs a specific task when the event is triggered. ALL_UP_AND_INSTALLED event handler shown in figure 3, for example, is fired when all the resources in your experiment are requested and reserved. The wait command pauses the execution for the specified amount of seconds. Starting and stopping of application instances are executed by the commandstartApplication and stopApplication respectively. Application IDs are used to start and stop specific applications. For example group('node1').startApplication('UDP') refers to the application iperf from node1 group with ID UDP.

So far we walk you through the default OEDL constructs that are used throughout the ISR.rb file. It is also possible to define custom OEDL constructs and one such use is custom event definition. We used custom event definition in our ISR.rb file to check if all applications are stopped. If so we trigger the EXPERIMENT_DONE event and end the experiment execution. Figure 4 shows the custom event definition used inside ISR.rb file.

Figure 4. custom event definition section in ISR.rb file

We create a custom event using the command defEvent. By default custom defined events are checked every five seconds for possible event triggering. Inside the event definition, we wait until all applications are finished and fire the event handler afterwards. The event handler is defined following the onEvent commandpassing name of the event definition APP_COMPLETED. Finally when the event is triggered, the handler calls the EXPERIMENT_DONE method which stops the experiment execution and releases all allocated resources.

Mapping OEDL to XML experiment description
Recall from experiment definition tool section that an experimenter passes three steps to finish configuring its experiment and produce XML, OEDL, and ns files. It is hidden to the experimenter however that an OEDL file is generated from the XML file. During the making process of configuration package, an XML template is first produced out of which the OEDL file is constructed.

The mapping of OEDL to XML is straight forward. It follows a one to one mapping except rearrangement of text. The following three figures show a graphical mapping of XML to OEDL on application definition, group definition, and timeline definition sections accordingly.

Figure 5. XML to OEDL application definition mapping inside ISR.rb file

Figure 6. XML to OEDL group definition mapping inside ISR.rb file

Figure 7. XML to OEDL timeline definition mapping inside ISR.rb file

Wireless press conference optimization using SUMO toolbox

Problem Statement

A wireless press conference scenario comprises of a wireless speaker broadcasting audio over the air and a wireless microphone at the listner end playing the audio stream. This type of wireless network is gaining attention specially in a multi-lingual conferencing room where the speaker's voice is translated into different language streams and broadcasted to each listner where they select any language they want to hear the translated version. However the wireless medium is a shared medium and it is possible to be interfered by external sources and we want to optimize the design parameters which gives the best audio quality. Moreover, we also want to reduce the transmission exposure which is a direct measure of electromagnetic radiation. Transmission exposure is gaining attention these days related to health issues and regulatory bodies are setting limits on maximum allowable radiation levels. Thus our second objective is to search design parameters that lowers transmission exposure of the wireless speaker exposed on each listner. For this tutorial, two design parameters of the wireless press conference are selected (i.e. transmit channel and transmit power) and we optimize these parameters inorder to increase the audio quality and decrease transmission exposure.

Experiment scenario

The experiment is composed of one speaker and 5 listeners making the Solution Under Test (SUT) and one public access point creating a background interference. The public access point is connected to three WI-FI users and provides a streaming video service at the time. Figure 1 shows the experiment scenario.

Figure 1. Experiment scenario of 5 listners, 1 speaker , 1 public access point, and 3 users.

On the left hand side, the realistic press conference scenario is shown. On the right hand side, the experimentation scenario as seen on the w.ilab.t zwijnaarde testbed [1] is shown. The horizontal and vertical distances between consecutive nodes is 6m and 3.6m respectively. All listener nodes (i.e. 38, 39, 48, 56, and 57) are associated to the speaker access point (i.e. node 47). Background interference is created by the public access point (i.e. node 49) transmitting on two WI-FI channels (i.e. 1 and 13) at 2.4Ghz ISM band.

The wireless speaker, configured on a particular transmit channel and transmit power, broadcasts a 10 second WI-FI audio stream and each listener calculates the average audio quality within the time frame. The wireless speaker at the end of its speech averages the overall audio quality from all listeners and makes a decision on its next best configuration. The wireless speaker using its newest configuration again repeats the experiment and produces a second newest configuration. This way the optimization process continues iterating until conditions are met.

The public access point on the other side transmits a 10 Mbps continuous UDP stream on both channels (i.e. channel 1 and 13) generated using iperf [2] application. In the presence of interference, audio quality gets degraded and the wireless speaker noticing this effect (i.e. lower audio quality) has two options to correct it. Either increase the transmission power or change the transmission channel. Making the first option is unlikely because it increases the transmission exposure. Changing the transmission channel also has limitations; the problem of overlapping channels interference [3]. Overlapping channels interference results in quality degradation much worse than identical channel interference. In identical channel interference, both transmitters apply CSMA-CA algorithm and collision is unlikely to happen. However in overlapping channels interference, transmitters don’t see each other and collision likely happens.

Two fold optimization

As was explained previously, we want to optimize the design parameters that brings an increased audio quality and decreased transmission exposure. A straight forward solution is to perform unique experiments at each design parameter combinations also known as exhaustive searching and locate the optimum design parameters which gives the highest combined objective (i.e. audio quality + transmission exposure).

Audio quality objective

In the earlier telephony system, Mean Opinion Score (MOS) was used for testing audio quality out of a 1 to 5 scale (i.e. 1 for the worst and 5 for the best quality). MOS is a subjective quality measure that no two people give the same score. However due to recent demands in Quality of Service (QoS), subjective scores are replaced by objective scores for the reason of standardization. Moreover, the earlier telephony system is now replaced by the more advanced Internet Protocol (IP) driven backbones where Voice over IP (VoIP) service becomes the prefered method of audio transportation.

In a VoIP application, audio quality can be affected by a number of factors. Amongst all packet loss, jitter, and latency take the most part. An objective quality measure based on these parameters is standardized as ITU-T PESQ P.862. A mapping of PESQ score to MOS scale is presented in [4]. Here packet loss, jitter, and latency are measured for a number of VOIP packets arriving at the receiver end. Averaging over arrived number of packets, jitter and latency are combined to form an effective latency which also considers protocol latencies. Finally, the packet loss is combined to effective latency in a scale of 0 to 100 called the R scale. Finally the R scale is mapped to 1 to 5 scale and MOS score is generated. Figure 2 shows the pseudo code excerpt of MOS calculation.

Figure 2. MOS score calculation code excerpt

Tranmission exposure objective

Transmission exposure is a direct measure of the electric-field strength caused by a transmitter. In [5] an in depth calculation of transmission exposure is presented. Exposure at a certain location is a combined measure of radiated power, transmitted frequency and path loss. Now a days regulatory bodies are setting maximum allowable exposure limits in urban areas. For example Brussels, captial city of Belgium, sets transmission exposure limit to 3v/m.

   
Characterizing the exposure model requires calculation of the path loss model specific to the experimentation site. The experimentation site at our testbed has a reference path loss of 47.19 dB at 1 meter and path loss exponent 2.65. Having this measurement, transmission exposure is calculated for each participating nodes and later average the sum over the number of nodes. Figure 3 shows the average exposure model.

Figure 3. Average transmission exposure

Exhaustive searching optimization

The exhaustive searching optimization performs in total 13 channels x 20 TxPower = 260 experiments. Figure 4 and 5 shows performance output of the exhaustive searching optimization.

Figure 4. Audio quality and exposure global accurate model with background interference at channels 1 and 13

Figure 5. Dual objective and contour global accurate model with background interference at channels 1 and 13

The dual objective model shown in figure 5 is combined from the audio quality and exposure models shown in figure 4. Looking into the audio quality model, areas with a higher transmission power have good performance in general. However, there is an area on the non-interfered channel (i.e. 6 to 8), where there is a sudden jump in performance as TxPower increases from 4 dBm to 8 dBm. This area is of interest to us where higher audio quality and lower transmission exposure are seen. Indeed this is shown as a dark red region in the dual objective global accurate model.

Another aspect to look is the area where worst performance is recorded. Look again on the left side of figure 5 between channels 2 to 4, 10 to 12 and TxPower 1 to 7. Interestingly, this region is not located on channels where background interference is applied on but on its neighboring channels. This is due to the fact that the speaker and interferer nodes apply CSMA-CA algorithm on identical channel but not on neighboring channels which results in worst performance [3].

Note: TODO
Click the different areas on figure 4 to hear the audio quality when streamed at the specific design parameters

SUMO toolbox optimization

The exhaustive searching optimization gives the global accurate model however it takes very long time to finish the experimentation. Now we apply SUMO toolbox to shorten the duration and yet achieve a very close optimum value compared to the global accurate model shown in figure 5.

Start experiment execution using the preconfigured files located at the end of this page. The optimization process starts by doing 5 initial experiments selected using Latin Hypercube Sampling (LHS) over the design parameter space. LHS selects sample points on the design space evenly and with minimum sample points it assures providing the best dynamics of the system. For each of the five intial sample points, combined objectives are calculated which forms the inital dataset. The SUMO toolbox reads the inital dataset, creates a surrogate model out of it, and provide the next sample for experimentation to be carried on. The iteration continues until the stopping criteria is met. The stopping criteria selected, at the end of every iteration, sorts the collected combined design objective in ascending order and take the last five elements. Next calculate the standard deviation of this list and stop iteration when the standard deviation falls below threshold. A threshold of 0.05 is selected such that it is twice the standard deviation of a clean repeatable experiment [6]. The idea behind choosing such criteria is that the output of a sorted experiment approaches a flat curve as the optimization reaches the optimum. And the sorted last few elements show a small standard deviation.

Three sets of experiments are conducted and the results are compared to the global accurate model interms of Performance Gain (PG) and Duration Gain (DG) metrics. figure 7 shows the plot of sorted last 5 iteration standard deviation as a function of iteration/duration gain until stopping criteria (i.e. STD_MIN < 0.05) is met.

Figure 7. sorted last 5 iterations standard deviation plot

Shown in the figure above, all solutions stop execution around the 12th iteration (i.e. DG=21.5). Their perfomance gain compared to the global accurate model gives

Solution_ONE PG = [Solution optimum]/[Global accurate optimum] = -0.8261/-0.8482 = 0.9739
Solution_TWO PG = [Solution optimum]/[Global accurate optimum] = -0.8373/-0.8482 = 0.9871
Solution_THREE PG = [Solution optimum]/[Global accurate optimum] = -0.8352/-0.8482 = 0.9846

From the above results we conclude that by applying SUMO toolbox to wireless press conference problem, we improved the optimization process around 21.5 faster than the exhaustive searching optimization. And yet their performance is almost identical.

[1] http://www.ict-fire.eu/fileadmin/events/2011-09-2OpenCall/CREW/Wilabt_crewOpenCallDay.pdf

[2] Carla Schroder, ”Measure Network Performance with iperf” article published by Enterprise Networking Planet, Jan 31, 2007

[3] W. Liu, S. Keranidis, M. Mehari, J. V. Gerwen, S. Bouckaert, O. Yaron and I. Moerman, "Various Detection Techniques and Platforms for Monitoring Interference Condition in a Wireless Testbed", in LNCS of the Workshop in Measurement and Measurement Tools 2012, Aalborg, Denmark, May 2012a

[4] http://www.nessoft.com/kb/50

[5] D. Plets, W. Joseph, K. Vanhecke, L. Martens, “Exposure optimization in indoor wireless networks by heuristic network planning”

[6] M. Mehari, E. Porter, D. Deschrijver, I. Couckuyt,  I. Morman, T. Dhaene, "Efficient experimentation of multi-parameter wireless network problems using SUrrogate MOdeling (SUMO) toolbox", TO COME

Benchmarking and result analysis

Most of the time the steps involved in wireless experimentation are predefined. Start by defining an experiment, execute the experiment, analyze and compare the result. This tutorial explains the last part which is result analysis and performance (aka score) comparison. In the result analysis part, different performance metrics are graphically analyzed. For example, application throughput or received datagram from two experiments can be graphically viewed and compared. In performance comparison part, we do mathematical analysis on a number of performance metrics and come up with objective scores (i.e. 1 to 10). Subjective scores can also be mapped to different regions of the objective score range (i.e. [7-10] good, [4-6] moderate, [1-3] bad) but it is beyond the scope of this tutorial.

Coming back to our tutorial, we were at the end of experiment execution tool where we saved the experiment traces into a tar ball package. Inside the tar ball package there is database.tar file containing all SQLite database files, exprSpec.txt file which holds the experiment specific details and an XML metaData.xmlfile containing the output format of the database files stored. For this tutorial we use three identical optimization problem packages (i.e. Solution_ONE,Solution_TWO, Solution_THREE). Download each package (i.e. located at the end this page) into your personal computer.

Load all tar ball packages into the result analysis and comparison tool each time by pressing the plus icon and press the load button. After loading the files, a new hyperlink START ANALYSIS appears and follow the link. Figure 1 shows front view of the tool during package loading

Figure 1. result analysis and performance comparison tool at glance.

Result Analysis Section

Using the result analysis section, we visualize and compare the bandwidth of Node3 as seen by Node2 for different experiment runs. Click the ADD Graph button as shown in figure 1 above. It is also possible to create as many graphs as needed. Each graph is customized by three subsections. The first is database selection. Here we select different experiment runs that we need to visualize. For this tutorial, three distinct solutions were conducted in search of the optimal bandwidth over the link Node3 to Node2. Figure 2 shows the database selection view of the ISR experiment.

Figure 2. database selection view of the ISR experiment over three distinct solutions.

From the figure above, the search parameter node3_UDP2_bandwidth indicates the tune variable that was selected in the parameter optimization section of experiment execution tool. Moreover, The number of search parameters indicates the level of optimization carried, thus for ISR experiment it is a single parameter optimization problem. The other thing we see on the figure is solutions are grouped column wise and each experiment is listed according to experiment #, run # and specific search parameter value. For example Expr 10/1 @ 13875000 indicates experiment # 10, run # 1 and node3_UDP2_bandwidth=13875000 bit/sec. Before continuing to the other subsections, deselect all experiment databases and select only Expr 1/1 @ 10000000 from all solutions.

The second subsection, graph analysis, selects x-axis and y-axis data sets. To this end, the database meta data file is parsed into structured arrangement of groups, applications, measurement points, and metrics out of which the data sets are queried. Besides this, there are three optional selection check-boxes; AVG/STDEV, LABEL, and RANGE.  AVG/STDEV is used to enable average and standard deviation plotting over selected experiments. LABEL and RANGE turns on and off plot labeling and x-axis y-axis range selection respectively.

The last subsection is custom graph analysis and it has similar function like the graph analysis subsection. Compared to the graph analysis subsection, SQL statements are used instead of parsed XML metadata to define x-axis and y-axis data sets. This gives experimenters the freedom to customize a wide of range of visualizations. Figure 3 shows graph analysis and custom graph analysis subsections of the ISR experiment. For graph analysis, begin_interval and size (i.e. bandwidth) from Node2 group, iperf_UDP application, transfer measurement point are used as x-axis and y-axis data sets respectively. For custom graph analysis, mean interval and size (i.e. bandwidth) are used as x-axis and y-axis data sets respectively.

Figure 3. graph analysis and custom graph analysis subsection view.

Now click the arrow crossed button  in either of the graph analysis subsections to visualize datagram plot for the selected experiments. Figure 4 shows such a plot for six experiments.

Figure 4. bandwidth vs. time plot for three identical experiments.

The first thing we see from the above figure is in each of the three experiments, Node2 reported almost identical bandwidth over the one minute time interval.Second the y-axis plot is zoomed within maximum and minimum result limits. Sometimes, however, it is interesting to see bandwidth plot over the complete y-axis range which is starting from zero up to the maximum. Click the RANGE check box and fill in 0 to 55 in the x-axis range and 0 to 1500000 in the y-axis range. Moreover, in repeated experiments one of the basic things is visualizing the average pattern of repeated experiments and see how much deviation each point has from average. Check the AVG/STDEV check-box (check the SHOW ONLY AVG/STDEV check-box if only AVG/STDEV plot is needed). Figure 5 shows the final modified plot when complete y-axis range and only AVG/STDEV plot is selected.

Figure 5. average bandwidth as a function of time plot from three identical experiments

Performance comparison Section

In performance comparison, a number of experiment metrics are combined and objective scores are calculated in order to compare how good or bad an experiment performs related to other experiments.

For this part of the tutorial, we create a simple score out of 100 indicating how much percent the receive bandwidth reaches and the transmit bandwidth. The higher the score the better the receive bandwidth approaches the transmit bandwidth and vise versa. Start by clicking the ADD Score button and a score calculation block is created ( Note: go further down to reach the score calculation section). A score calculation block has three subsections namely variable definition, score evaluation and database selection.

The variable definition subsection defines SQL driven variables that are used in the score evaluation process. For this tutorial, we create two variables one evaluating the average receive bandwidth and the second evaluating the transmit bandwidth. Click the plus icon twice and enter the following SQL command into the first text-area "SELECT avg(size) from iperf_transfer where oml_sender_id=(select id from _senders where name='node2_UDP')". For the second variable click the icon to select variables from search parameters and select node3_UDP2_bandwidth.

The next subsection, score evaluation, is a simple mathematical evaluator with built in error handling functionality. Coming back to our tutorial, to simulate the percentage of receive bandwidth to transmit bandwidth, enter the string 100*(8*x1)/x2 in the score evaluation text-area ( Note: x1 is multiplied by 8 to change the unit from byte/sec to bit/sec).

Finally, database selection subsection serves the same purpose as was discussed in the result analysis section. Now press the arrow crossed button in the score evaluation subsection. Figure 6 shows the scores evaluated for the ISR experiment.

Figure 6. score evaluation section showing different configuration settings and score results for the ISR experiment.

Performance comparison is possible for identical search parameter settings among different solutions. For example comparing Expr1/1 @ 10000000 experiments from the three solutions reveals that all of them have receive bandwidth about 99.98% of the transmit bandwidth and thus they are repeatable experiments. On the other hand, the scores from any Solution show the progress of the optimization process. Recall the objective function definition (i.e. (8*x1)/x2 < 0.95 => stop when 8*x1 < 0.95*x2) in the ISR experiment such that the IVuC optimizer triggers the next experimentation cycle only when the receive bandwidth is less than 95% of the transmit bandwidth. Indeed on the figure above, search bandwidth decreases as the performance score decreases below 95%. Therefore performance scores can be used to show optimization progress and compare among different Solutions.