This document below describes the CREW methodology for experimental performance evaluation. While the scope of the CREW methodology is the analysis of cognitive networking and cognitive radio solutions, the methodology is broader in a sense that it may be applied to a wider range of (wireless) networking experiments.

The content in this document is largely taken from CREW deliverable D4.2, which is yet to be approved by the European Commission. As soon as deliverable D4.2 is approved, the latter document will supersede the information contained in this document.

Description	Download
CREW methodology for experimentation

In addition to the methodology described in the above document, (a growing list of) more concrete hints and best practices on using the different testbeds is found below.

Experiments using iMinds w-iLab.t

a.General comments

1. Extensive documentation on the w-iLab.t testbed was added to the CREW portal.  Prior to executing any experiment on w-iLab.t, read the documentation on http://www.crew-project.eu/portal/wilabdoc carefully and go through the various tutorials that are listed.  Please also read the FAQ section at http://www.crew-project.eu/content/faq.
2. As listed on the portal, there are two locations in the w-iLab.t testbed.  Choosing the right location for your experiment is an important choice, as –at the moment of writing- the sensor side of the testbed is not yet 100% compatible, and the tools that are used to control the environments are not (yet) identical.  As such, while far from impossible, porting experiments from one environment to another environment takes time. 

b.Good practices for experiments involving USRP devices

1. The USRP is one of the important cognitive components in the w-iLab.t testbed. Each USRP has a specific IP address. Unlike the standard configuration, the USRPs here are first attached to an high speed switch and then to interfaces of several quad-core servers. The alternation of this configuration should be avoided in any case, since it might influence the internal network address for other devices.
2. As the quad-core servers for controlling USRPs are shared by multiple users, it is a good practice to save the operating system’s image each time after the design and experiment, next time reload it again.
3. Timing is a crucial factor for cognitive experiments. OMF control and manage framework keeps very good log by default. The log file is located both on the central experiment controller and on the distributed nodes, which can provide precious information for debugging and improvement.
4. Repeating experiments with nodes at different locations of the testbed is also a good practice. Since not all nodes in w-iLab.t have line-of-sight connection, even the relative positioning of antennas on different nodes can some time affect the experiment result.
5. All the Wi-Fi interfaces in w-iLab.t Zwijnaarde have a 10 dB attenuator attached, which makes it easier to form multi-path scenario. Of course this fact should be kept in mind for experiments including power or pathloss measurement. Note that no attenuators are installed on the USRPs.

Experiments using the IMEC sensing engine

There are 2 instances of the IMEC sensing engine (both are present in the iMinds testbed), the differentiator between the two instances is the radio board. A first version uses the Wireless Open-Access Research Platform (WARP) radio board and the second version uses the IMEC SCALDIO IC. The main difference between the two instances is the RF range, the WARP radio board can only measure in the 2.4 and 5 GHz ISM, whereas the SCALDIO IC can measure signal between 0.1 and 6 GHz. Therefore it is mandatory that the experimenter selects the device that suits the experiment’s frequency requirements. Beware that in the iMinds testbed there are 10 IMEC sensing engines installed and only 2 SCALDIO sensing engines.

Before starting the actual experiment it is good practice to run a calibration/characterization phase if possible. In most cases the ideal sequence for this would be to execute a measurement of known signal and the absence of the signal, e.g. executing a spectrum measurement with a known signal connected directly to the input and executing a spectrum measurement with a 50Ω terminator connected to the antenna input. These measurements can afterwards serve as a reference measurement and can provide e.g. the noise floor of the spectrum sensing operation. Unfortunately connecting a known signal is directly to the input is only possible when one has full physical access to the device, which is not always the case when accessing a testbed remotely. In this case we would recommend transmitting a continuous signal with only one source at a time and measure this signal with all sensing devices. Afterwards the user can repeat this with sources at several locations in the testbed to check if all sensing devices can pick up the signal.

Experiments using IRIS and the CR testbed at TCD

1. Everything in the testbed has an exact place

• Each USRP and node has been assigned a table and number.
• Unused daughterboards will be placed in proper storage places.

2. Everything goes back to the exact place after any experiment that causes it to be moved.
3. Clonezilla is used on all nodes meaning that nodes will be reset to a specific version of IRIS on startup.
4. Bearing this in mind everyone should take care to store data in the data partition and not elsewhere on a node as it will be lost otherwise.
5. The firmware in the USRPs will be updated when a new release becomes stable. All hardware will be updated at once rather than a subsection of hardware.
6. If it is found that any piece of equipment gets broken, or if there is an issue with its functionality (e.g. only works for a certain bandwidth or really low powered) the IRIS testbed users mailing list iris-testbed-users@scss.tcd.ie must be informed. This will be relayed this to the wider group and a note will be made of this on the appropriate wiki pages https://ntrg020.cs.tcd.ie/irisv2/wiki/TestbedInventory.
7. All experiments must be scheduled using the Google calendar <ctvr.testbed> specifying all of the following:

• Name of experimenter
• Date and time of booking
• Testbed node number(s)
• Daughtboard(s) of use
• Frequency range(s) of use

8. The testbed should not be used for simulations.
9. The testbed room should be kept secure.
10. Testbed users should sign up to the following mailing lists:

• IRIS support mailing list https://lists.scss.tcd.ie/mailman/listinfo/iris2
• IRIS testbed users mailing list https://lists.scss.tcd.ie/mailman/listinfo/iris-testbed-users for enquiries regarding the Iris testbed.
• IRIS commit mailing list https://lists.scss.tcd.ie/mailman/listinfo/iris2commit for commit notifications.

11. Short descriptions of all experimental work using the testbed should be provided in the projects section of the IRIS wiki https://ntrg020.cs.tcd.ie/irisv2/wiki/ActProjects.

Experiments using TWIST at TUB

The 2.4GHz ISM band is usually very crowded. Although, we have managed to switch the university wireless access network (eduroam) to the 5GHz ISM band and focus the TWIST testbed activities on 2.4GHz band, being now reserved only for experiments, it is a good practice to monitor the frequency band of interest for unsuspected interferences. The university network was moved to 5GHz band only in the building with the testbed and it is still possible to catch transmissions coming from either other buildings nearby or outdoor mesh network. It is possible to use low cost spectrum analyzers, WiSpy’s, in the TWIST testbed to perform constant monitoring of the spectrum, while performing an experiment. This enables validation of an experiment, before, during and after an experiment as explained in 3.3.1.

Proper description of the experiment and publishing the raw data adds to the transparency of the results. It makes it easy to verify if the experiment performed correctly as well as gives the possibility to explore other aspects using the same set of data. The proper description of the experiment should enable the repetition of the same it later on, also by another experimenter.

There is a set of good practices for experiments involving TUB testbed hardware:

1. Extensive documentation on the TUB testbed can be found on the CREW portal.  Prior to executing any experiment in the TUB testbed please read the documentation on http://www.crew-project.eu/portal/twistdoc carefully and go through the various tutorials that are listed.  Please also read the FAQ section at http://www.crew-project.eu/content/faq.
2. There are dedicated mailing lists, which users can subscribe and ask questions concerning TWIST. The following mailing lists are available:

• <twist-users AT twist.tu-berlin.de>  for discussions on TWIST setup and usage
• <twist-devel AT twist.tu-berlin.de> for discussions on TWIST software development (more information can be found online: http://www.twist.tu-berlin.de/wiki/TWIST/Support/MailingLists)

3. Before an experimenter is granted access to TWIST she/he must agree to the TWIST       terms of usage, which includes

• the nature of the intended experiments (due to the privacy implications)
• commitment to a reciprocal access to eventual testbed resources; and
• agreement on proper acknowledgement of TWIST in the produced publications.

4. All experimenters must be registered via the TWIST web interface
5. When the experimenter uses TWIST she/he can reserve a time slot to have exclusive access to some hardware using a dedicated registration service (via the TWIST web interface)
6. The integration of WiSpy spectrum devices has only recently been finished, therefore spectrum sensing is in an alpha-stage. Experimenters may be asked to provide TUB with feedback after using WiSpy monitoring during an experiment. This allows TUB to improve the service.
7. During an experiment on TWIST experimenters can access data from their system under test in real-time over a separate channel; the connection is realized over SSH and the password will be provided on request.
8. Repeating experiments with nodes using different topologies is a good practice. Furthermore, since the TUB building is a public building that is usually closed after working hours (and on weekends) it is a good practice to compare experimental results for working vs. non-working hours. This allows investigating effects due to external (uncontrolled) RF interference but also to mobility of the environment (affecting multipath fading, etc.).

Experiments using the LTE/LTE advanced testbed at TUD

a.Before the experiment:

• Contact the testbed staff, make sure the hardware is compatible (frequencies) and the testbed supports all features necessary for the intended experiment
• Make sure there will be enough testbed hardware available (indoor/outdoor?)
• If reasonable, ask for reference signal files to check compatibility with external hardware
• Get familiar using the spectrum analyzer R&S FSQ
• Prepare UE/eNB configuration files or ask testbed staff to do it

b.During the experiment:

• Carefully check the setup

• Use a terminator when there is no antenna/cable plugged
• Check if all cables are ok

• Make sure you are using the latest version of the config tool to configure the hardware
• Keep a record of the config files you use as you are changing parameters
• Check the signal on the spectrum analyzer, several things can be validated that way
• If in nothing else seems to work reboot and reconfigure the UE/eNB prototype hardware
• When in doubt, ask the testbed staff

c.After the experiment:

• Put everything back where you took it from

Experiments using LOG-a-TEC testbed (JSI)

1. All experiments have to be scheduled and approved on LOG-a-TEC web portal: https://crn.log-a-tec.eu/. If unavailable or unresponsive, please notify the testbed staff.
2. The communication between application and the testbed is based on a custom protocol, which is abstracted by a proxy server based on standard HTTP protocol.
3. Users have the opportunity to interact with the testbed in three different ways, depending on the difficulty of performed task:

• by using pre-prepared GET and POST requests (trivial),
• by using a web GUI to write GET and POST requests one by one or to put more in a text file and upload it to the server (medium),
• by using a programming language with a HTTP client to write custom code based on GET and POST requests to the sensor network e.g. Python or Java.

4. Several firmware options are preinstalled on VESNA platforms deployed in the LOG-a-TEC testbed and can selected for execution. If an experiment requires a new firmware, this should be prior to upload and execution in the LOG-a-TEC testbed at JSI testbed for proper functioning and compliancy with the hardware used. This testing is required to ensure that the reprogrammed devices are able to join the control network and accept commands from it.
5. In case any of the nodes is inaccessible, please notify operators of the testbed.
6. In LOG-a-TEC testbed there are 3 types of spectrum sensing modules installed, with only one available in the particular device. Make sure that the devices used are equipped with the modules required the experiment.
7. When in doubt or experiencing unexpected behavior, please notify and investigate with the testbed staff.
8. The radio environment on LOG-a-TEC testbed is not controlled, so unexpected interference can appear during experiments, especially in the crowded 2.4 GHz band. From the two clusters of the LOG-a-TEC testbed, one is located in the city center and the other in the industrial zone. Their interference profiles may be substantially different and can differ with respect to time of day (e.g. less Wi-Fi users in the night). In case of potential or experienced problems, please coordinate with the testbed staff.
9. When planning an experiment keep in mind that the control network in LOG-a-TEC testbed is based on a ZigBee network, which occupies only one channel in the 868 MHz frequency band and offers only a low transmission rate. On average 1 kB/s transmission rate can be achieved, and the latency of the network is a few hundred milliseconds. If the data cannot be collected in real-time, SD card storage available in every device should be used.
10. The storage on the SD card can hold traces up to 1 MB. This is enough for storing the results of at least 20 minutes of sniffing in the ISM bands, and 8 hours of sniffing in the UHF bands. During one experiment, more traces can be created, if necessary.
11. Time synchronization of the nodes in the network is not implemented explicitly, so be aware of the possible drift between the traces. For achieving better synchronization, one can start all of the sniffing nodes planned to be used in an experiment, and then transmit a short synchronization burst which to be recorded by all sniffing nodes. Then by using the synchronization burst, the collected traces can be aligned.
12. The GRASS-RaPlaT extension can be used either in experiment planning via simulations or cross validation of simulation and experiment results. Several dedicated processes are pre-prepared and available for the execution. Most common way of accessing the functionalities is through the LOG-a-TEC portal, which calls proper Linux processes and latter graphically visualizes the results. The GRASS-RaPlaT web interface can also be made accessible upon user request. In case experiencing problems, please contact testbed staff.

Experiments using the TCS sensing platform

a.Before going on the field

• Make sure to have all the necessary cables in order to connect the four parts of the test bed (Laptop, acquisition board, receiver, antennas)
• Make sure the hardware is compatible in frequencies with the LTE network to be sensed

b.Once on the field

• Make sure all cables are plugged correctly
• Switch on the acquisition board before booting the laptop.
• Switch on the receiver before launching the SMARTAIR3G software.

c.Acquisition part

• Run the setup test in order to verify that everything works correctly
• Set the acquisition parameters i.e.

• Acquisition length
• Number of antennas
• Path for file storage
• Generic file name

• Change the center frequency to the one to be analyzed
• Check on the time and spectral views that a signal is received
• Verify that the AGC works properly and that the signal is not saturating. If so, adjust the gain manually.

d.Analysis part

• Set the analysis parameters i.e.

• Number of antennas
• Precision of the analysis (number of frames used)
• Path for report file storage

e.After the experiment

• Switch of the laptop before unplugging any other equipment

Experiments using EADS Aircraft Cabin Mock-Up

The EADS aircraft cabin mock-up is a realistic replica of an Airbus A340 cabin using original equipment for the cabin interiors, such as seats, overhead compartments and lining material. The outside structure of the mock-up consists of wood. To make it useable for realistic wireless experimentation, all structural components have been coated with metal foil. In the technically relevant frequency range, due to the skin effect, the metallic foil has the same properties than solid metal parts would have regarding reflection and dispersion of electromagnetic fields. The interior of the mock-up as well as parts of the metallic coating are shown in Figure 23.

Although the mock-up is not part of the CREW federated testbed and not openly accessible, experiments referring to the special environment of an aircraft cabin can be performed according to explicit prior agreement.

Any kind of equipment can temporarily be installed in the mock-up, as long as it does not require irreversible modifications, causes damage at structure or interior, or interferes with the normal usage of the mock-up for tests and demos by EADS. 230 V power sockets are available at multiple sockets in the floor of the cabin and can be used to power the installed equipment. To reduce the occupation time of the mock-up, experimenters should have a precise concept for their planned tests and optimally should have performed a dry run before.

It is also possible to install components of the CREW federated testbed in the cabin mock-up for cognitive radio experimentation. This mobile testbed approach has been used for a series of internal experiments described in deliverable D6.2. Depending on the component selection, usage of all functionalities, such as the common data format or benchmarking is supported. Also a portal for remote access and remote experiment execution in principle can be set up (It is rather difficult to occupy the mock-up for longer time periods, so that in most cases a temporarily condensed experimentation campaign is preferred). It has been shown that installation of the mobile testbed can be done in a timescale of few hours to enable rapid experimentation in an aircraft cabin environment using the known CREW testbed components.