Project Name: FITTEST – Future Internet Testing
Duration: September 2010 – September 2013 (36 months)
Project coordinator: Tanja Vos
download the FITTEST papers and reports http://crest.cs.ucl.ac.uk/fittest/
download the FITTEST Tools http://code.google.com/p/fittest/
- Universidad Politecnica de Valencia (Spain)
- University College London (United Kingdom)
- Berner & Mattner (Germany)
- IBM (Israel)
- Fondazione Bruno Kessler (Italy)
- Universiteit Utrecht (The Netherlands)
- Softteam (France)
- Sulake (Finland)
The Future Internet (FI) will be a complex interconnection of services, applications, contents and media, possibly augmented with semantic information and based on technologies that offer a rich user experience, extending and improving current hyperlink-based navigation. Our society will be increasingly dependent on services built on top of the FI, for critical activities such as public utilities, social services, government, learning, finance, business, as well as entertainment. As a consequence, the applications running on top of the FI will have to satisfy strict and demanding quality and dependability standards.
FI applications will be characterized by an extreme level of dynamism. Most decisions made at design or deploy time are deferred to the execution time, when the application takes advantage of monitoring (self-observation, as well as data collection from the environment and logging of the interactions) to adapt itself to a changed usage context. The realization of this vision involves a number of technologies, including:
- Observational reflection and monitoring, to gather data necessary for run-time decisions.
- Dynamic discovery/composition of services.
- Hot component loading and update.
- Structural reflection, for self-adaptation.
- Asynchronous communication.
- High configurability and context awareness.
- Composability into large systems of systems.
While offering a major improvement over the currently available Web experience, complexity of the technologies involved in FI applications makes testing them extremely challenging. Some of the challenges are:
CH1: Self modification. Rich clients have increased capability to dynamically adapt the structure of the Web pages; server-side services are replaced and recomposed dynamically based on Service Level Agreements (SLA) and newly discovered services; components are dynamically loaded
CH2: Autonomic behaviour. FI applications are highly autonomous; their correct behaviour cannot be specified precisely at design-time.
CH3: Low observability. FI applications are composed of an increasing number of 3rd-party components and services, accessed as a black box, which are hard to test.
CH4: Asynchronous interactions. FI applications are highly asynchronous and hence hard to test. Each client submits multiple requests asynchronously; multiple clients run in parallel; server-side computation is distributed over the network and concurrent.
CH5: Time and load dependent behaviour. For FI applications, factors like timing and load conditions make it hard to reproduce errors during debugging.
CH6: Huge feature configuration space. FI applications are highly customisable and self-configuring, and contain a huge number of configurable features, such as user-, context-, and environment-dependent parameters.
CH7: Ultra-large scale. FI applications are often systems of systems; traditional testing adequacy criteria cannot be applied, since even in good testing situations low coverage will be achieved.
The FITTEST project aims at addressing these testing challenges. Search Based Software Testing (SBST) is the FITTEST basis for test case generation, since it can be adopted even in the presence of dynamism and partial observability that characterize FI applications. The key ingredients required by search algorithms are:
- an objective function, that measures the degree to which the current solution (e.g. test case.) achieves the testing goal (e.g., coverage, concurrency problems, etc.);
- operators that modify the current solution, producing new candidate solutions to be evaluated in next iterations of the algorithm.
Such ingredients can usually be obtained even in the presence of high dynamism and low observability. Hence, it is expected that most testing techniques that will be developed to address the project objectives will take advantage of a search based approach.
For example, model-inference for model-based testing requires execution scenarios to limit under-approximation. We can generate them through SBST, with the objective of maximizing the portion of equivalent states explored.
Oracle learning needs training data that are used to infer candidate specifications for the system under test. We can select the appropriate executions through SBST, using a different objective function (e.g., maximizing the level of confidence and support for each inferred property).
We will generate classification-trees from models and use SBST to generate test cases for classification-trees
Input data generation and feasibility check are clearly good candidates to resort to search based algorithms, with an objective function that quantifies the level to which a test case gets close to satisfying a path of interest.
In concurrency testing, test cases that produce critical execution conditions are given a higher objective value defined for this purpose.
Coverage targets can be used to define the objective function to be used in coverage and regression testing of ultra-large scale FI systems.
SBST represents the unifying conceptual framework for the various testing techniques that will be adapted for FI testing in the project. Implementations of such techniques will be integrated into a unified environment for continuous, automated testing of FI applications as illustrated in the figure. Quantification of the actually achieved project objectives will be obtained by executing a number of case studies, designed in accordance with the best practices of empirical software engineering.