Testing is a crucial part of any software development process. Testing is also very expensive: Common estimations list the effort of software testing at 50% of the average budget. Our society increasingly depends on a working information infrastructure for more and more aspects of civic, commercial, and social life, while software at the same time becomes ever more complex. For example, a modern car has up to 100 million lines of software code, and software errors can easily lead to fatal consequences. Improving techniques to identify errors in software is therefore of utmost importance.
Manual testing is common practice in software development. As manually testing a program is a laborious and error prone task, automation is desirable. However, automation requires the user to specify the correct behaviour up-front in terms of a specification, or later by adding test oracles to automatically generated tests - both alternatives are difficult. This problem is obliterated as test quality is usually measured with oracle-agnostic code coverage metrics. In truth, however, a test without a good oracle cannot find software bugs. This is the oracle problem, one of the longest standing and greatest remaining challenges in software testing.
As both writing specifications and writing test oracles is difficult and needs to be done manually, this proposal aims to push automation further by exploring the middle ground: The novel concept of an oracle template allows to specify what should be tested and checked, but crucially, it does not require specifying the expected behaviour. Instead, automated test generation instantiates user-specified oracle templates to concrete tests with oracles, and the developer decides case by case about correctness. Thus, programs can be tested without the developer needing to write a specification or having to suffer through seemingly purposeless generated tests. Because test generation is driven by oracles, all tests have a purpose and the essential oracles required to be effective at finding software bugs.
The novel concept of oracle templates requires extension of the current state of the art in test generation, as current techniques either assume the existence of an automated oracle (e.g. a specification) or focus exclusively on the code. This creates three challenges, which will be addressed in this project:
-- Existing code-based testing techniques focus on reaching points in the code. This project will define the concept of oracle templates, and will explore test generation based on oracle templates as a search problem. Given an oracle template, search-based testing techniques will automatically create instances, which are test cases with oracles.
-- Systematic testing is traditionally driven by the idea that a good test set covers all the code, which completely ignores the test oracle problem. This project will define systematic criteria and corresponding search-based test generation techniques to thoroughly test programs based on oracle templates. These criteria will ensure coverage of oracle templates, but will also ensure that the code is executed and checked by oracles (e.g. by applying mutation and data-flow analysis).
-- It is impossible to take the human out of the software testing loop completely. Oracle templates are an attempt at minimizing the human effort, but the task of writing oracle templates still requires manual effort. Therefore, this project will explore strategies to automatically synthesise oracle templates based on standard testing patterns and usage examples. Ultimately, a developer would have all tests and oracles generated automatically on the click of a button, leaving only the task of confirming correctness of the produced examples.
The success in addressing these challenges will be measured using automated experiments, controlled studies with student subjects, and industrial case studies at Google and Microsoft.
|