Testing cyber-physical systems with explicit output coverage

When testing safety-critical systems such as cyberphysical systems, it is important to observe the system in different operating conditions. Moreover, it is helpful for the developers if a failure can be observed under distinct conditions. Exploring a system in a variety of situations can be achieved by solving an output requirement problem, that is, by finding system inputs that together satisfy a set of output test requirements. In this paper, we address the challenge of solving the output requirement problem for a given set of output test requirements for a given deterministic black-box system with real-valued signal inputs and outputs. We focus on output test requirements specified in signal temporal logic which means that solving the output requirement problem can be transformed into solving several optimization problems that ask to minimize fitness functions related to each requirement. We propose a novel Explicit Output Coverage (EOC) algorithm that solves the optimization problems by training online and concurrently multiple generative machine learning models which share a common training data. We evaluate the EOC algorithm on a problem that concerns the validity of the lane keeping assist system of an autonomous car and compare it against two baseline random search algorithms and a sequential version of EOC where sharing of training data between models is disabled. The results show that EOC achieves the highest requirement coverage and witness frequency and is the most efficient in the sense that it requires fewest system evaluations to achieve high coverage and witness frequency.