Configuration selection using code change impact analysis for regression testing

Configurable systems that let users customize system behaviors are becoming increasingly prevalent. Testing a configurable system with all possible configurations is very expensive and often impractical. For a single version of a configurable system, sampling approaches exist that select a subset of configurations from the full configuration space for testing. However, when a configurable system changes and evolves, existing approaches for regression testing select all configurations that are used to test the old versions for testing the new version. As demonstrated in our experiments, this retest-all approach for regression testing configurable systems turns out to be highly redundant. To address this redundancy, we propose a configuration selection approach for regression testing. Formally, given two versions of a configurable system, S (old) and S' (new), and given a set of configurations CS for testing S, our approach selects a subset CS' of CS for regression testing S'. Our study results on two open source systems and a large industrial system show that, compared to the retest-all approach, our approach discards 15% to 60% of configurations as redundant. Our approach also saves 20% to 55% of the regression testing time, while retaining the same fault detection capability and code coverage of the retest-all approach.