An empirical evaluation of the subtlety of the data-flow based higher-order mutants

Mutants are erroneous forms of the source code generated by deliberately inserting one fault (first-order mutant) or more (higher-order mutant) into the source code. Smart mutants that require numerous numbers of test cases to be killed are called Subtle Mutants (SMs). These mutants are required in order to increase the efficiency and effectiveness of test cases. Creation of these mutants is an expensive step especially in higher-order mutation testing. Data-flow analysis has been effectively applied to create higher-order mutants and overcome the explosion problem. To the best of our knowledge, subtle mutant generation with the aid of data-flow concepts and identifying them among all mutants have not been studied adequately. In this paper, an empirical study to evaluate the impact of data-flow analysis on the subtlety of higher-order mutants is introduced. Therefore, this study discusses two research questions: which mutants are more subtle data-flow based second order mutants (DFSOMs) or their constitute FOMs mutants? And which mutants are harder to be killed or covered DFSOMs or all du-pairs criterion? The results of the conducted experiments showed that the subtlety of data-flow based second order mutants (DFSOM) is higher than their constitute first-order mutants by 6% in average. In addition, DFSOM criterion dominates all du-pairs criterion and covering (killing) DFSOM criterion is harder than covering all du-pairs criterion by 14.6% in average.