Guidelines For Benchmarking Automated Software Traceability Techniques

To comparatively evaluate automated traceability solutions, we need to develop standardized benchmarks. However there is currently no consensus on how a benchmark should be constructed and used to evaluate competing techniques. In this paper we discuss recurring problems in evaluating traceability techniques, identify essential properties that evaluation methods should possess, and provide guidelines for benchmarking software traceability techniques. We illustrate the properties and guidelines using empirical evaluation of three software traceability techniques on nine data sets.