Variable Discovery with Large Language Models for Metamorphic Testing of Scientific Software.

When testing scientific software, it is often challenging or even impossible to craft a test oracle for checking whether the program under test produces the expected output when being executed on a given input – also known as the oracle problem in software engineering. Metamorphic testing mitigates the oracle problem by reasoning on necessary properties that a program under test should exhibit regarding multiple input and output variables. A general approach consists of extracting metamorphic relations from auxiliary artifacts such as user manuals or documentation, a strategy particularly fitting to testing scientific software. However, such software typically has large input-output spaces, and the fundamental prerequisite – extracting variables of interest – is an arduous and non-scalable process when performed manually. To this end, we devise a workflow around an autoregressive transformer-based Large Language Model (LLM) towards the extraction of variables from user manuals of scientific software. Our end-to-end approach, besides a prompt specification consisting of few examples by a human user, is fully automated, in contrast to current practice requiring human intervention. We showcase our LLM workflow over three case studies of scientific software documentation, and compare variables extracted to ground truth manually labelled by experts.