Designing an “Other Race Effect” test for forensic facial identification experts using the performance of deep networks and untrained humans.

Forensic face examiners outperform untrained participants in face identity matching (Phillips et al., 2018), though it is unclear whether this superiority generalizes to other-race faces. We developed a challenging test that can be performed with the limited time available to professional examiners. To select the most difficult image pairs from a set of Black (n= 3,102) and White (n= 122,728) faces (self-identified race when images were collected), we employed a deep convolutional neural network (DCNN) (Deng et al., 2019) and an experiment with untrained participants. Image pairs (n= 36 per race) were assembled using a DCNN “perceptual” similarity measure. Same-identity (different-identity) image pairs with the lowest (highest) similarity scores were selected from all possible pairs. Untrained participants (White: n= 26, Black: n= 11) judged whether the images showed the same identity or different identities. Ranking by perceptual difficulty, we created a set of 10 Black and 10 White face pairs (half same-identity pairs). “Difficulty” was measured by tallying the number of participants who incorrectly indicated same-identity pair as different identities, and vice versa. Next, we benchmarked the test by computing participants’ accuracy (area under the ROC curve) on the subset of pairs. The test proved challenging for untrained participants [Black participants: (faces: Black= 0.66, White= 0.53); White participants: (faces: Black= 0.56, White= 0.49)]. Participant race, face race, and the interaction did not affect accuracy (p > 0.05). Notably, additional DCNNs performed more accurately on the White face pairs than Black face pairs (Szegedy et al., 2017: Black= 0.72 , White= 1.0; Ranjan et al., 2017: Black= 0.5, White= 0.92). Given that the pattern of performance across race differed for humans and the DCNNs, we conclude that untrained human benchmarks are critical in building a challenging and balanced cross-race test for experts.