UWaterlooMDS at the TREC 2021 Health Misinformation Track

In this report, we discuss the experiments we conducted for the TREC 2021 Health Misinformation Track. For our manual runs, we used an improved version of our high-recall retrieval system [2] to manually search and judge documents. The system is built to efficiently retrieve the most-likely relevant documents based on a Continuous Active Learning (CAL) model and allows a speedy document assessment phase. Using the judged documents, we built CAL models to score documents that are part of our filtered collections. We also experimented with neural reranking methods based on question answering and stance detection methods to modify our CAL-based runs and a traditional BM25 run. For our automatic runs, we filtered the collection by running PageRank with a seed set of reliable domains and then using a text classifier and further refined the collection by including only medical web pages. We then ran traditional BM25 on this smaller and more reliable collection. to score documents with assessors making manual judgements for the track’s topics. The second method implemented a combination of CAL, and the RoBERTa language model [6], where we scored paragraphs using CAL trained on assessors’ manual judgements and then reranked based on RoBERTa to match each topic has given stance field. The last method was to fine-tune T5-Large [10] to acquire a binary classification model to predict the stance of each document. We built our automatic runs using Anserini’s BM25 on our different filtered collections. Results show that in terms of the compatibility measure, using our filtered collections produced runs with better performances than just using the entire collection (as was done to create the baseline run). Based on the nDCG measure, several of our runs achieved higher scores than the baseline run. Precision at 10 (P@10) scores show that the use of our filtered collections produced runs with better credibility. Overall, creating filtered collections allowed for a boost in performance.