Hierarchical graph attention networks for multi-modal rumor detection on social media

The wide spread of rumors across online microblogs has caused a series of adverse impacts on our daily lives. Traditional multi-modal rumor detection models ignore the investigation of deep fusion of different granularity within inter-modality and intra-modality, leading to the information loss and model inexplicability. In this paper, we propose a novel Hierarchical Graph Attention network based framework for Multi-Modal Rumor Detection (HGA-MMRD). More specifically, for a given multi-modal post, we first construct a fine-grained text-image graph which consists of three sub-graphs where the nodes include four types of information (e.g., words, entities, objects from images, and image patches), while six types of edges are built to capture the different semantic interactions of the intra-modality and the inter-modality simultaneously. In order to fuse multi-modal information at different levels, we adopt a hierarchical graph attention network to model each sub-graph of the fine-grained text-image graph, and employ a global feature alignment module to fuse an entire image and a text. Our extensive experimental results and visualization demonstrate that our HGA-MMRD outperforms state-of-the-art methods on five benchmark datasets (i.e., two English datasets and three Chinese datasets) in rumor detection.