EKGRL: Entity-Based Knowledge Graph Representation Learning for Fact-Based Visual Question Answering

Fact-based Visual Question Answering (FVQA) is a task aiming at answering question based on given image and external knowledge associated with it. The reasoning abilities of current FVQA models including query-based and joint learning methods are insufficient. To achieve stronger reasoning ability, we propose an entity-based knowledge graph representation learning (EKGRL) method. Our model achieves state-of-the-art performance on FVQA dataset. Furthermore, we build a psychological fact-based VQA dataset (PFVQA) containing 6129 questions from six different types, which is, as far as we know, the first VQA dataset built on psychological knowledge. We demonstrate that EKGRL continues to achieve state-of-the-art performance on PFVQA, showing the ability to maintain a good performance on reasoning and knowledge representation based on external knowledge from both commonsense and psychological domains.