Multimodal Analytics in Real-world News

URL: https://labs.tib.eu/newsanalytics

This tool serves for multimodal analysis of news articles and evaluates the probability of the intermodal occurrences of entities in text and image information. It uses natural language processing techniques to extract persons, places and events from news articles. Based on a web search, suitable images for the entities mentioned in the text are automatically downloaded from Microsoft Bing and then compared with the news image. In this context, state-of-the-art computer vision approaches are applied to extract visual information, which is ultimately used to evaluate the cross-modal occurrence of the entities. The tool enables users to evaluate the occurrence of persons, places and events in news texts and images in order to analyze them with regard to cross-modal relationships and in some cases even to detect false information.

The tool is based on the paper:

Eric Müller-Budack, Jonas Theiner, Sebastian Diering, Maximilian Idahl, and Ralph Ewerth. 2020. Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency. In Proceedings of the 2020 International Conference on Multimedia Retrieval (ICMR '20). Association for Computing Machinery, New York, NY, USA, 16–25. Best Paper Award. DOI: https://doi.org/10.1145/3372278.3390670