QuTI! Quantifying Text-Image Consistency in Multimodal Documents

Eric Müller-Budack, Jonas Theiner, Sebastian Diering, Maximilian Idahl, Ralph Ewerth: Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency. In: ACM International Conference on Multimedia Retrieval (ICMR), Dublin, Ireland, June 8-11, 2020, ACM, pages 575-592, Best Paper Award.
You can click on the image to change it
Responsive image
No publisher information

Example Documents:

Named Entity Linking:

Calculate Cross-modal Similarity for:

+−
Leaflet | © OpenStreetMap contributors
(1) Specify an image-text pair by pasting an URL or by selecting an image (click on the image) and text manually. (2) The results of the named entity linking will be displayed in this text box. Detailed information on the entities from Wikipedia and Wikidata are provided while hovering over an entity. (3) After the named entity linking has been applied, you can choose to measure the cross-modal similarity regarding three entity types (persons, locations, and events) by clicking on the respective button. (4) The cross-modal similarities will be shown in this text box. Furthermore, the retrieved reference images from Bing can be accessed by hovering over an entity.