Building a multimodal entity linking dataset from tweets

Omar Adjali (1) , Romaric Besançon (1) , Olivier Ferret (1, 2) , Hervé Le Borgne (1) , Brigitte Grau (3, 4)

1 LIST (CEA) - Laboratoire d'Intégration des Systèmes et des Technologies
2 DIASI (CEA, LIST) - Département Intelligence Ambiante et Systèmes Interactifs
3 ENSIIE - Ecole Nationale Supérieure d'Informatique pour l'Industrie et l'Entreprise
4 LIMSI - Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur

The task of Entity linking, which aims at associating an entity mention with a unique entity in a knowledge base (KB), is useful for advanced Information Extraction tasks such as relation extraction or event detection. Most of the studies that address this problem rely only on textual documents while an increasing number of sources are multimedia, in particular in the context of social media where messages are often illustrated with images. In this article, we address the Multimodal Entity Linking (MEL) task, and more particularly the problem of its evaluation. To this end, we propose a novel method to quasi-automatically build annotated datasets to evaluate methods on the MEL task. The method collects text and images to jointly build a corpus of tweets with ambiguous mentions along with a Twitter KB defining the entities. We release a new annotated dataset of Twitter posts associated with images. We study the key characteristics of the proposed dataset and evaluate the performance of several MEL approaches on it.

Mots clés: Entity linking; social media; multimodality; multimedia entity linking

Domaines: Intelligence artificielle [cs.AI]; Vision par ordinateur et reconnaissance de formes [cs.CV]

Omar Adjali, Romaric Besançon, Olivier Ferret, Hervé Le Borgne, Brigitte Grau. Building a multimodal entity linking dataset from tweets. LREC 2020 - Language Resources and Evaluation Conference, May 2020, Marseille, France. pp.4885-4292. ⟨hal-04315504⟩