Building and Using Comparable Corpora for Multilingual Natural Language Processing

Serge Sharoff (1) , Reinhard Rapp (2) , Pierre Zweigenbaum (3)

1 University of Leeds
2 HSMD Hochschule Magdeburg-Stendal
3 STL - Sciences et Technologies des Langues - LISN

This book provides a comprehensive overview of methods to build comparable corpora and of their applications, including machine translation, cross-lingual transfer, and various kinds of multilingual natural language processing. The authors begin with a brief history on the topic followed by a comparison to parallel resources and an explanation of why comparable corpora have become more widely used. In particular, they provide the basis for the multilingual capabilities of pre-trained models, such as BERT or GPT. The book then focuses on building comparable corpora, aligning their sentences to create a database of suitable translations, and using these sentence translations to produce dictionaries and term banks. Then, it is explained how comparable corpora can be used to build machine translation engines and to develop a wide variety of multilingual applications.

Keywords: Natural Language Processing; Machine Translation; Parallel Corpora; Comparable Corpora; Multilingual Corpora; Multilingual Embeddings; Sentence Alignment; Word Alignment

Domains: Computation and Language [cs.CL]; Artificial Intelligence [cs.AI]; Machine Learning [cs.LG]; Document and Text Processing

Serge Sharoff, Reinhard Rapp, Pierre Zweigenbaum. Building and Using Comparable Corpora for Multilingual Natural Language Processing. Springer Cham, 2023, Synthesis Lectures on Human Language Technologies, Graeme Hirst, 978-3-031-31383-7. ⟨10.1007/978-3-031-31384-4⟩. ⟨hal-04470213⟩