Skip to Main content Skip to Navigation
Conference papers

French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English

Aurélie Névéol 1, 2 yoann Dupont 3 Julien Bezançon 3 Karën Fort 4, 3 
2 ILES - Information, Langue Ecrite et Signée
LISN - Laboratoire Interdisciplinaire des Sciences du Numérique, STL - Sciences et Technologies des Langues
4 SEMAGRAMME - Semantic Analysis of Natural Language
Inria Nancy - Grand Est, LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery
Abstract : Warning: This paper contains explicit statements of offensive stereotypes which may be upsetting. Much work on biases in natural language processing has addressed biases linked to the social and cultural experience of English speaking individuals in the United States. We seek to widen the scope of bias studies by creating material to measure social bias in language models (LMs) against specific demographic groups in France. We build on the US-centered CrowS-pairs dataset to create a multilingual stereotypes dataset that allows for comparability across languages while also characterizing biases that are specific to each country and language. We introduce 1,677 sentence pairs in French that cover stereotypes in ten types of bias like gender and age. 1,467 sentence pairs are translated from CrowS-pairs and 210 are newly crowdsourced and translated back into English. The sentence pairs contrast stereotypes concerning underadvantaged groups with the same sentence concerning advantaged groups. We find that four widely used language models (three French, one multilingual) favor sentences that express stereotypes in most bias categories. We report on the translation process, which led to a characterization of stereotypes in CrowS-pairs including the identification of US-centric cultural traits. We offer guidelines to further extend the dataset to other languages and cultural environments.
Document type :
Conference papers
Complete list of metadata
Contributor : Karën Fort Connect in order to contact the contributor
Submitted on : Monday, April 4, 2022 - 3:00:07 PM
Last modification on : Wednesday, April 13, 2022 - 9:14:59 AM


Files produced by the author(s)


  • HAL Id : hal-03629677, version 1


Aurélie Névéol, yoann Dupont, Julien Bezançon, Karën Fort. French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, May 2022, Dublin, Ireland. ⟨hal-03629677⟩



Record views


Files downloads