Quantifying the Bias of Transformer-Based Language Models for African American English in Masked Language Modeling - Equipe Réseaux, Mobilité et Services Access content directly
Conference Papers Year : 2023

Quantifying the Bias of Transformer-Based Language Models for African American English in Masked Language Modeling

Abstract

In the last three years we witnessed the proliferation of innovative natural language processing (NLP) algorithms attempting at solving different tasks and designed for the most diverse applications. Despite groundbreaking transformer-based language models (LMs) have been proposed and widely adopted, the measurement of their fairness with respect to different social groups still remains unsolved. In this paper, we propose and thoroughly validate an evaluation technique to assess the quality and the bias of the predictions of these LMs on transcripts of both spoken African American English (AAE) and Standard American English (SAE). Our analysis reveals the presence of a bias towards SAE encoded by state-of-the-art LMs, like BERT and DistilBERT, a lower bias in distilled LMs and an opposite bias in RoBERTa and BART. Additionally, we show evidence that this disparity is present across all the LMs when we only consider the grammar and the syntax specific to AAE.
Fichier principal
Vignette du fichier
Quantifying_the_Bias_of_Transformer_Based_Language_Models_for_African_American_English_in_Masked_Language_Modeling.pdf (425.24 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04067844 , version 1 (13-04-2023)

Identifiers

  • HAL Id : hal-04067844 , version 1

Cite

Flavia Salutari, Jerome Ramos, Hosein A Rahmani, Leonardo Linguaglossa, Aldo Lipani. Quantifying the Bias of Transformer-Based Language Models for African American English in Masked Language Modeling. PAKDD, May 2023, Osaka, Japan. ⟨hal-04067844⟩
89 View
138 Download

Share

Gmail Facebook X LinkedIn More