Systematic, Stereotypical or an Educated Guess? Gender inference by humans and machine translation systems of words in and out of context
Janiça Hackenbuchner, Aaron Maladry, Arda Tezcan, Joke Daems
Ghent University
Machine translation (MT) systems exhibit gender bias by unfairly discriminating against individuals or groups of people (Savoldi et al. 2021). Unlike MT systems, humans rely on contextual information to infer a person’s gender (Vanmassenhove et al. 2018). In ambiguous cases in the absence of grammatical cues (e.g. pronouns), however, both a human and a system has to infer a person’s gender and make an “educated guess”. When translating from a notional gender language to a grammatical gender language, state-of-the-art MT systems continue to exhibit gender bias by systematically assigning a stereotypical male or female norm or by opting for the generic masculine. The generic masculine is intended as generic but interpreted as masculine (Gygax et al. 2008), excluding half the world’s population (Vanmassenhove et al. 2018).
We conducted an annotation study to analyse to what extent MT systems and humans base their gender translations and associations on role names (words referring to a person, e.g. therapist) and on stereotypicality in the absence of (generic) grammatical gender cues in language. We compare an MT system’s choice of gender for a certain word in translation with the gender associations of humans both for words in isolation (out-of-context) and words in sentence contexts. We look at English, a notional gender language, and the translations into German and Spanish, two grammatical gender languages.
Previous work on word level shows that a large number of word embeddings, on which MT systems are trained, carry an inherent gender inflection (Caliskan et al. 2022). Furthermore, limited research on sentence level shows that occupation nouns and adjectives greatly influence the gender inflection in a machine translated target sentence. In our study, we combined and extended both the word-level and sentence-level research to focus on how the entire source sentence context influenced the gender inflection of a role name in a target translation. To analyse this, annotators were asked to specifically annotate which words (of which we analysed the parts-of-speech [POS]) in a sentence context they assumed to influence the gender of a certain role name.
In our work, we highlight the demonstrable impact that context has on the gender association and on machine translation. We underline that in the absence of grammatical gender cues in language, human associations of gender are much more varied and greatly influenced by context, whereas MT primarily translates into the generic masculine, with the exception of certain stereotypical norms. We will present a detailed analysis of (1) the comparison of human gender associations and MT gender translations for words out of context and in context, (2) POS patterns of the annotated words that influence gender translation, (3) how human gender associations were influenced by individual factors (including their own gender and implicit association tests).
References
Caliskan, Aylin; Parth Ajay, Pimparkar; Charlesworth, Tessa; Wolfe, Robert; Banaji, Mahzarin R. (2022): Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics. AIES '22: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society: 156–170.
Gygax, Pascal, Ute Gabriel, Oriane Sarrasin, Jane Oakhill, and Alan Garnham. 2008. Generically intended, but specifically interpreted: When beauticians, musicians, and mechanics are all men. In LANGUAGE AND COGNITIVE PROCESSES, volume 23:3, pages 464–485.
Savoldi, Beatrice; Gaido, Marco; Bentivogli, Luisa; Negri, Matteo; Turchi, Marco (2021): Gender bias in machine translation. Transactions of the Association for Computational Linguistics 9: 845–874.
Vanmassenhove, Eva; Hardmeier, Christian; Way, Andy (2018): Getting Gender Right in Neural Machine Translation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics: 3003– 3008.