lllusion of Plausibility with Large Language Models
Baoyi Zeng
Faculty of Engineering Science, KU Leuven, Leuven, Belgium
Leonidas A. A. Doumas
School of Philosophy, Psychology, and Language Sciences, The University of Edinburgh, Edinburgh, UK
Humans exhibit both illusions of grammaticality and plausibility when interference is caused by an attractor. Using materials from previous psycholinguistic studies, we test whether large language models are also susceptible to the latter linguistic illusion. We examine the model's performance in three ways: (a) by recording surprisal scores under different conditions; (b) by a plausibility judgement task; and (c) by identifying relevant attention heads and examining their attention patterns. A plausibility effect is observed for surprisal scores and plausibility ratings. GPT-2 and GPT-3 both exhibit effects similar to the illusion of plausibility but GPT-3 yields a more human-like pattern, where interference only affected surprisal scores under implausible conditions. Our analysis of the model's attention patterns yielded promising but non-significant results. This work extends existing research on language model behaviour and its potential parallels with human cognitive processes in language comprehension.