Do Tomatoes Sing Skyscrapers? The Curious Case of Absurd Text Generation
Tim Van de Cruys
KU Leuven
The advancement of large language models (LLMs) has markedly improved the coherence and contextual relevance of automatically generated text. However, this progress has paradoxically made it challenging to intentionally generate semantically incoherent utterances, even when specifically prompted to do so. In our study, we investigate the phenomenon of "absurd text generation", i.e. the ability of a model to generate syntactically correct but semantically implausible text. Our study investigates this phenomenon across different models of various sizes, such as GPT-2, GPT-4, and Phi-3. Our analysis focuses on the ability of these models to produce text that is grammatically coherent yet semantically absurd. By analyzing outputs under controlled prompts and varying conditions, we explore how different model architectures and sizes handle this task. Our research assesses selectional preference and syntactic correctness as indicators of the models' performance in generating absurd text. We discuss how the improvement in semantic understanding in modern LLMs has inadvertently limited their ability to produce creatively absurd outputs, contrasting this with the ease of generating such text in earlier, smaller models. Our findings highlight the evolution of LLM capabilities and the implications for creative applications such as humor and artistic expression.