Interpreting Subjectivity Regressor Trained on Native English Speakers’ Judgments: Theory vs. Speaker Intuitions
Elena Savinova, Jet Hoek
Centre for Language Studies, Radboud University
Subjectivity analysis, i.e. identifying opinions, attitudes, beliefs and private states in a text, is often approached as a preparatory binary task for fact-checking or sentiment analysis. However, according to linguistic theories, subjectivity is rather a gradual concept. Moreover, because of this applied approach to subjectivity, the problem of explainability of state-of-the-art subjectivity detection models has largely been overlooked by scholars. To address the issue of the continuous nature of subjectivity, we earlier created a transformer-based sentence-level subjectivity regressor trained on native English speakers' judgements. In this contribution, we look into the explainability of our regressor, applying a combination of two approaches: 1) a top-down approach using manual selection of theoretically defined subjectivity features and 2) a bottom-up approach using an automatic local interpretable model-agnostic explanation method (LIME). We evaluate the explainability of the regressor on a British news dataset containing sentences taken from Facebook news posts and from articles on the websites of the same news outlets. For the top-down approach, we annotated a number of theoretically defined subjectivity features in the dataset and built a linear model to estimate the importance of these features as subjectivity predictors. Although all theory-based features were found to be significant, the results showed that emoji, first and second person pronouns, exclamations, questions, intensifiers (“really”, “very”) and epistemic phrases (“I think”, “I believe”) are the most prominent features. With the bottom-up approach, we applied LIME to every sentence in the dataset, extracted the mean weights associated with every word/feature and analyzed the top 200 subjective and objective features with a frequency above 5 in our dataset. The analysis revealed that the top objective LIME features did not contain any theory-based subjectivity features and instead included numbers, dates and times, proper nouns, verbs and non-evaluative adjectives. On the contrary, the top subjective LIME features included all theory-based subjectivity features and were dominated by evaluative adjectives. The results of both approaches thus indicate that the features used by the subjectivity regressor align with subjectivity theory. Since the regressor was trained on native English speakers' intuitions and, therefore, represents an average speaker's perception of subjectivity, our findings also indicate that the naïve speakers' perceptions correspond to theoretical accounts of subjectivity. In addition, we analyzed whether the distinction between author subjectivity and subjectivity of reported sources was important for the regressor by looking at the effects of theory-based subjectivity features within author and source text separately. The results showed that for four features, their presence in the author text led to a larger change in subjectivity score as compared to the source text, which is expected if the author vs. source distinction is salient. However, the absence of interactions for the other subjectivity features and the main effect of the source text being more subjective suggest that quotations from third person sources are still considered subjective by the model and native speakers, which is not in line with theory. Future work is needed to investigate author vs. source subjectivity distinction in speakers' intuitions more comprehensively, potentially within context.