Evaluating User trust for the FarmBuddy chatbot: a mixed methods study
Mario Perez Eguiluz
TNO; Leiden University
Background: Trust has been found to play a pivotal role in the way that interactions take place between users and chatbots, as well as significantly impacting technology adoption, continuance intention and user behavior. The evaluation of trust in relation to chatbots has been a subject of exploration in the existing literature. However, to my knowledge, there are currently no tools or instruments available, which are specifically designed for assessing user trust in chatbots used as information systems. Furthermore, gaining insight on the factors that affect user trust, and to what extent this occurs, can ultimately lead to improvements in the development or design of chatbots as information systems, such as FarmBuddy. FarmBuddy is a chatbot that is being developed and will be implemented in the EU-FarmBook project, in order to assist users in finding information related to the agriculture and forestry sectors. Via the use of a mixed methods study, this research seeks to enhance the validity and reliability of its findings, ultimately contributing to a more nuanced understanding of the findings related to the dynamics of trust found in chatbot interactions. Taking into account the novelty of the chatbot there are no previous studies assessing the perceived user trust and the factors affecting this trust towards FarmBuddy.
Objective: Through this study I aim to develop a validated questionnaire that will not only assess the latent variable of user trust in chatbots as information system, but also provide an in-depth understanding corresponding to the factors affecting trust implemented in the survey. One of the objectives behind this study is for this validated questionnaire to be used by other researchers from the field and facilitating future research on this topic. Furthermore, via the use of a semi-structured interview, further insight will be gained regarding the findings of the questionnaire, as well as any potentially significant factors that were not present in the survey. In the literature review I delve into the importance of trust as a component of user experience as well as how this impacts the users’ continuance intention towards chatbots. Finally, I will be using the validated questionnaire and semi-structured interviews in order to assess the factors influencing user trust in relation the FarmBuddy chatbot as well as evaluating user trust overall towards the aforementioned system.
Methods: The study distributed the questionnaire to FarmBuddy users and used the responses obtained to develop the following 11 latent constructs, via the use of a 7-point Likert scale; Familiarity, Propensity to Trust, Competence, Understandability, Appearance, Brand, Human-likeness, Likeability, Predictability, Transparency, and Risk. Besides the latent constructs, we gather demographic information for further observation. The study uses Partial Least Squares Structural Equation Modeling (PLS-SEM) in order to analyze the data obtained. After the process of data collection and analysis, participants are asked to take part in the semi-structured interview.