Can ChatGPT Match Experts? Comparing input for Serious Game Development

Abstract

This paper investigates the validity of ChatGPT as a tool to generate meaningful input for the serious game design process. Input is collected from game designers, students and teachers via surveys, individual interviews and group discussions inspired by a description of a simple educational drilling game and its context of use. In these mixed methods experiments ChatGPT 3.5 and 4.0 is prompted with the same description to validate findings with expert participants. In addition, the impact on the models' suggestions from integrating the expert's role (e.g., "Answer as if you were a teacher.", "game designer", or a "student") into the prompt is investigated. The findings show that ChatGPT can produce statistically similar input, depending on the group of experts. ChatGPT 3.5 outperforms 4.0 with the student input. The integration of expert's role in prompt is found to be unreliable, and necessary only with game designer input with the version 3.5 of ChatGPT. The practical implications are that multiple ChatGPT versions should be used when collecting input. In addition, it is shown that experts can provide unique insight to the development process. This research opens the discussion on the trustworthiness of ChatGPT generated input for serious game development.