Combining games and speech recognition in a multilingual educational environment

Abstract

Playing has been part of people's lives since the beginning of time. However, play does not take place in silence (isolated from speech and sound). The games people play allow them to interact and to learn through experiences. Speech often forms an integral part of playing games. Video games also allow players to interact with a virtual world and learn through those experiences. Speech input has previously been explored as a way of interacting with a game, as talking is a natural way of communicating. By talking to a game, the experiences created during gameplay become more valuable, which in turn facilitates effective learning. In order to enable a game to "hear", some issues need to be considered. A game, that will serve as a platform for speech input, has to be developed. If the game will contain learning elements, expert knowledge regarding the learning content needs to be obtained. The game needs to communicate with a speech recognition system, which will recognise players' speech inputs. To understand the role of speech recognition in a game, players need to be tested while playing the game. The players' experiences and opinions can then be fed back into the development of speech recognition in educational games. This process was followed with six Financial Management students on the NWU Vaal Triangle campus. The students played FinMan, a game which teaches the fundamental concepts of the "Time value of money" principle. They played the game with the keyboard and mouse, as well as via speech commands. The students shared their experiences through a focus group discussion and by completing a questionnaire. Quantitative data was collected to back the students' experiences. The results show that, although the recognition accuracies and response times are important issues, speech recognition can play an essential part in educational games. By freeing learners to focus on the game content, speech recognition can make games more accessible and engaging, and consequently lead to more effective learning experiences.