Assessing the impact of academic literacy interventions in higher education : an evaluation design

Abstract

Responsibly designing academic literacy interventions is becoming increasingly important in a higher education environment where resources for student development are scarce. Providing proof of the effectiveness of such interventions is equally important – academic literacy specialists must show that what they are doing has a significant and meaningful impact on student success. If certain aspects of an intervention were shown not to work optimally, these should be addressed. This cycle of providing evidence of an intervention’s successes and shortcomings, and addressing any such shortcomings, is the goal of impact measurement. However, very few studies have attempted to comprehensively measure the impact of academic literacy interventions, probably because measuring impact in the social sciences is a challenging undertaking. The goal of this study was to develop an evaluation design that could be used to effectively and responsibly measure the impact of a wide range of academic literacy interventions. The first step in developing this evaluation design was to survey the available literature on impact measurement, specifically in the field of academic literacy, so as to propose a theoretically sound design and accompanying research instruments. Through a process of inquiry, piloting and critical reflection, several of these research instruments were adapted to make them applicable to the wide variety of academic literacy interventions that are presented in the South African context. After having proposed an initial evaluation design, this design was verified and validated by i) implementing it in measuring the impact of an academic literacy course in the South African context and ii) obtaining feedback from academic literacy specialists from across the country on how the design could be further improved to suit their respective contexts. After having critically reflected on the implementation process, and after having analysed responses from academic literacy specialists, a revised evaluation design and accompanying research instruments were proposed. These should assist researchers in comprehensively and responsibly measuring the impact of a wide range of academic literacy interventions, and consequently benefit the field of academic literacy as a whole.