An investigation of the suitability of agile system development methodologies for the development of data warehouses

Abstract

The aim of this study is to investigate whether agile system development methodologies (ASDMs) are suitable for the development of data warehouses. To reach this aim, a literature study was conducted on the relatively settled ASDMs by firstly defining a system development methodology (SDM) and an ASDM. Each ASDM explanation contains the identified key factors, unique process model, and method of use. The seven ASDMs investigated in this study, are: Dynamic System Development Methodology (DSDM), Scrum, Extreme Prograrr~ming (XP), Feature Driven Development (FDD), Crystal ASDMs - especially Crystal Clear (CC), Adaptive Software Development (ASD), and Lean Development (LD). In addition, a literature study is conducted on the data warehouse approaches of lnmon (1996) and Kimball et a/. (1998). Each data warehouse approach is explained using the architecture, lifecycle and four distinct phases within the lifecycle. The four distinct phases include: collecting requirements, data modelling, data staging, and data access and deployment. After this was done, lnmon and Kimball's approaches were compared. After studying the ASDMs and data warehousing approaches, theoretical deductions were made regarding the suitability of ASDMs in data warehouse development. General deductions (including the applicability of agile processes) for all ASDMs as well as unique deductions for each of the seven ASDMs mentioned above were formulated in theory. The theoretical deductions lead to the limitation of the empirical section of the study to the suitability of ASDMs within the framework of Kimball's approach. Theoretical deductions were empirically tested by conducting an interpretive experiment where seven data warehouse development teams used an assigned ASDM to develop a data warehouse. The data warehouse consisted of one data mart. Each team was expected to develop their data mart incrementally, one sub-data mart at a time. Every sub-data mart was developed iteratively to form the data mart. The data mart was then deployed as a whole (including everything from collecting requirements to report generation) to the users. The findings of the study are a combination of the theoretical deductions and interpretive results (propositions) of the interpretive experiment conducted. These findings indicate that ASDMs are suitable to develop data warehouses in a constantly changing environment.