Distributed data warehouse management in a parallel environment using compressed relational structure

Abstract

Data warehousing is a key technology in everyday activity, which usually contains historical data derived from transaction data, but it can include data from other sources. The objective of a data warehouse is to provide analysts and managers with strategic information underlying the business to consolidate data from several sources. Unfortunately, the emergence of e-application has been creating extremely high volume of data that reaches to .terabyte threshold. The conventional data warehouse management system is costlier in terms of storage space and processing speed, and sometimes it is unable to handle such huge amount of data. As a result, queries and analyses are becoming more complex and time consuming. Therefore, there is a crucial need for the new algorithms and techniques to store and manipulate these data. Parallel and distributed data warehouse architectures have been evolved to support online queries on massive data in a short time. The database compression can be used for scalable storage and faster data access. In this thesis, we have presented a compressionbased distributed data warehouse architecture for storage of warehouse data, and support online queries efficiently. We have achieved a factor of 25-30 compression compared to conventional SQL server data warehouse. The main computational component of data warehouse is the generation and querying on the data cube. Our algorithm generates data cube directl/from the compressed form of data in parallel. The reduction in the size of data cube is a factor of 30-45 compared to existing methods. The response time has also been significantly improved. These improvements are achieved by eliminating the suffix and prefix redundancy, virtual nature of the data cube, direct addressability of compressed form of data and parallel computation. Experimental evaluation shows the improved performance over the existing systems.