Scalable strorage in compressed representation for terabyte data management

Abstract

The emergence of e-application has been creating extremely high volume of data that reaches to terabyte threshold. Many'organizations are producing data that are doubling every year. The conventional data management system is costlier in terms of storage space and processing speed, and sometimes it is unable to handle such huge amount of data. New algorithms and techniques need to develop to store and manipulate these data. The database compression can be used for scalable storage and faster data access. We propose compression based data management system architecture that can be used to handle terabyte level of relational data. The existing compression schemes e.g. HIBASE or Three Layer Database Compression Architecture work well for memory resident data and provide good performance. These are low cost solution for highperformance data management system but are not scalable to manage terabyte level of data. We have developed a disk based columnar multi-block vector structure (CMBVS) that can be used to store relational data in a compressed representation with direct addressability. Parallel data access can be achieved by distributing the vector structure into multiple servers to improve the scalability. The lowest layer of the model is the block structure to store the compressed representation of data. The next higher level is the vector-structure that relates the block structure to an attribute of the relational data model. The structures are capable of carrying out query directly on the compressed form of data. This reduces query time drastically. We have compared our system with conventional relational DBMS. The experimental results show that our system is about twenty five times efficient in storage cost and twenty-seven to seventy-seven times faster in retrieval time performance than that of the conventional systems.