Conceptual clustering and classification of information using vector space model

Abstract

Document clustering is a popular t,ool for organizing a large collection of documents. Clustering algorithms are usually applied on documents, represented as vectors, in a high dimensional term space. The main two problems related to such clu~tering approach are accurately cluster the co-related documents and determine the proper number of clust,ers. The first feature is being analyzed in current literature in different ways including active CltlStering, partitional k-means algorithm, project,ion based methods including LSI, self-organizing maps, multi dimensional scaling, graph-theoretic techniques and many more. As for the second feature most of the clustering approaches assumes the number of clusters as a pre-requisite quantity such in case of Markov State Cluster, partitional methods and most of the graphtheoretic techniques. A few of the clustering algorithms have been analyzed those can automatically determine the number of clusters. A popular approach is based on the idea borrowed from Principal Component Analysis. Another approach uses self-refinement process of discriminative feature identification and cluster label voting to converge to optimal number of clusters. In this work we have implemented iterative solution with inductive knowledge base to achieve the optimal clustering. Both the inter-cluster distance and number of clusters are iteratively varied to have this optimization. This new technique to determine the number of clusters and document clustering shows promising result with 81% percent clustering accuracy. For classification we studied unsupervised clustering technique together with the group vector that also minimizes the computational cost that is usually associated with ordinary classification approaches. The outcome reveals comparable result to current practices and gives 78% classification accuracy.