© 2000 by British Computer Society
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
Overcoming the Curse of Dimensionality in Clustering by Means of the Wavelet Transform
1 School of Computer Science, The Queen's University of Belfast, Belfast BT7 1NN, Northern Ireland Email: f.murtagh@qub.ac.uk 2 CEA/DSM/DAPNIA, F-91191 Gif-sur-Yvette cedex, France 3 Ayres Hall 114, Department of Computer Science, University of Tennessee, TN 37996-1301, USA
We use a redundant wavelet transform analysis to detect clusters in high-dimensional data spaces. We overcome Bellman's `curse of dimensionality' in such problems by (i) using some canonical ordering of observation and variable (document and term) dimensions in our data, (ii) applying a wavelet transform to such canonically ordered data, (iii) modelling the noise in wavelet space, (iv) defining significant component parts of the data as opposed to insignificant or noisy component parts, and (v) reading off the resultant clusters. The overall complexity of this innovative approach is linear in the data dimensionality. We describe a number of examples and test cases, including the clustering of high-dimensional hypertext data.
Received 11 December, 1998. Revised 14 November, 1999.