Curse of Dimensionality

Tags

text-processing adaptive-intelligence

A general point of difficulty in building machine-learning systems [see page 2, where] a high dimensional-space with a modest number of samples is mostly empty. Furthermore the more classes we need to consider, the more features we'll need to consider.

The main problem is given a binary classifier we can generally pretty easily separate terms into either class. Adding another class to this (now ternary classifier) means we:

1. Must introduce every term that may be associated with that class to our feature vectors. More dimensions -> More work, more memory, more data.
2. May reduce the number of terms that can clearly distinguish the existing classes (they may better identify with the new class and so get put into their). Reducing the accuracy for two classes.