From: Ken Bloom <kbloom@...>
Date: 2007-02-07T08:10:08+09:00
Subject: Re: Classifier::Bayes - handling "none of the above" cases

On Wed, 07 Feb 2007 02:14:23 +0900, Giles Bowkett wrote:

>> > Imagine three categories (shopping, health, and technology) and I want
>> > to classify the text "cows dirt barn". Obviously, those words aren't a
>> > good fit for any of the three. What I want is a way to determine how bad
>> > an attemped classification is, and react to it. I was thinking maybe a
>> > catch-all, "empty" category could handle this?
>>
>> Wouldn't it be nice? Unfortunately for you, a bayesian classifier (and
>> most other classification algorithms) require examples for every class
>> that it could possibly categorize. The classifier just chooses the most
>> like class from what it's been trained on. If you wanted a "none of the
>> above" class, then you'd need to provide examples from that none of the
>> above class. It's not so easy to decide what's representative of "none of
>> the above", and even if you could do so, it would probably violate the
>> assumptions of the classifier and lead to reduced performance. Thus, we
>> have to come up with more creative problem-specific solutions to handle
>> something resembling a "none-of-the-above" case, usually solutions that
>> change the definition of the problem quite dramatically.
> 
> Really a "none of the above" filter is of limited usefulness.

A "none of the above" filter can be quite useful. Supposing you have an
unknown text, and a sample of text from possible authors of the document,
and you want to know who wrote it, so you set up a classifier[1], train it
on the known authors and stick your text in. The answer will be one of the
authors who you trained the classifier for. Do you actually know that your
text was written by one of these guys? Maybe you don't. Then you need a
different problem: authorhship verification, which can be solved with
different techniques.

Authorship verification[2] is a completely different problem. You have
text by an unknown author, and text by a known author, and you ask "are
these texts written by the same author?" The technique for doing this 
abuses machine learning classifiers a bit, and as you can see it altered
the problem definition quite dramatically, but this is a
"none-of-the-above" capable version of the first problem.

--Ken Bloom
[1] Note that authorship classifiers use much more interesting features
than just word frequencies.
[2] http://www.cs.biu.ac.il/~koppel/papers/authorship-icml-formatted-01.04.pdf

-- 
Ken Bloom. PhD candidate. Linguistic Cognition Laboratory.
Department of Computer Science. Illinois Institute of Technology.
http://www.iit.edu/~kbloom1/