Re: Optimizing large-scale Markov chain handler

-------- Original-Nachricht --------
Datum: Wed, 1 Aug 2007 23:42:10 +0900
Von: "M. Edward (Ed) Borasky" <[e-mail]>
An: [e-mail]
Betreff: Re: Optimizing large-scale Markov chain handler

> Axel Etzold wrote:
> > 
> > What you would need in order to generalize
> > LSA to higher orders is a generalization of the singular value
> > decomposition (SVD) to tensors. There are plenty of links, such
> > as this one:
> > 
> > http://www.cs.cornell.edu/cv/OtherPdf/William.pdf
> > 
> > to do that.
> > 
> > If you'd consider doing statistical tests to find longer
> > chains that occur significantly more often than they should
> > given the product of shorter chains, you can make use
> > of the following:
> 
> [snip]
> 
> > Mmmh, it is often argued that Ruby is slow in comparison to other
> > languages, when you consider the execution time of some pre-defined
> task. 
> > Entire webpages are dedicated to this claim.
> > But rarely ever do these comparisons take into account how much time you
> need to THINK and code. I've often found the execution
> > time of some task infinite with other languages, because they don't
> leave
> > you enough time to think about the problem, as you always  bother
> > about a million implementation details that obscure the real memory and
> > computing needs.
> > I now think that it is perfectly feasible to do a 6 or longer Markov
> chain handler  entirely in Ruby, and fast enough, whereas many brute-force
> > approaches with some "fast" language will take an infinite amount of
> > both space and time.
> 
> Well ... OK ... but ...
> 
> A couple of years back I got interested in chatbots for a brief period.
> Google for AIML and the Loebner Prize and you'll probably find some of
> my thoughts on the subject. Giuseppe IIRC wants to build a chatbot and
> is doing it with Markov chains. I argued, as you appear to be arguing,
> that the "right way" to build a chatbot involves statistical analysis of
> semantics, and the AIML people argued that their way was a lot simpler
> and worked just as well.
> 
> I conversed with a number of chatbots over a few months and attempted to
> train a couple, one using AIML and one using what appears to be the most
> credible technology, unfortunately proprietary. It was an exercise in
> frustration. I ended up concluding that statistical analysis of
> semantics was the right way to do it, but didn't see any payoff for
> investing the effort in trying to build a working code.
> 
> So two comments:
> 
> 1. Check out AIML -- they believe in it even if I don't, and I think
> there is a Ruby interface now. There is a Java interface, so perhaps
> jRuby can be the glue. http://www.alicebot.org/
> 
> 2. Check out the chatbot I though was the most credible, and which won
> the Loebner Prize a couple of times. http://www.jabberwacky.com/


Dear Ed,

I have had a look at both bots ... I can't say I'd find any one particularly
appealing as a substitute for a human conversation partner, e.g., to solve personal problems,
even though I've read reports about psychiatrists for whom 
even Eliza passed the Turing test in the Sixties already.

In the discussion with Giuseppe, I was mainly concerned with
reducing the amount of data in the n-chains he wants to use ...
as one first application of statistics.
Now that you ask, I wonder how much statistics there actually
is in the part of latent semantic analysis I find appealing ;-)

I like the fact that you merely do an SVD of the word-in-sentence
count matrix, then cut off after the first n eigenvalues ( giving
n concepts) and multiply everything up together.

One can interpret the resulting matrix as a linear mapping
and each sentence as a vector, with entries describing the words in it,
which gets mapped to some lower-dimensional subspace when
multiplied with the matrix.
This image subspace is a "subspace of concepts". In a general text 
about pets, sentences containing "dog" and "cat" will probably get mapped to
very close points in the subspace ... but you could introduce separating
hyper-planes to distinguish between anything doggish and cattish.

I like this way of constructing relatively few concepts each time anew from the actual 
context of the available text rather than trying to find > 40000 categories 
that will make up the whole world, as seems to be the case in ALICE and
similar bots. It might be that one day, I want to discuss cats as
mean predators eating poor mice, another day, as nice pets .
Then, a pre-configured categorization might have difficulties ...
and the number of categories is way too big in my taste.

Some neuroscientist said that humans can only deal with about seven
concepts at once:

http://www.musanim.com/miller1956/

In Ruby, there is the classifier gem by
Lucas Carlson and David Fayram II that will do LSA, text classification,
Bayesian analysis in the context of general texts.

Best regards,

Axel 
-- 
Psssst! Schon vom neuen GMX MultiMessenger geht?
Der kanns mit allen: http://www.gmx.net/de/go/multimessenger