From: "M. Edward (Ed) Borasky" <znmeb@...>
Date: 2007-08-01T23:42:10+09:00
Subject: Re: Optimizing large-scale Markov chain handler

Axel Etzold wrote:
> 
> What you would need in order to generalize
> LSA to higher orders is a generalization of the singular value
> decomposition (SVD) to tensors. There are plenty of links, such
> as this one:
> 
> http://www.cs.cornell.edu/cv/OtherPdf/William.pdf
> 
> to do that.
> 
> If you'd consider doing statistical tests to find longer
> chains that occur significantly more often than they should
> given the product of shorter chains, you can make use
> of the following:

[snip]

> Mmmh, it is often argued that Ruby is slow in comparison to other
> languages, when you consider the execution time of some pre-defined task. 
> Entire webpages are dedicated to this claim.
> But rarely ever do these comparisons take into account how much time you need to THINK and code. I've often found the execution
> time of some task infinite with other languages, because they don't leave
> you enough time to think about the problem, as you always  bother
> about a million implementation details that obscure the real memory and
> computing needs.
> I now think that it is perfectly feasible to do a 6 or longer Markov chain handler  entirely in Ruby, and fast enough, whereas many brute-force
> approaches with some "fast" language will take an infinite amount of
> both space and time.

Well ... OK ... but ...

A couple of years back I got interested in chatbots for a brief period.
Google for AIML and the Loebner Prize and you'll probably find some of
my thoughts on the subject. Giuseppe IIRC wants to build a chatbot and
is doing it with Markov chains. I argued, as you appear to be arguing,
that the "right way" to build a chatbot involves statistical analysis of
semantics, and the AIML people argued that their way was a lot simpler
and worked just as well.

I conversed with a number of chatbots over a few months and attempted to
train a couple, one using AIML and one using what appears to be the most
credible technology, unfortunately proprietary. It was an exercise in
frustration. I ended up concluding that statistical analysis of
semantics was the right way to do it, but didn't see any payoff for
investing the effort in trying to build a working code.

So two comments:

1. Check out AIML -- they believe in it even if I don't, and I think
there is a Ruby interface now. There is a Java interface, so perhaps
jRuby can be the glue. http://www.alicebot.org/

2. Check out the chatbot I though was the most credible, and which won
the Loebner Prize a couple of times. http://www.jabberwacky.com/