From: Charles Hixson Date: 2004-09-20T15:18:07+09:00 Subject: Re: Method improvement request .-- Bill Guindon wrote: >On Mon, 20 Sep 2004 07:24:29 +0900, Charles Hixson > wrote: > > > >>Bill Guindon wrote: >> >> >... > >>>ght be easier if you could show a sample of what you're starting >>>with, and what you'd like it to become. If the sample is large, you >>>can always paste it into codepaste (http://www.codepaste.org), and >>>just put the link to it in the email >>> >>> >... > >>OK, I've copied it over to codepaste here: >>http://www.codepaste.org/paste/comment/218 >>http://www.codepaste.org/view/paste/163?show_comments=1 >>etc. >>(How long does code stay up here? I never knew the site existed.) >> >> > >New site, written in Ruby. For now, there's no time limit. > >I was bored with my own stuff, so trimmed down the 'run' a bit... >http://www.codepaste.org/view/paste/163?show_comments=1 > >Hope it still runs, couldn't test it for lack of the 'word' file you required. >There are links on the right to see the originals, and a diff between the two. > > Actually, after several comments, and thinking about things, I've started trying to use rewrite rules. Thinks like replacing /(\w)n't/ with '\1n\xB9t'. This means that I need to reserve the first 255 word Id's for character names, but by replacing internal apostrophe's with a Right Single Quotation Mark I don't need to worry about matches with ' (since the replacement character isn't defined in std. ANSI-7. It does mean that the elipsis (...) had to be translated to /x91 (Private Use 1) since no elipsis character was defined, but that's OK. But with these translations, a lot of the special cases disappear, and I was going to need to do that anyway as there's no other way to determine things like whether or not the period of Dr. denotes the end of a sentence. I'm still dithering about what the period should be rewritten as, but making it something other than a period simplifies things alot, at the cost of misidentifying cases where a sentence ends with Mr. >>But I think that I put the relevant pieces in the first e-mail. >>However, I don't intend that elipsis and double-dashes be deleted. >>They merely need to be parsed separately from the words that they appear >>with. They do contain significant meaning, so merely deleting them >>would be anti-productive. >> >> > >Right, much easier to understand what you seek now. Sounds like >you're looking at a text formatter, I'm guessing for printing. > > Actually, I'm reading text files, assigning word id's to various word & wordish fragments, sticking them in a database for memory. The next step is to build real phrases (That word routine, and it's backfill is just the part that builds the database, sticks the words into it, and retrieves them when a previously encountered word is again encountered.) I'm not yet to the interesting stuff. Right now I'm just trying to create phrases of a reasonable length that break in gramatically reasonable places. I'd prefer that no phrase be longer than 5 or 6 words. At a later stage it will start babbling using reasonable phrases as chunks, and transitionsing from phrase to phrase based on some kind of statistical relationship. Still later...well, I don't yet know just how far this can go. I'm hoping it will become interesting. I intend to feed it a bunch of books from Gutenberg as background, but I'm starting with Alice30.txt (Alice in Wonderland). > ... > >With a bit of time, the 'nil' bugs will go away, or at least have >semi-obvious solutions. >