From: Matthew Smillie <M.B.Smillie@...>
Date: 2005-11-09T21:09:28+09:00
Subject: Fwd: Question about symbols


Thanks for the replies, and if there are no objections to indulging  
me a little further...

On Nov 9, 2005, at 10:57, Robert Klemme wrote:

> Matthew Smillie wrote:
>> I wonder if I could trouble the list a bit further on this one: I've
>> got a collection of newswire articles, and I was thinking of using
>> symbols to represent the words in each.
>
> Also, this smells a bit like premature optimization.  Do you  
> actually need
> to have those words separately?  What exactly are you doing with your
> articles?

They're being tokenised, lemmatised/stemmed, tagged for part of  
speech, checked for named entities, and then I generate a dependency  
graph from the parse of each sentence.  I need to glue all that  
information to the relevant words, and then I spend a couple of years  
writing a dissertation on what comes out when I poke it with a sharp  
stick.

The reasoning behind using symbols was that it struck me as a sort of  
built-in flyweight pattern.  For example, if I have a couple of  
hundred instances of the word 'and', using :and struck me as a simple  
way to mash them into the same bit of memory without writing my own  
lookup table. e.g:

articles[x].sentences[y].add( Word.new(:"@# 
{current_word}", :conjunction, etc...) )

> RK:
> I would not use Symbols for that and I'd also not change the  
> application
> architecture (i.e. using fork) because forking essentially would  
> try to
> fix a problem introduced by using Symbols.

Using fork actually isn't a big change in the architecture, since  
these subsets are going to have to be spun off to separate machines  
for processing at some point, otherwise I'd never get through the  
entire document corpus.  I didn't plan to do it at this (prototyping)  
stage, but it's fairly trivially parallel, so the fork isn't likely  
to cause many problems.

This, on the other hand, might do:

> On Nov 9, 2005, at 10:37, daz wrote:
>
> I'll wave my hands (in a completely non-scientific way) at 300..500  
> max.
> (Table look-up isn't hot.)

Is it really that low?  I thought that since symbols are generated  
for every name in Ruby (aren't they?), the ceiling would be a little  
higher before they started being problematic.

thanks once again,
matthew smillie.