From: Robert Klemme <bob.news@...>
Date: 2005-11-09T21:36:21+09:00
Subject: Re: Question about symbols

Matthew Smillie wrote:
> Thanks for the replies, and if there are no objections to indulging
> me a little further...
>
> On Nov 9, 2005, at 10:57, Robert Klemme wrote:
>
>> Matthew Smillie wrote:
>>> I wonder if I could trouble the list a bit further on this one: I've
>>> got a collection of newswire articles, and I was thinking of using
>>> symbols to represent the words in each.
>>
>> Also, this smells a bit like premature optimization.  Do you
>> actually need
>> to have those words separately?  What exactly are you doing with your
>> articles?
>
> They're being tokenised, lemmatised/stemmed, tagged for part of
> speech, checked for named entities, and then I generate a dependency
> graph from the parse of each sentence.  I need to glue all that
> information to the relevant words, and then I spend a couple of years
> writing a dissertation on what comes out when I poke it with a sharp
> stick.

Sounds interesting.

> The reasoning behind using symbols was that it struck me as a sort of
> built-in flyweight pattern.  For example, if I have a couple of
> hundred instances of the word 'and', using :and struck me as a simple
> way to mash them into the same bit of memory without writing my own
> lookup table. e.g:
>
> articles[x].sentences[y].add( Word.new(:"@#
> {current_word}", :conjunction, etc...) )

I guess in this case (i.e. with high volume expected) it makes sense to
build a more complex solution.  You can even start with a simple class
implementation and make it more intelligent as you go.  Off the top of my
head, some options

1.
Use a hash where keys and values are identical

h = Hash.new {|h,s| s.freeze; h[s]=s}
def make_word(s) h[s] end

Pro: fast, easy
Con: need to somehow remove unused stuff

2.
Similar but using WeakReference for easy GC
Pro: less mem consumtion

3.
Build a trie like structure where each string is stored quite efficiently.
A trie is basically a tree like data structure where each node represents
a char.  Then you just need to store nodes of this structure.

Pro: That way you might even save so much mem, that you can forget cleanup

4.
Do 3 in a C extension where you can save even more mem.

>> RK:
>> I would not use Symbols for that and I'd also not change the
>> application
>> architecture (i.e. using fork) because forking essentially would
>> try to
>> fix a problem introduced by using Symbols.
>
> Using fork actually isn't a big change in the architecture, since
> these subsets are going to have to be spun off to separate machines
> for processing at some point, otherwise I'd never get through the
> entire document corpus.  I didn't plan to do it at this (prototyping)
> stage, but it's fairly trivially parallel, so the fork isn't likely
> to cause many problems.

DRB may help here, too.

> This, on the other hand, might do:
>
>> On Nov 9, 2005, at 10:37, daz wrote:
>>
>> I'll wave my hands (in a completely non-scientific way) at 300..500
>> max.
>> (Table look-up isn't hot.)
>
> Is it really that low?  I thought that since symbols are generated
> for every name in Ruby (aren't they?), the ceiling would be a little
> higher before they started being problematic.

No idea.  You might obtain better info by trying it out and / or looking
at the source.

Kind regards

    robert