From: Jano Svitok <jan.svitok@...>
Date: 2007-10-13T01:33:18+09:00
Subject: Re: Text/Tag Cloud generation

On 10/12/07, Alex Young <alex@blackkettle.org> wrote:
> Jano Svitok wrote:
> > There can be
> > only so many words there.
> Actually, no.  The number of different words in a corpus goes up roughly
> as O(sqrt(n)).  That's Heap's law.

Ok ;-)  What I meant was that for given size of corpus (~10GB) the
hash (word->frequency) should be reasonable large to fit in memory. I
haven't known the Heap's law, thanks for the info. In some reference
on the web, they say for English Vr(n) ~ K * sqrt(n)  where 10 < K <
100. That means, for n = 3*10^9 Vr(n) < 5*10^6, and that's not large
(well... ;-)

Anyway, thanks for the pointer.

Jano