From: Eric Hodel <drbrain@...7.net>
Date: 2003-02-20T10:22:56+09:00
Subject: Re: optimization question


--ey/N+yb7u/X9mFhi
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Daniel Carrera (dcarrera@math.umd.edu) wrote:

> On Thu, Feb 20, 2003 at 07:53:14AM +0900, Travis Whitton wrote:
>=20
> > # This is ran about a million times
> > h =3D Hash.new(0)
> > data.scan(iptok).each do |tok|
> >   h[tok] +=3D 1
> > end
>=20
> 1) Are the tokens strings?
> 2) Are you running Linux?
>=20
> You might be able to use glib hashes to do this in C and then translate=
=20
> those to Ruby hashes.  Just a thought.  I'm putting together some code to=
=20
> see if I can figure it out.

(I'm making a big assumption here, so if I'm wrong, please kindly
disregard)

Since I've lost the original email, I'll just say that I there's a
more efficient way to do what you're doing.  Keep a real database of
tokens, and use mail to update it, rather than re-tokenizing the whole
email set.

See: http://glueless.net/bayes/ (not by me)

My .rbayes-corpus.db is 10MB, and I started using it in Sept.

--=20
Eric Hodel - drbrain@segment7.net - http://segment7.net
All messages signed with fingerprint:
FEC2 57F1 D465 EB15 5D6E  7C11 332A 551C 796C 9F04


--ey/N+yb7u/X9mFhi
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (FreeBSD)

iD8DBQE+VC3tMypVHHlsnwQRAryuAJsGHlJYioHcXUy5yQ0Pdg55VCCOBQCeJ3YE
DD+lNWROzT5aHWg47htWzpk=
=P8Lh
-----END PGP SIGNATURE-----

--ey/N+yb7u/X9mFhi--