From: Shashank Date <shanko_date@...>
Date: 2005-01-23T07:11:46+09:00
Subject: Re: [ANN] Lafcadio 0.7.0, 0.6.1: Excessively Clever Query Caching

Hi Francis,

> Intriguing stuff. Once you've set this up in MS SQL
> 2k, what requirements are there for a client to 
> manage them? I mean, besides what the database 
> takes care of for you automatically.

Umm .... mantaining the indexes comes to mind. I don't
know the details since we never actually used it as it
comes out of the box. We found out that the queries
were not being executed in parallel. Hence we wrote
our own version (in Ruby of course) and called it
"parallel query dispatcher" :-)

> And by the way, if you're working with federated
> databases, how big are 
> these tables you're dealing with? I'm just wondering
> how much bigger 
> the tables at my work can get before I need to look
> into something like 
> this.

It is not only the size that matters (in this case
;-)) but the nature of the application. Our data is
being collected at various data centers and then
coalesced at the central server. So it comes naturally
partitioned. Further our queries are rarely (almost
never) across the partitions. This is a very important
aspect which lends itself to federation.
Add to that the fact that our combined database is
about 100GB and tables are typically over 5 Million
rows. So when we did not have the budget to scale up
we decided to scale out and were reasonably
successful. We were in production for almost a year on
four 3-server clusters throwing hundreds of  queries
every day. We did dynamic load balancing and were
working on query caching (like the one you have
provided in Lafcadio) when the project got the
attention of higher-ups and a more generous budget to
scale up ... which almost always is a better
alternative.
 
> Well, I'll put "port to MS SQL" on my to-do list and
> let you know when 
> a new beta release has MS SQL support ... then I
> just need a steady supply of specific bug reports 
> to chase down, after that.

Great ! Let me know ...
 
> Francis Hwang
> http://fhwang.net/

-- shanko


		
__________________________________ 
Do you Yahoo!? 
Yahoo! Mail - 250MB free storage. Do more. Manage less. 
http://info.mail.yahoo.com/mail_250