From: khaines@...
Date: 2007-03-08T05:41:59+09:00
Subject: Re: Horizontal scaling - advice needed

On Thu, 8 Mar 2007, Greg Lorriman wrote:

>
>> Drb would rapidly become a bottleneck there.
>
> I've not seen any articles on the characteristics and limitations of
> Drb but it isn't immediately apparent to me that it should be a
> bottleneck. Why do you think it would be a problem?

Sure.  Because while DRb is terribly cool and useful for a lot of things, 
it's not particularly fast.  Even if everything that you do with it is 
very lightweight, it is not hard for it to become a bottleneck when 
talking about heavy, large scale usage.  But, benchmark it yourself for 
the use case you envision and see if it'll move the volume of traffic that 
you would need it to.

> I would agree except that the application is simple while at the same
> time such federation is the ultimate in horizontal scaling AFAIK, and
> that is the exercise I would like to embark on. I want to find out the
> ins and outs, costs and difficulties.

Yeah, but you should be able to scale to a _very_ large amount of traffic 
before having to even consider adding in the complication of federating 
your data based on some criteria.

> Eventually the database will be the bottleneck, hence federation.

Again, though, you'll have to scale very, very large before you ever have 
to worry about the db being a bottleneck, provided you pay attention to 
schema design and proper db tuning.

> I'll keep that in mind. I hope you are right. But I've always had the
> idea that tcp/ip is a somewhat painful affair, and I suspect one would
> have to drop to that level to get adequate efficiency/speed to act as
> a router/redirector of incoming requests.

It's not that hard, even writing it in a language like Ruby.  Over the 
last two weeks I have implemented a very fast, purpose built proxy for 
clustering web apps.  It's not a big, general purpose tool like HAProxy. 
It's narrowly targetted, and for it's narrow purpose, it is comparable in 
speed to HAProxy (faster as of the last tests this weekend, but I am 
adding some code that will cut into that a bit).


Kirk Haines