From: khaines@...
Date: 2007-03-08T03:08:54+09:00
Subject: Re: Horizontal scaling - advice needed

On Thu, 8 Mar 2007, Greg Loriman wrote:

> I'm putting together a relatively simple site which I want to design from
> the ground up for horizontal scalability, partly for the challenge, partly
> because I need to learn and get experience. To help me do this I am going to
> run at least two virtual machines to enforce the correct environment.
>
> Currently my idea is to federate the data so that the users are divided
> between the 2 or more machines, perhaps splitting alphabetically by user
> name (ie. A-G to machine 1, etc). Where there is interaction between account
> holders I am thinking of Drb'ing. Obviously rails is not going to be able to
> do the interaction side of things, but I am fine with that; I'm prepared for
> a bit of manual labour.

Drb would rapidly become a bottleneck there.  And explicitly federating 
your data like that seems needlessly complicated.

Put a fast proxy of some sort in front of your backend processes.  When 
you need more throughput, add another machine and some more processes.

> To facilitate the above I need some kind of proxy in front of the two
> machines directing incoming requests to the correct machine based on the
> login name which will be part of the url. Here I come unstuck since I have
> no idea how to do this.
>
> There must be proxies of this kind, but I'll be blowed if I know what an
> appropriate one would be, or where to start in making it do what I want.

If I were doing something that required some very specific proxying 
behavior that I couldn't get with an off-the-shelf solution (HAProxy is a 
very nice general purpose proxy with a ton of features), I'd write a 
purpose built proxy.  It's not really too hard to do.


Kirk Haines