From: Tom Copeland <tom@...>
Date: 2006-10-13T00:13:48+09:00
Subject: Re: rubyforge.org down

> my experience is that each percentage point of uptime 
> increase adds exponential effort for the maintainers.  
> clustered setups are insanely complicated with any 
> application that has state - which rubyforge does. people 
> think it's straightforward, but i can't tell you how many 
> boxes we've seen that were 'highly-available' which ignored 
> situations like split-brain, failback, etc. and therefore 
> merely threw money and time at a problem without increasing 
> reliability...  lazy replication and all the quick fixes 
> cause more problems than they solve - to do it right one 
> needs good shared storage with serious hardware redundancy 
> and remote controlable stonith devices for prevent split 
> brain.  a modest setup would start at 50k depending on 
> storage needs. some of our systems are more like 300k - and 
> that's for one pair.
> 
> anyhow, i think a more reasonable approach is
> 
>    a) minimize downtime.  rubyforge is already there, i'm 
> sure it's uptime is
>    above 98%
> 
>    b) run hot-warm with manual failover and failback.  this 
> is cheap and a
>    helps 'a' greatly.
> 
>    c) drb (distributed block device) could drastically reduce cost and
>    eliminate the need for shared storage - but i have no 
> experience with it...

Yup, and since there's a DB involved, cutting over to a backup system
would involve losing some data, blah.  I guess we could do replication,
but what about commits to cvs/svn... there are a lot of issues.   And
yup, setting up a SAN or something like that would get spendy.

Yours,

Tom