From: ben@...
Date: 2006-09-25T18:20:13+09:00
Subject: Re: Ruby, Analysis, and Tons of RAM

Thank you all for the excellent suggestions and links.

More about the problem domain: Linguistic modeling with lots of
posterior (Bayesian) inference maths. The probability matrices involved
can easily grow into the multiple GB's range, and obviously I'm
completely hosed if I keep things disk-based. (I've tried. Even with
hip and sexy paging.) The process is only somewhat parallelizable, but
I've gotten a nasty hit in the past from the IPC's. (This IPC hit was
probably my fault, damn "3.14159".to_f.)

Obviously, I'm doing the real maths with C routines called from Ruby.

I'm quite happy to hack around and compile everything from scratch. On
the other hand, I'm not happy with expensive support agreements or
using "traditional techniques" (FORTRAN, shudder). So maybe it's
reasonable to consider me 1) short on cash, 2) short on processing
time, 3) long on Linux admin skillZ, 4) long-ish on coding time.

Thanks again, folks.


M. Edward (Ed) Borasky wrote:
> ben@somethingmodern.com wrote:
> > Does anyone have experience with using Ruby for analysis (*lots* of
> > maths), on a machine with a ridiculous amount of RAM? For example, a
> > hip 64-bit Linux kernel on a machine with 32 or 64 GB of physical RAM.
> >
> > Are there any "gotchas" I should be aware of? Would all the RAM be
> > addressable by a given Ruby process? Or would I still have to be
> > forking a number of processes, each allocated a bit of the address
> > space (blech)?
> >
> > Thanks, oh Ruby masters.
>
> 1. Again, you can contact me off list for some ideas ... without knowing
> your goal, it's difficult for me to know what steps you should take to
> reach it.
>
> 2. Assume a properly working state-of-the-art 64-bit dual-core AMD or
> Intel hardware platform with 64 GB of RAM and an appropriate SAN for
> storage from a major vendor like IBM. That severely limits your OS
> choices; last time I looked you need to be running either RHEL or SUSE
> Enterprise Linux. I don't know about the other vendors, but IBM has a
> marvelous document on performance tuning humongous servers at
>
> http://www.redbooks.ibm.com/redbooks/pdfs/sg245287.pdf
>
> 3. OK, now you've purchased a high-end server and a *supported*
> enterprise-grade Linux, and you want to do some serious number crunching
> on it, and you want to do it in Ruby, possibly augmented by libraries in
> C, Fortran or assembler for speed. You will need to recompile
> *everything* -- Ruby, the math libraries, and the compiler itself -- to
> use 64-bit addressing. There are some hacks and workarounds, but pretty
> much this is required. If you end up with an Intel server, you might
> want to have a look at the Intel compilers instead of GCC. Intel also
> has some highly-tuned math libraries, as does AMD.
>
> My point here is that you are "exploring" realms in Ruby that are
> "usually" addressed using "more traditional" techniques, so you're going
> to need to do a fair amount of testing. That kind of server costs a lot
> of money, and for that kind of money, you'll get lots of support from
> the vendor, coupled with strong incentives to do your job in ways that
> are tested and proven to work and supported by said vendor.That may or
> may not include Ruby, and if it does include Ruby, it may or may not
> involve a small number of monolithic Ruby scripts directly addressing a
> large address space.
>
> There is a lot of help available on the Internet from people like me who
> love challenges like this. :)