Re: [OT] RubyGoLightly Progress Report

An eminently entertaining post! I think there's even a few areas
you've added since we had a chance to talk last!

Seriously though...I think I'm starting to understand what you really
want from Ruby: real systems-level programming without a lot of weird
dynlang/VM encumbrances and a consistent but transparent abstraction
of platform-specifics. C with fewer "types", higher-level APIs, and no
requirement for explicit memory-management. I'm in :)

On Fri, Jul 2, 2010 at 5:57 PM, Eleanor McHugh
<[e-mail]> wrote:
> I seem to have been missing in action for the best part of six months so =
I thought it was about time I put together a progress report and pimped the=
 vapourware in hopes of finding some likeminded miscreants/dilettantes to t=
urn it into something more substantial.
>
> There're probably only a handful of people on here who'll be interested i=
n what follows so I've marked it OT for those of you interested exclusively=
 in Ruby. Feel free to press delete, I won't be offended :)
>
>
> =3D=3D BACKGROUND =3D=3D
>
> For those of you without long memories, a quick recap. At the tail-end of=
 last year Google released their Go language (http://www.golang.org) onto a=
n unsuspecting world. The brainchild of Rob Pike and Ken Thompson, Go's sor=
t of a reworking of Limbo from Plan 9/Inferno with baked-in concurrency and=
 an inside-out approach to object orientation.
>
> As those of you who've had the dubious pleasure of either sharing bar spa=
ce with me or sitting through one of my conference sessions on Ruby & Unix,=
 for all that Ruby's my language of choice I still spend a fair amount of t=
ime grubbing around in the C world. Love them or loathe them, there are tim=
es when we all need systems-level languages to efficiently solve certain ki=
nds of problems.
>
> The longterm goal of RubyGoLightly is to make Ruby a first-class citizen =
in that sense, moving systems coding firmly and permanently into our high-l=
evel world. I know it can be done as Ruby's pretty much as flexible as Lisp=
 and Symbolics proved that an OS could be written in Lisp with minimal depe=
ndence on assembly language. So the aspiration isn't so lousy even if I'm f=
ar from certain that I'm personally capable of achieving it.
>
> There is however one major roadblock to turning Ruby into a systems langu=
age: existing runtimes just aren't written that way. The closest we have at=
 present is JRuby, and even there if you want complete control you still ha=
ve to step outside of the Ruby mindset and hack Java, JVM bytecode or proba=
bly even write JNI extensions in C.
>
> Then there's the question of an optimal Ruby runtime. There's a lot of im=
plicit concurrency in the average Ruby program which to the best of my know=
ledge isn't being addressed by any of the existing implementations. We live=
 in a multicore world where it's only natural that collection enumerators e=
tc. should be exploiting those cores for cheap scalability. Admittedly writ=
ing concurrent systems in C can be a right pain in the arse, and even in Ja=
va there are subtleties arising from the architecture of the JVM so I'm not=
 levelling criticism at anyone. It's just a fair observation that if I have=
 four processor cores and a list with 10K entries that map/reduce algorithm=
s should use all four cores if at all possible and return their results in =
approximately 1/3rd the time of a naive sequential implementation.
>
> Given that Intel are playing with 80+ core processors in the lab - and th=
at modern graphics cores have even more available horsepower via OpenCL - t=
here's clearly an opportunity worth investigating.
>
> Go does concurrency pretty well, especially considering that the runtime =
is still in its infancy and will probably undergo considerable optimisation=
 in the future. It's basically an implementation of Hoare's CSP so all the =
tricks that can be used in Occam etc. for clean concurrency can be used in =
Go, and whilst it doesn't yet support OpenCL that's just a matter of time a=
nd a sufficiently nasty itch being worth scratching.
>
> Another desirable trait for a Ruby runtime is decent browser integration.=
 I get really annoyed having to write heavyweight client-side web code in J=
avaScript as to me it's an ugly language. I've probably never recovered fro=
m my painful experiences with NewtonScript as I find the entire Self langua=
ge family unbearable, and the same goes for the slot-based GTK GUI stuff (s=
orry webGTK). Go has support for Google's Native Client so is ideal for inj=
ecting native code into web content, including a Ruby runtime were it to ex=
ist. I love that idea. Write your whole web app in Ruby, with identical abs=
tractions.
>
> The same thing could be achieved using a Ruby runtime written in JavaScri=
pt (such as HotRuby) but to date that approach has stalled. Basically writi=
ng any significant runtime platform is a significant cost and taking anythi=
ng beyond the complexity of a simple bytecode interpreter means months of f=
ull-time hacking and serious cogitation.
>
> The final thing in Go's favour is that it can compile to native code for =
bare metal - i.e. no OS at all, just a bootloader and an executable. I've y=
et to play with that side of the language, but having written assembler for=
 embedded systems with no OS I would love to see a Ruby runtime deployed in=
 the same manner. From there it's a short step to an OS written in Ruby, an=
d that means an OS any half-competent programmer could understand in detail=
.
>
> My colleague Romek and I have been boring people with that idea for about=
 five years now, and mad though it probably seems I still believe it's a go=
al worth pursuing.
>
> Anyway, to recap: the world might be a better place with a Ruby runtime t=
hat embraced pervasive concurrency in the runtime implementation, that was =
platform agnostic, and that allowed systems-level coding to be performed wi=
th the same ease we associate with Rails or even Sinatra. I won't pretend R=
ubyGoLightly will tick all these boxes - right now it's an idea and ideas a=
re by their nature transient. However it is certainly a framework in which =
to explore whether such a Ruby is possible.
>
>
> =3D=3D GoLightly =3D=3D
>
> There are two aspects to this project. One is the Ruby runtime, which rig=
ht now I have only the vaguest of ideas about how to implement. I'm followi=
ng a hunch that a Forth-like Threaded Interpreter might allow optimisations=
 that we're not going to see with mainstream bytecode interpreters and a lo=
t of current effort is devoted to writing such a Forth implementation. Alth=
ough I hesitate to call it Forth given the usual expectations of F79, F83, =
figForth or ANS94 compliance - none of them particular of particular import=
ance.
>
> Threaded interpretation isn't a million miles removed from the naive view=
 of Ruby's object model so I'm hoping my current work will generate insight=
s relevant to that, however its main purpose is to drive design of the GoLi=
ghtly virtual machine which is an abstraction of the kind of hardware I'm i=
nterested in seeing Ruby run on.
>
> There's an early branch of GoLightly on github (http://github.com/feyelea=
nor/GoLightly) which exposes some basic despatch loops (inlined bytecode an=
d indirect function call) and a preliminary model for SIMD-style opcodes fo=
r basic vector processing and a supporting register model.
>
> There are a few examples of hand-coded assembler in the tests to give a f=
eel for the machine language along with some micro-benchmarks for anyone wh=
o fancies playing. GoLightly already copes with multiple virtual processors=
, all working on their discrete problems but there's a lot of work yet to b=
e done in inter-core communication, the memory allocation model, and parall=
elising SIMD instructions.
>
> One of the few things I dislike about Go as it currently stands - and whi=
ch alas is considered a strength by the core team so is unlikely to change =
- is that the language doesn't support a representation-agnostic datatype s=
uitable for holding bit-strings: basically numeric types always convert the=
ir contents to suit the type to which they're being coerced, or else a pani=
c occurs. Getting around this involves playing games with pointers that mak=
e a fair amount of (void *) usage in C look clear and reasonable. The justi=
fication is that this sort of thing shouldn't be encouraged (with which I h=
appen to disagree anyway) and that the extra pain is a bit of nudge to prev=
ent it. I'll keep raising the issue with the core team every few weeks as I=
 personally believe maintainability is much more important than type safety=
: after all, we can always write tests...
>
> Anyway GoLightly is stalled whilst I evolve a reasonable API for handling=
 bitstring manipulations, which is proving a subtly intractable problem. I'=
m hoping that hacking together a few languages will help me figure out most=
 of the implementation details which are currently eluding me and then I'll=
 roll a bitstring library back into the Go core library if the team will ha=
ve it.
>
> The unnamed Forth I'm currently working on is running into the same probl=
ems, but I'm still hoping to get a reasonable build ready for release on gi=
thub over the summer. So far I have a good first cut of a compile/execute/d=
ecompile model which supports dynamic coding and I hope to reduce much of t=
he Go code for implementing a core vocabulary to GoLightly-style bytecode. =
These bytecodes (which are effectively unique to the currently running syst=
em, so more a dynamic object code than a formal bytecode) would themselves =
be a first-class element of the language so that's an interesting model for=
 both a native code interface and for allowing inline use of multiple langu=
ages.
>
>
> =3D=3D The Roadmap =3D=3D
>
> From Forth to Ruby is one hell of a leap, and whilst I don't doubt it can=
 be done in one step there are other problems I want to solve along the way=
.
>
> I definitely see a simple Lisp being in the mix as I've always felt there=
 is an equivalence between Forth and Lisp which goes unremarked: I guess th=
ere aren't that many people who've hacked with both, let alone who've writt=
en runtimes for both. Lisp will be the testbed for doing standard function =
call stack allocations and a few other aspects of implementing a high-level=
 language which Forth skirts. Common Lisp it most definitely won't be thoug=
h (or Arc for that matter).
>
> From there I want to get a Scheme working, mostly to figure out how to ha=
ndle tail-call recursion efficiently. It could be that this will be a speci=
fic compiler optimisation required separately for each supported language, =
but I'd really prefer it to work at the object code level so that it's appl=
ied across all compiled code automatically. Go uses relatively small stacks=
 to improve concurrent performance compared to pThreads and whilst GoLightl=
y threads are currently specced to use their own user-space threads I'd lik=
e to move beyond that in the future so tail call culling will be essential =
to not swamping the runtime stack.
>
> After Scheme I'm hoping to implement a simple Logo system to get a handle=
 on the interfaces offered by Google's Native Client etc. - it's a bit of a=
 gratuitous diversion as Logo is hardly a mainstream language, but the unde=
rlying structure is similar enough to Lisp that I think it will be an easy =
build and there's been some interesting work in Logo-land to do with multi-=
threaded graphics which might have interesting tie-ins with OpenCL.
>
> Much of the impetus to start hacking on RubyGoLightly is thanks to TinyRb=
, a really sweet subset of Ruby running on a Lua-style VM. Once I've solved=
 the main runtime problems addressed by the Lisp arc of languages I plan to=
 write a runtime interpreter for the TinyRb bytecode so its compiled form c=
an run natively in GoLightly. My gut instinct is that this will be a great =
starting point for runtime optimisations that apply to a richer subset of R=
uby.
>
> At LSRC last year I showed off some of the code from a port of SQLite to =
Ruby that I was working on at the time. That project got put on hold when I=
 started RubyGoLightly but given that SQLite is a really sweet engine and i=
nternally uses a register based VM it'd be easy to port to GoLightly and pr=
ovide something like CoreData out of the box.
>
> There could be a few steps after this: for one thing I've this gut feelin=
g that doing a dirty implementation of Io could be instructive, though I ca=
n't really explain way. Something to do with the method passing semantics. =
A roadmap's no good if it doesn't allow for interesting detours :)
>
> However the ultimate destination is RubyGoLightly, the Ruby runtime that =
scratches the itch of using Ruby as a systems language. No promises about w=
here or when, and very few about how... but I promise to check in with anot=
her status report in a few months that will hopefully be more insightful.
>
> In the meantime, happy hacking,
>
>
> Ellie
>
> Eleanor McHugh
> Games With Brains
> http://feyeleanor.tel
> ----
> raise ArgumentError unless @reality.responds_to? :reason
>
>
>
>