Re: Windows Compilation Madness

Some of this has been said, some of it hasn't. I'm also not 100% on  
everything here, but please discuss. This is poorly understood and  
needs documenting in order to really progress on the issue. Too many  
people I've heard saying "bah windows is too much of a pain" - but go  
onto rubyforge, and look at the OCI download numbers. Whatever you're  
feelings, that's far from irrelevant. The fact that only a handful of  
rubyists are actually capable of building ruby + all of ext/* is quite  
sickening, really (at least to me).

On 2 Jan 2008, at 13:02, Gary Wright wrote:

> No point in keeping this discussion in the Zed thread...

Absolutely. I have been thinking about raising this topic for a long  
time, but I'm also working on fixing some parts of it, so I was  
holding off until I have time to complete that work. I guess though,  
as it's here, now is as good a time as any. I've approached a few  
people for assistance on the ideas in the past, but we're all busy  
trying to survive ourselves to really get things going (yet). Even  
with a lack of time, if some people want to group together, I'd be  
more than happy to donate what I can to getting a more stable  
environment up on the Windows platform - and that includes code I've  
written against the issues.

> I could be wrong, but I sense a deeper question in Thurfir's message.
>
>  What is it about Linux (or *BSD or Mac OS X) that avoids all
>  the compilation problems that arise in the Windows environment?
>
> Can a Windows expert confirm my understanding that:

I am by no means an expert, but I have been studying this recently,  
and have come to some understanding of the issues. I'm not much of a C/ 
C++ programmer, but I have done a lot of building, and sent /  
committed patch files to build processes of quite a number of  
projects. I also actively use Windows, *nix (in many flavours), and  
more recently OS X.

> Unlink Linux (and *BSD, Mac OS X...), Windows does not come packaged  
> with a development environment and in fact there are multiple  
> mutually-incompatible development environments (Cygwin, MinGW, VC6,  
> VC8). For the most part you can not mix-and-match object code (dlls)  
> created by these different environments with the biggest problem  
> being that there is no common memory allocation library.

Well, actually OS X doesn't come "pre-packaged" either, but the build  
chain there is based on Open Source stuff from the *nix world, and for  
ruby, most of the pre-requisites are already available as libs  
(readline, curses, openssl, zlib, etc). Once you install Xcode, you  
pretty much get a production grade compiler setup for Ruby that will  
Just Work.

Under Windows, you can compile 1.8.6 trivially on any of the compilers  
mentioned. Ruby herself, in 1.8.6 is a very very clean compilation and  
Just Works. The ext/* stuff however, is a different story. OpenSSL  
compilation on Windows is actually a real pain - you need to get a  
copy of perl if you want to use the MS compilers (if you're using the  
pre-packaged build chain that is), and under MinGW you have to rebuild  
the directory structure by hand, as the symlinks in the tarball will  
not expand properly. I don't know why the OpenSSL devs ship their  
build packages like this, I find it terribly annoying, but it is what  
it is.

As for readline, well it's broken on win32, and several of the builds  
out there are archaic. GnuWin32 is useful - but again, stupid choices  
make life difficult, such as the pre-compiled binaries being hardcoded  
(!!!) to particular locations on disk. It's just insane - and this  
isn't the fault of the operating system - this is third party  
craziness. Other libs have similar issues, I won't got through the  
whole ext/* stack now. zlib is easier. Getting just rubygems + ruby up  
and running is quite easy, if you use one of the many pre-compiled  
openssl builds out there and build + link against that.

As for the MSVCRT issues, I have read many many mixed reports. The  
summary is roughly like this, AFAIK:
  - You *can* mix msvcrt versions safely *if* you don't pass structs /  
pointers across the boundaries for which the api has changed. In  
reality, for some applications, this is unworkable - however there are  
several production ready gems out there that are working just fine on  
the OCI (MSVCRT) which are compiled with VS 2005 (MSVCR7/8 (I can't  
remember the exact version -> VS pairings, sorry).
  - Due to the above, and several apps having issues with it - as well  
as the bloat it causes, no one likes mixing up MSVCR versions. That's  
fair enough.
  - I have been told by several MS developers (whether or not this is  
true, someone with more knowledge of the linking strategies and  
A[PB]Is will need to confirm - that it's possible to make these things  
safe using particularly compiled .lib files. I've also heard stories  
of shims being put in place, but they may come with a performance  
overhead, and are complex low level things in this domain (I suspect  
also quite specific to particular problems).
  - MinGW and MSVCRT (VS 6) are compatible.
  - Performance on MinGW is up about 30% over VS6, for 1.8.6. The gap  
is closing in 1.9, and isn't so bad for later MS compilers.
  - I haven't tried Intels compiler, and someone probably should,  
although it's buyware.
  - MinGW has cross-compiler capabilities, and that might be worth  
noting here. I know there are a couple of gems out there that are  
built like this. If possible, it might make sense to make something  
available for building pre-compiled gems like this. The problem is, C  
toolchains and dependancies are such a PITA very often.
  - In other build environments, I have built software that doesn't  
link against the standard runtimes locally at all. I need to become  
more familiar with the build tools and ABI to say more than that though.
  - There are bugs in the 1.8.6 ext/* makefiles which means that  
subshells spawned seem to use different relative / absolute paths in  
different ways. In order to compile everything off of a single 'make'  
under mingw, I found I had to add -I [absolute_path] AND -I  
[relative_path]. I think the same errors have thrown quite a few  
people who later just gave up. IIRC it's openssl vs. readline that  
don't compile properly - and those are a harder two anyway, so people  
get angry and give up very rapidly.
  - The version of bison in MinGW doesn't build parse.c properly  
anymore :-(

There is absolutely *nothing* stopping the ruby community getting into  
MinGW and helping everyone out by providing better build chains for  
them based on ruby apps and capabilities. Seriously, rubys core  
interpreter (1.8.6) builds really easily on top of the standard lib  
under mingw and other OSes, and isn't packed up as share/buy ware like  
the most commonly use perl / python builds on win32 - that's important  
to some people (whatever you might think of that as a merit yourself).  
More than this though, a dose of pragmatic powerful scripting  
capability could go a *long* way in helping MinGW move forward, so if  
anyone is interested, take a serious look at a project like this. Even  
just bootstrapping the default required mingwPORT.sh files would go a  
long way into making the MinGW build process simpler.

What actually needs "fixing" is we need someone to release a build  
chain for the build chains (for ruby herself). I'm working on  
something, but I have not had time to finish it, as I'm in a busy  
startup at present, so it's only good for internal use right now. I  
want to try and get into working with the rubygems folks once I'm more  
familiar with the chains and issues. Also interesting to join forces  
with might be the multiruby and related projects (there's a build-farm  
type project somewhere too but I can't remember the details, Ryan?)


> I assume this is why some Windows software comes distributed with  
> its own collection of 'standard' dlls compiled in the same  
> development environment as the underlying application.

Well, XP comes with two really, a native api, and the MSVCRT. As you  
install software, at some point you'll get an app with newer MSVCR  
versions being installed from the re-distributable. There are  
disgusting rumors about GPL violations and other complex licensing  
issues with linking against their newer libs - but this is just so  
totally anti-pragmatic. I found the Wireshark toolchain documentation  
very good - and something we should aspire too as a base minimum.

> The end result is that the least-common-denominator for all these  
> environments is source code, which still requires the 'end-user' to  
> install a development environment of some sort and to manage the  
> compile/link process for Ruby and for any and all 3rd party  
> libraries/gems/extensions that they need.

This is actually true of all the operating systems. It's very common  
to have it there already on *nix, and many developers install Xcode  
very early on in their configuration of a new OSX box, as this is  
required for fink / macports / general open source software  
compilation. This would be very similar on win32 if only mingw's build  
chain wasn't so nasty at this point in time.

> I also gather that each of those environments has very different  
> build utilities (the Windows equivalent of make, config, autoconf,  
> and so on). For an extension writer, the problem is that they can't  
> really know what the build process is going to look like on an  
> arbitrary Windows system making it very difficult to

Well, sure, but most of that actually works for a C app, at least,  
there are good example builds around that people can "borrow". Going  
into C++ is a different matter (there are more compiler and api  
differences there - i'm also less familiar though), but lets start  
with the most common first.