Re: file reading impossibly slow? 1.6.6 vs. 1.6.2 numbers

On Fri, 8 Mar 2002, Pete McBreen wrote:
>
> I missed the first two, but having recently installed the 1.6.6 version I
> was amazed at the difference. I don't think it is something that 1.7 ot
> 1.8 will solve, unless they are using completely different file system
> calls. I think it is likely to have something to do with the way 1.6.6 is
> built in Visual C++ and/or the fact that we have to use different file
> system calls to get good performance under mswin (if this is the case then
> cygwin version must be doing somethin under the covers to invoke different
> file system calls).
>

I think I described one of the major reasons why the native Windows
version of Ruby 1.6.5 was slow in my mail from 6 Dec [ruby-talk:27728]
(I include a copy of that mail below).

It was just very bad coding, making unnecessary system-calls for
*each* character of a file. No wonder it was slow.

(I believe the reason was the thread-support, that was very naively
implemented. Other people can describe this better ....)

As already mentioned, 1.7 did fix this.

I *relly* hope that a non-development-Ruby-version will be made soon
that fixes this problem.

/Johan Holmberg

-------------------------------------------------------------------
Subject: File IO very slow on Windows ("i586-mswin32")
Date: Thu, 06 Dec 2001 18:50:24 GMT

Ruby built natively for Win32 is very slow when reading large files.
I use 1.6.5 built on "i586-mswin32". Recently I had a little program
essentially containing just a loop like

    while line = gets
        ...
    end

When I ran this script on a file looking like:

    $ wc foo.log
    19130  692976 16813844    foo.log

it took about 90 seconds. This seemed very strange. So I recoded the
script in Perl. Result: 2 seconds. Quite a difference ...

I also tried the two scripts on Solaris, to see if the problem was
Win32-specific. Result:

            Perl:  2 seconds
            Ruby: 12 seconds

Not so good for Ruby, but not as bad as on NT.
Finally I tried to look in the Ruby code, to see where the time is spent.
After a while I found a section in the function 'rb_io_gets'. I tried to
comment out a couple of lines:

    ------------------------------------------------------------
    *** io.c~ Wed Sep  5 07:53:28 2001
    --- io.c Thu Dec  6 18:24:18 2001
    ***************
    *** 746,755 ****
    --- 746,759 ----
	again:
	  bp = buf;
	  for (;;) {
	    READ_CHECK(f);
    + #ifdef KOKO
	    TRAP_BEG;
    + #endif
	    c = getc(f);
    + #ifdef KOKO
	    TRAP_END;
    + #endif
	    if (c == EOF) {
		if (ferror(f)) {
		    if (errno == EINTR) continue;
    ------------------------------------------------------------

After this change, the script took 4 seconds (instead of 90 as before).
TRAP_BEG implies that for example the Win32-function
'InterlockedExchange' is called *once for each character* as far as
I can see. This seems totally  insane ...

Is this an already known fact ?
Is anything being done to fix it ?

Recently there has been talk of using Visual C++ (or another "native"
compiler) for the Win32 version of Ruby. I suppose things like this
has to be dealt with before  Ruby can be used seriously on Win32
(and I very much want that to happen).

Regards,

Johan Holmberg

---------------------------------------------------------------