[ruby-core:40754] [ruby-trunk - Feature #5562] Improvement of Windows IO performance

Issue #5562 has been updated by Jon Forums.


I am seeing a strange issue with the patch on my Win7 32bit system. I'm not yet sure whether it's my setup or a real issue.

In summary, although the patch applies cleanly (with offsets) and tests pass, at runtime I get `This program cannot be run in DOS mode` failures for `ruby.exe`. It appears `ruby.exe`, `msvcrt-ruby191.dll`, and others are being corrupted and not recognized as PE files. This occurs regardless of whether I build with RubyInstaller/MinGW or Windows SDK. Strangly, when I cross-compile on Arch and MinGW the binaries appear to be correctly built and execute on Win7.

Here's what I know so far:

* failure occurs when building with Win7 on either ruby_1_9_3@33569 or trunk@33634
* failure occurs using TDM-GCC 4.5.2/4.6.1 and Windows SDK for Windows 7
* no failure when cross-compiled with Arch VM (VirtualBox) with `i486-mingw32-gcc` 4.6.1
* `miniruby.exe` is not corrupted as both `make test` and `make test-all` complete
* good `msvcrt-ruby191.dll` is 2,138,112 bytes, good `ruby.exe` is 73,728 bytes
* bad `msvcrt-ruby191.dll` is 4,096 bytes, bad `ruby.exe` is 4,096 bytes

C:\Users\Jon\Documents\RubyDev\ruby-git>git apply --verbose win_io.patch
# many messages about hunks applied with offsets
Applied patch include/ruby/encoding.h cleanly.
Applied patch io.c cleanly.
Applied patch win32/win32.c cleanly.


C:\Users\Jon\Documents\RubyDev\ruby-git>git st
## trunk
 M include/ruby/encoding.h
 M io.c
 M win32/win32.c


### FAILURE - RubyInstaller build recipes using TDM GCC 4.5.2/4.6.1 ###

C:\projects\rubyinstaller-git>rake ruby19 local=C:\Users\Jon\Documents\RubyDev\ruby-git nogems=1
...
mkdir -p sandbox/rubygems
mkdir -p sandbox/ruby19_mingw/lib/ruby/1.9.1/rubygems/defaults
cp resources/rubygems/operating_system.rb sandbox/ruby19_mingw/lib/ruby/1.9.1/rubygems/defaults
mkdir -p sandbox/rb-readline
"C:/projects/rubyinstaller-git/sandbox/extract_utils/7za.exe" x -y -o"sandbox/rb-readline" "C:/projects/rubyinstaller-git/downloads/rb-readline-0.4.2.zip" > NUL 2>&1
cd sandbox/rb-readline
ruby setup.rb
This program cannot be run in DOS mode.
rake aborted!
Command failed with status (1): [ruby setup.rb ...]

Tasks: TOP => ruby19 => dependencies:rbreadline:install19


C:\projects\rubyinstaller-git>rake devkit:sh
Temporarily enhancing PATH to include DevKit...
sh-3.1$ export PATH=/c/projects/rubyinstaller-git/sandbox/ruby19_mingw/bin:$PATH
sh-3.1$ ruby --version
This program cannot be run in DOS mode.
sh-3.1$ objdump -h bin/ruby.exe
C:\projects\rubyinstaller-git\sandbox\devkit\mingw\bin\objdump.exe: bin/ruby.exe: File format not recognized
sh-3.1$ objdump -h bin/msvcrt-ruby191.dll
C:\projects\rubyinstaller-git\sandbox\devkit\mingw\bin\objdump.exe: bin/msvcrt-ruby191.dll: File format not recognized
sh-3.1$ objdump -h lib/ruby/1.9.1/i386-mingw32/dl.so
C:\projects\rubyinstaller-git\sandbox\devkit\mingw\bin\objdump.exe: lib/ruby/1.9.1/i386-mingw32/dl.so: File format not recognized
sh-3.1$ objdump -h miniruby.exe

miniruby.exe:     file format pei-i386

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         001675fc  00401000  00401000  00000600  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE, DATA
  1 .data         0000196c  00569000  00569000  00167c00  2**5
                  CONTENTS, ALLOC, LOAD, DATA
...
sh-3.1$ make test-all TESTS='openssl fiddle psych'
./miniruby.exe ...
...
684 tests, 3621 assertions, 0 failures, 0 errors, 1 skips

ruby -v: wio-ruby 2.0.0dev (2011-11-04 trunk 33634) [i386-mingw32]
sh-3.1$ make test
...
PASS all 943 tests
./miniruby.exe ...
Driver is wio-ruby 2.0.0dev (2011-11-04 trunk 33634) [i386-mingw32]
Target is wio-ruby 2.0.0dev (2011-11-04 trunk 33634) [i386-mingw32]


KNOWNBUGS.rb .
PASS all 1 tests


V:\sandbox\ruby19_mingw>dumpbin -summary bin\ruby.exe
Microsoft (R) COFF/PE Dumper Version 10.00.30319.01

Dump of file bin\ruby.exe
LINK : warning LNK4094: 'bin\ruby.exe' is an MS-DOS executable; use EXEHDR to dump it

V:\sandbox\ruby19_mingw>dumpbin -summary bin\msvcrt-ruby191.dll
Dump of file bin\msvcrt-ruby191.dll
LINK : warning LNK4094: 'bin\msvcrt-ruby191.dll' is an MS-DOS executable; use EXEHDR to dump it


### FAILURE - Build with Windows SDK for Windows 7 ###

C:\Users\Jon\DOCUME~1\RubyDev\ruby-git\build>..\win32\configure.bat --prefix=C:\projects\ruby200-mswin
--target=i686-mswin32 --disable-install-doc --disable-win95

C:\Users\Jon\DOCUME~1\RubyDev\ruby-git\build>nmake
...
C:\Users\Jon\DOCUME~1\RubyDev\ruby-git\build>nmake test
...
PASS all 943 tests
...
KNOWNBUGS.rb .
PASS all 1 tests
C:\Users\Jon\Documents\RubyDev\ruby-git\build>nmake install
...
C:\Users\Jon\Documents\RubyDev\ruby-git\build>\projects\ruby200-mswin\bin\ruby.exe --version
This program cannot be run in DOS mode.

C:\Users\Jon\Documents\RubyDev\ruby-git\build>dumpbin -summary \projects\ruby200-mswin\bin\ruby.exe
Microsoft (R) COFF/PE Dumper Version 10.00.30319.01

Dump of file \projects\ruby200-mswin\bin\ruby.exe
LINK : warning LNK4094: '\projects\ruby200-mswin\bin\ruby.exe' is an MS-DOS executable; use EXEHDR to dump it

----------------------------------------
Feature #5562: Improvement of Windows IO performance
http://redmine.ruby-lang.org/issues/5562

Author: Hiroshi Shirosaki
Status: Open
Priority: Normal
Assignee: 
Category: 
Target version: 


=begin
I suggest a patch to improve Windows IO performance.

Ruby's text mode IO is much slower than binary mode.
On Windows text mode is default, so Windows IO is slow.
I assume that's mainly because of CRLF linefeed code conversion.

My idea to improve IO performance is as below.
- Change default linefeed conversion from Universal newline to CRLF newline on Windows
- Use binary mode process with OS's text mode if only CRLF conversion is needed
- Use Ruby's text mode with universal newline conversion if encoding conversion is needed

Although that causes io.c code to be more complicated, IO with CRLF conversion performance seems to be improved much.
I confirmed "make test-all TEST=ruby" have been passed. There was 3 errors, but ruby without this patch had same errors.
I think this patch doesn't affect other OS.

Line endings of "p" or "puts" writing is LF on trunk, but CRLF on 1.8.7 or 1.9.2.
This patch reverts to CRLF.


Here is #1332 benchmark test and results.

time = [Time.new]
c = ''
'aaaa'.upto('zzzz') {|e| c << e}
4.times { c << c }
time << Time.new
File.open('out.file','w') { |f| f.write(c) }
time << Time.new
c = File.open('out.file','r') { |f| f.read }
time << Time.new
0.upto(time.size - 2) {|i| p "#{i} #{time[i+1]-time[i]}" }


- Result

ruby 1.8.7 (2011-06-30 patchlevel 352) [i386-mingw32]
"0 0.78125"
"1 0.6875"
"2 0.5625"

ruby 2.0.0dev (2011-11-03) [i386-mingw32]
"0 0.59375"
"1 1.09375"
"2 1.296875"

ruby 2.0.0dev (2011-11-03 trunk 33615) [i386-mingw32] with this patch
"0 0.625"
"1 0.65625"
"2 0.34375"
=end



-- 
http://redmine.ruby-lang.org