From: "luislavena (Luis Lavena)" <luislavena@...>
Date: 2012-04-24T21:10:16+09:00
Subject: [ruby-core:44588] [ruby-trunk - Bug #6352][Open] Windows: FD_SET and FD_SETSIZE segv due different compilation flags


Issue #6352 has been reported by luislavena (Luis Lavena).

----------------------------------------
Bug #6352: Windows: FD_SET and FD_SETSIZE segv due different compilation flags
https://bugs.ruby-lang.org/issues/6352

Author: luislavena (Luis Lavena)
Status: Open
Priority: Normal
Assignee: usa (Usaku NAKAMURA)
Category: core
Target version: 1.9.3
ruby -v: 1.9.3-p194


Hello,

As mentioned in #6228 [ruby-core:43951]:

- Ruby compiled with -DFD_SETSIZE=32767 will allocate 32K fd_array elements for fd_set structure [1]
- FD_SET() macro has been redefined in win32/win32.h to use rb_w32_fdset instead [2]
- Other programs (like EventMachine) compiled with a different FD_SETSIZE will cause SEGV.

The technical details for this SEGV were provided by Hiroshi Shirosaki in Note 16, which I'm quoting:
https://bugs.ruby-lang.org/issues/6228#note-16

I think above issue is cause of `fd_array` buffer overflow.

typedef struct fd_set
{
        u_int   fd_count;
        SOCKET  fd_array[FD_SETSIZE];
} fd_set;

On EM, FD_SETSIZE = 1024 and fd_array[1024].
EM uses FD_SET() and FD_SET() seems rb_w32_fdset() on Windows.

In rb_w32_fdset(), FD_SETSIZE = 32767 since rb_w32_fdset is compiled with -DFD_SETSIZE=32767. [3]

    if (i == set->fd_count) {
        if (set->fd_count < FD_SETSIZE) { // FD_SETSIZE = 32767
            set->fd_array[i] = s;                 // `i` could be over 1023
            set->fd_count++;
        }
    }

If above scenario is correct, FD_SETSIZE of Ruby should be equal or less then FD_SETSIZE of EM.

include/winsock2.h has FD_SET macro on mingw, but MRI undef FD_SET and uses rb_w32_fdset() function. It might be better that FD_SET() is macro instead of function.

SEGV is caused by that discrepancy between rb_w32_fdset thinking have 32K of sockets and EventMachine only having 1K to iterate over.

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/ms737873(v=vs.85).aspx
[2] https://github.com/ruby/ruby/blob/trunk/include/ruby/win32.h#L583-590
[3] https://github.com/ruby/ruby/blob/trunk/win32/win32.c#L2457-2474


-- 
http://bugs.ruby-lang.org/