[ruby-core:44919] [ruby-trunk - Bug #6352] Windows: FD_SET and FD_SETSIZE segv due different compilation flags

Issue #6352 has been updated by usa (Usaku NAKAMURA).


Perhaps there is no big difference in this case between macros
and inline functions.
You say that you have already testet your patch with EM, so
it's good to commit yours.
----------------------------------------
Bug #6352: Windows: FD_SET and FD_SETSIZE segv due different compilation flags
https://bugs.ruby-lang.org/issues/6352#change-26508

Author: luislavena (Luis Lavena)
Status: Assigned
Priority: Normal
Assignee: usa (Usaku NAKAMURA)
Category: core
Target version: 1.9.3
ruby -v: 1.9.3-p194


Hello,

As mentioned in #6228 [ruby-core:43951]:

- Ruby compiled with -DFD_SETSIZE=32767 will allocate 32K fd_array elements for fd_set structure [1]
- FD_SET() macro has been redefined in win32/win32.h to use rb_w32_fdset instead [2]
- Other programs (like EventMachine) compiled with a different FD_SETSIZE will cause SEGV.

The technical details for this SEGV were provided by Hiroshi Shirosaki in Note 16, which I'm quoting:
https://bugs.ruby-lang.org/issues/6228#note-16

I think above issue is cause of `fd_array` buffer overflow.

typedef struct fd_set
{
        u_int   fd_count;
        SOCKET  fd_array[FD_SETSIZE];
} fd_set;

On EM, FD_SETSIZE = 1024 and fd_array[1024].
EM uses FD_SET() and FD_SET() seems rb_w32_fdset() on Windows.

In rb_w32_fdset(), FD_SETSIZE = 32767 since rb_w32_fdset is compiled with -DFD_SETSIZE=32767. [3]

    if (i == set->fd_count) {
        if (set->fd_count < FD_SETSIZE) { // FD_SETSIZE = 32767
            set->fd_array[i] = s;                 // `i` could be over 1023
            set->fd_count++;
        }
    }

If above scenario is correct, FD_SETSIZE of Ruby should be equal or less then FD_SETSIZE of EM.

include/winsock2.h has FD_SET macro on mingw, but MRI undef FD_SET and uses rb_w32_fdset() function. It might be better that FD_SET() is macro instead of function.

SEGV is caused by that discrepancy between rb_w32_fdset thinking have 32K of sockets and EventMachine only having 1K to iterate over.

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/ms737873(v=vs.85).aspx
[2] https://github.com/ruby/ruby/blob/trunk/include/ruby/win32.h#L583-590
[3] https://github.com/ruby/ruby/blob/trunk/win32/win32.c#L2457-2474


-- 
http://bugs.ruby-lang.org/