From: Daniel Berger <djberg96@...>
Date: 2008-02-08T14:22:14+09:00
Subject: Possible bug with bignum.c

Hi all,

Ruby 1.8.6 (SVN trunk)

In bignum.c the rb_big2ulong_pack() function is declared to have a 
return type of unsigned long, but there's a conditional where it could 
return a negative (signed) value:

unsigned long
rb_big2ulong_pack(x)
     VALUE x;
{
     unsigned long num = big2ulong(x, "unsigned long");
     if (!RBIGNUM(x)->sign) {
	return -num; /* HERE */
     }
     return num;
}

The same is true of the rb_big2ulong() function.

unsigned long
rb_big2ulong(x)
     VALUE x;
{
     unsigned long num = big2ulong(x, "unsigned long");

     if (!RBIGNUM(x)->sign) {
	if ((long)num < 0) {
	    rb_raise(rb_eRangeError, "bignum out of range of unsigned long");
	}
	return -num; /* HERE */
     }
     return num;
}

How should these functions be refactored?

Regards,

Dan