From: Wes Nakamura <wknaka@...>
Date: 2000-10-16T05:24:58+09:00
Subject: [ruby-talk:5554] Problem with pack on 64-bit machines


I think I've found a problem with ruby 1.6.1 (non-cvs) Array#pack when 
on a 64-bit machine (in this case, an alpha running linux, so it's
64-bit little-endian).

e.g.:

>> [1].pack "N"
=> "\000\000\000\000"
>> [0x0102030405060708].pack "N"
=> "\001\002\003\004"
>> [0x0000000001020304].pack "N"
=> "\000\000\000\000"

After investigating quite a few things (tracking through the code to
rule things out ended up being a bigger effort than I thought for such a
short block of code!), the problem narrows down to approx line 600 in
pack.c:

rb_str_cat(res, OFF32(&l), NATINT_LEN(long,4));

I just checked on a 32-bit machine, and [0x0102030405060708].pack "N"
gives a RangeError, which I believe is because 0x0102030405060708
is a Bignum on a 32-bit machine, but a Fixnum on a 64-bit one, so
it follows a different path of execution (namely, within NUM2ULONG).

I'm not sure what the solution is.  Have it check the value of the long
during the pack "N" conversion on 64 bit machines and raise a range
error if it doesn't fit into 4 bytes?  Then where OFF32 normally has no
offset on a little endian machine, offset it (by 4 in this case) so
rb_str_cat uses the big-endian 4 least significant bytes?

The problem also affects unpacking, and maybe a couple of other things;
I wasn't able to exhaustively test, so it may not be only "N" that
doesn't work.

Thanks,
Wes.

##  Wes Nakamura  -  wknaka@pobox.com