From: Ben Nagy <ben@...>
Date: 2006-06-06T14:25:37+09:00
Subject: Re: Like BitStruct, but better?

> -----Original Message-----
> From: Joel VanderWerf [mailto:vjoel@path.berkeley.edu] 
[...]
> Ben Nagy wrote:
[...]
> > What I need is a way to rapidly build classes that 
> represent structured,
> > binary data. Bit-struct only lets me define fields up to 32 
> bits long, which
> > is bad, and there isn't (AFAICT) a way to have variable 
> length fields in
> > mid-structure. As an example I might want to represent a 
> protocol header
> > which has a Length field followed by some data of the 
> appropriate length,
> > and then followed by some more fixed structure elements. 
> 
> Hi, Ben.

Hi! Thanks for the response. :)

> Unsigned and signed integer fields >32 bits (and 1..16, 24, 
> or 32 bits)
> are supported since bit-struct-0.8. And of course the various 
> character
> field types can be any _fixed_length.

Ahh, the latest code I googled when I started was 0.5, and I just wrote my
classes that needed 64 bit ints with a hi and lo part. Thanks for the tip,
I'll grab the newer code.

> However, you're right about variable length fields that occur 
> somewhere
> except at the end of the structure: bit-struct doesn't support them.
[...]
> If you want to handle variable length embedded fields in a 
> single class,
>  that's going to be tricky. How long are the length fields? Are the
> length fields fixed length? Or are they like netstrings[1]? Encoded as
> ascii, big-endian unsigned, or ...?

In general, the length fields in network protocol headers are going to be
fixed length, from what I've seen. Protocols that need variable length data
all over the place seem to be using ASN.1/PER these days (another library on
my wishlist ;). I can't use ASN.1 for my purposes because I need to break
things before I send them, and the parser classes immediately choke at that
point when trying to marshal the data for sending.

> When you access fields at a higher offset than the beginning of the
> var-length field, the accessor will have to read the length and adjust
> the offset accordingly. If there is another var-length field, that may
> require skipping past that field as well...

Have you thought about not using String as the base class? For instance,
OpenStruct would be almost OK for my purposes, if it sustained ordered
output. If I had to hack things up without guidance I would probably start
with a Hash and have :fieldname -> pos, val, type internally. You wouldn't
be able to treat the whole object like a string directly, but overloading
to_s shouldn't be too ugly syntactically? The type definition would still be
used to meta-create a class 'parse' method that does the parsing, to convert
from a raw string (or I guess you could just use o=Class.new(String)).

The trouble is that I'm still getting to grips with the nontrivial parts of
Ruby metaprogramming, so there are a few fiddly details that I'm mentally
glossing over. I _think_, for example, that it would be cool to be able to
define the class fields as using any calculated value at the time of
instantiation. Take UDP for example, where the checksum is performed over a
pseudoheader + payload. That rapidly starts to twist my brain though, since
the UDP object would need to know if it is the payload of an IP object
before being able to calculate the checksum. Gah. Maybe some Proc that is
called when you call o.field.refresh (which gets called the first time
during instantiation)... but then the checksum depends on other calculated
fields like length so it needs to be done last... ok my brain just exploded.
:(

> Anyway, I have wanted such a field, too, so maybe I will implement one
> of these variations someday. Or if anyone has ideas...

I can't wait. :)

Cheers,

ben