From: Lauri Pesonen <lauri.pesonen@...>
Date: 2007-05-06T00:23:38+09:00
Subject: Re: C DSL anyone?

Related to what the OP was asking for, I've been thinking of
implementing a DSL in Ruby for defining message types for a C-based
distributed framework. We're building a distributed system that
consists of tasks sending each other messages. The messages have types
and payloads.

As things are now, we have to define message types by hand. This
includes defining a payload type, which might be as simple as a single
int, a struct consisting of primitive types, or a hierarchical struct
containing dynamic sized variables. In addition to the payload type we
have to implement functions for marshalling/unmarshalling the payload
and to print the contents out as text.

Writing all this for simple payload types is relatively painless, but
boring. Writing all this for complex hierarchical payloads with
dynamic sized variables is painful, boring, and extremely error prone,
because it usually involves a lot of copy-pasting.

What I have at the moment is something like this (this is from memory,
because I don't have the code at hand):

messages "c-file-basename" do |msgs|
  msgs.define_message "message_type_name" do |m|
    m.add_member :uin32_t, "uint32_t_variable_name"
    m.add_pointer :uint8_t, "uint8_t_pointer_name"
  end

# define other message types that will be included in the same C-file.
# ...
end

This would result in something like the C-code at the end of the email.

The difficulties come with more complicated messages. What if I have a
struct used elsewhere in the system that should be a part of many
different message types? For example, we describe task ids with
structs and these are passed from task to task in message payloads
quite often. One solution is to add it as a basic type to the DSL.
That is, in the same way that the DSL understands what an uint32_t is
and how to marshall and print an uint32_t, I enhance it to understand
what the struct is. The definition of the struct would be in the
common header files of the system. The down side is that whenever
someone defines a new complex type, we must implement that type in the
DSL. Me being the only Rubyist in the project would mean that I'll end
up doing the enhancing ;-)

Another approach would be to enhance the DSL so that you can import
other files. I.e. we could define common structs in files that are
then imported to the message definitions. This to me seems overly
complicated in our case. I think it would make the DSL a lot more
complicated to implement and I'm not sure our use cases require the
functionality. We can always implement the more complex messages by
hand if the DSL is not able to handle them.

I'd appreciate any kind of input on this that the list members might
have. I've read as much about DSLs in Ruby as I could find on the web,
but none of them really covered what I'm trying to do.

-- Lauri

-- generated C-code --

/ * I wrote this on the fly to this email, so it will most likely
contains errors */

typedef struct MESSAGE_TYPE_NAME_ {
  uint32_t uint32_t_variable_name;
  uint32_t uint8_t_pointer_name_len;
  uint8_t* uint8_t_pointer_name;
} MESSAGE_TYPE_NAME

uint32_t message_type_name_marshall(void* msg, void* buf, uint32_t buf_len) {
  MESSAGE_TYPE_NAME* my_msg = (MESSAGE_TYPE_NAME*) msg;
  uint8_t* ptr = buf;

  if (buf_len < (sizeof(uint32_t) + sizeof(uint8_t) *
my_msg->uint8_t_pointer_name_len)) {
    return 0;
  }

  memcpy(ptr, &my_msg->uint32_t_variable_name, sizeof(uint32_t);
  ptr += sizeof(uint32_t);

  memcpy(buf, my_msg->uint8_t_pointer_name, my_msg->uint8_t_pointer_name_len);
  ptr += my_msg->uint8_t_pointer_name_len;

  return (ptr - buf);
}

/* And similar unmarshall and to_text functions */

MSG_TYPE_DEF message_type_name_type = {
  message_type_name_marshall,
  message_type_name_unmarshall,
  message_type_name_to_text
};

-- 
  ! Lauri