From: "Florian Groß" <florgro@...>
Date: 2005-04-25T04:37:33+09:00
Subject: Re: announcing RubyLexer 0.6.0

vikkous wrote:

>>One simple example would be adding a ".=" assign-result-of-method-call
>>operator as in "foo = 'bar'; foo .= reverse"
> 
> At first, I thought, "This guy is dreaming; my code is just too rigid
> to allow extensions of that kind very easily.". But of course, it
> wouldn't be too hard for me to special case this one operator in for
> you if you wanted to... it'd just be a quick hack in RubyLexer#dot...
> in fact, it could be done in a subclass:
> [warning: untested code!]
> 
> class FlorianRubyLexer < RubyLexer
>   def dot(ch)
>      #this is the routine in RubyLexer that handles tokens beginning
> with '.'
>      if readahead(2)=='.='
>         KeywordToken.new(@file.read(2),@file.pos-2)
>      else
>         super
>      end
>   end
> end

Which is exactly what I thought would be a good way of extending. This 
looks good.

Another thing that I would be able to make good use of is getting the 
next expression, whatever it might be.

Let's say I have this code:

z = if (x + y) * 2 > 2 then
   code here
end

It would then be very nice if I could lex until I see the 'if' then say 
'give me an atomic expression' which would parse until the 'then' and 
then say 'give me an atomic expression' again which would parse until 
the 'end'. Basically I don't want to match paired things (parentheses, 
do .. end, class definitions etc.) at the transformation level.

Yup, that sample does not introduce any new syntax -- I would like to 
transform it to this:

z = if ((x + y) * 2 > 2).true? then
   code here
end

Which is why I would need to find a sub-expression.

Also note that just grabbing everything until the next 'then' would not 
be good enough:

# Nonsense code, but still valid
if x > if x < 5 then 3 else 2 end then
   puts "Good!"
end

If it weren't for that point then IRB's lexer would be a more or less 
nifty match already.

Does this sound like something that can be done without too much trouble?

For doing code transformations it is of course also important that you 
can turn back the stream of tokens into a String easily. I did this with 
IRB's lexer by using the .line_no and .pos methods of tokens, but that 
was not too good a match, actually.

>>See attachment.
>>�    pre.rb �
> 
> Now that's deviant! Whitespace as a fancy string delimiter... I don't
> even know if that's what breaks RubyLexer, but that's sick, man, really
> sick.

Oh, that is still relatively simple. There's worse stuff happening under 
the surface.

> Ps: what does the code do?

If you invoke it as  ruby -rpre file.rb  it will pre-process file.rb 
before letting Ruby handle it. It parses simple directives that look 
like this:

#!if rand > 0.5 then
   }{}{ # Cause a Syntax Error
#!else
   puts "Hello World"
#!end

That file would produce a Syntax Error at parse-time half of the time 
and output Hello World in the other cases.

 > "Hello"
 > 1+5
 > Time.now
#!gsub!(/^>/, "puts")

And that would make '>' at the beginning of a line mean 'output this: '.

It's basically something like the C preprocessor, but in a more Rubyish 
manner written in obscure style. I guess it is pretty useless after all.