From: Tom Sawyer <transami@...>
Date: 2002-07-04T15:46:28+09:00
Subject: Re: the pains of parsing

by the way, here's my result:

def tokenizer3 string
  tokens = {'<'=>'>', '['=>']', '"'=>'"', '{'=>'}', '*'=>'*', "'"=>"'"}
  items = []
  while string.size > 0
    if tokens.keys.include?(string[0,1])
      end_index = string.index(tokens[string[0,1]], 1)
      item = string[0..end_index]
      items << item
      string = string[end_index+1..-1]
      while item.count(item[0,1]) > item.count(tokens[item[0,1]])
        end_index = string.index(tokens[item[0,1]])
        item << string[0..end_index]
        string = string[end_index+1..-1]
      end
    else
      end_index = string.index(/([[{<*"'\s]|\z)/, 1)
      item = string[0..end_index-1].strip
      items << item if not item.empty?
      string = string[end_index..-1]
    end
  end
  items
end

~transami (tom)


On Wed, 2002-07-03 at 21:54, Sean Russell wrote:
> Tom Sawyer wrote:
> 
> > i find myself spending alot of time writing routines to parse complex
> > strings. here's a mock example string of my current problem:
> > 
> > a<b>[c]{d}"e"f {{g}} [[h]]*i**j*"k"
> 
> Hi Tom,
> 
> This isn't the most beautiful or efficient code I've ever written, but the 
> method is only 22 lines long.  You could probably get it down to 18 lines 
> if you obfuscated it a little more.  Sorry for the perlisms.  Once you 
> start using them, they're a hard habit to kick.  
> 
> You can't solve your problem with regexps alone.  Oh, BTW, I wasn't sure if 
> you really meant you wanted whitespace to be stripped; if so, change the 
> line:
> 
>   string = $'
> 
> to read
> 
>   string = $'.strip
> 
> <code>
> def tokenizer string
>   tokens = { ?< => '>', ?[=>']', ?"=>'"', ?{=>'}', ?*=>'*', ?'=>"'"}
>   items = []
>   while string.size > 0
>     if tokens.keys.include? string[0]
>       end_index = string.index( tokens[string[0]], 1 )
>       item = string[0..end_index]
>       items << item
>       string = string[end_index+1..-1]
>       while item.count( item[0,1] ) > item.count( tokens[item[0]] )
>         end_index = string.index( tokens[item[0]] )
>         item << string[0..end_index]
>         string = string[end_index+1..-1]
>       end
>     else
>       string =~ /(.*?)(?=[<[{"*'])/
>       items << $1
>       string = $'
>     end
>   end
>   items
> end
> 
> puts tokenizer( %Q{a<b>[c]{d}"e"f {{g}} [[h]]*i**j*"k"} ).inspect
> </code>
> 
> <example>
> ser@ender ~% ruby tokenizer.rb
> ["a", "<b>", "[c]", "{d}", "\"e\"", "f ", "{{g}}", " ", "[[h]]", "*i*", 
> "*j*", "\"k\""]
> </example>
> 
> 
> -- 
>  |..  "A scientist is one who finds interest in the kinetic energy of
> <|>    Jell-O moving at ridiculous velocities...an engineer is one who can
> /|\    find a real-life application for such silliness."
> /|    -- anon
>  |         
>