From: Robert Klemme <shortcutter@...>
Date: 2006-08-08T03:35:06+09:00
Subject: Re: Simple question: combine a quoted string into a single token

Squeamizh wrote:
> Hi,
> 
> I have a program which separates each line of a text file into tokens,
> using whitespace as a delimiter (I do this with String.split).  This
> suits my needs for the most part, but now I need the ability to treat
> quoted strings as single tokens.  Note that the quoted string could be
> multiple words, or even a 0-length string.
> 
> Could anyone recommend a basic strategy for doing this?  Should I deal
> with this when I first tokenize each line, or should I combine tokens
> appropriately during parsing when I see a double-quote?
> 
> Help would be greatly appreciated.
> 

Something along the lines of

line.scan %r{
   "[^"]*"   |
   \S+
}x

HTH

	robert