From: Robert Klemme <bob.news@...>
Date: 2005-03-21T23:54:50+09:00
Subject: Re: Iterating through a string and removing leading characters


"Randy Kramer" <rhkramer@gmail.com> schrieb im Newsbeitrag
news:200503210919.00242.rhkramer@gmail.com...
> On Monday 21 March 2005 04:44 am, Robert Klemme wrote:
> > Some remarks:
> >
> >  - The comparison between 5 and 6 does not seem fair, as you iterate
in 6
> > but not in 5.
>
> Oops, for a minute I thought I had really screwed up (like by not doing
5 the
> 10000 times).  AFAIK, I don't need to iterate (through the string) in 5
as
> the RE is not anchored to the beginning of the string--it still checks
the
> entire string (line) for that pattern, which is what I try to achieve in
6 by
> the iteration through the string and check a character then invoke RE
> approach.  I am sure that my Ruby code to do that is not the best, and I
may
> learn something by making it better, but I agree with your conclusion /
> recommendation (below) at least for the time being (although I do plan
to
> play with str::scan and StringScanner at least a little bit (I presume
they
> do similar things, but perhaps the 2nd is optimized somehow,
particularly if
> I "require C" or whatever)).

A particular performance show stopper in test 6 is String#[] i.e. you
create a new String object for each test; object creation is comparatively
expensive even though Strings share their internal buffer.  But the GC has
to be informed etc. and this is quite some overhead.  If you want fast
code, create as few instances as possible.  The same holds for Java in 99%
of all cases.

Another general remark: it should be faster to use a range, Fixnum#upto or
Fixnum#times for iterating because then you have iteration in C and you
don't need to recalculate the limit on each iteration:

# old
i = 0
until i==s1.length-6 do
  if s1[i] == 91
    s1[i,s1.length] =~
/\A\[(([A-Z]\w*)\.)?(.*)(#([A-Z]\w*))?\](\[(.*)\])?\]/
  end
  i += 1
end

# with range
(0...(s1.length-6)).each do |i|
  if s1[i] == ?[
    s1[i,s1.length] =~
/\A\[(([A-Z]\w*)\.)?(.*)(#([A-Z]\w*))?\](\[(.*)\])?\]/
  end
end

# with upto
0.upto(s1.length-7) do |i|
  if s1[i] == ?[
    s1[i,s1.length] =~
/\A\[(([A-Z]\w*)\.)?(.*)(#([A-Z]\w*))?\](\[(.*)\])?\]/
  end
end

# with times
(s1.length-6).times do |i|
  if s1[i] == ?[
    s1[i,s1.length] =~
/\A\[(([A-Z]\w*)\.)?(.*)(#([A-Z]\w*))?\](\[(.*)\])?\]/
  end
end

And you can use "?[" instead of "91" which is far less readable.

> >  - You don't use String#scan or #split which you are likely to need in
> > practice, because  you want to sift through complete documents and
want to
> > treat all occurrences.
>
> I need to let that sink in a bit.  In general I do want to treat all
> occurrences, but I plan to scan a line (actually a paragraph) at a time,
and
> some things can only occur at the beginning of a line, so those would
only be
> checked at the beginning of a line.

Hm, if you know that the size of files is limited (i.e. something like
just a few KB) then it's usually worth slurping in the whole file with
something like this

contents = File.open(f){|io| io.read}

and then iterate through the whole thing with #scan.  You can still use ^
to anchor at line beginnings.

# get the initial sequen until the first non whitespace
# just an example
contents.scan /^\s+\S/ do |m|
  p m[0]
end

> >  - The differences between the check-first-char approach and the pure
RE
> > approach are so insignificant that I'd not bother using the more
complex
> > code.  I'd stick with pure RE based approaches and only try to
optimize if
> > performance is bad.  (You mentioned premature optimization already...
:-))
>
> Thanks!  I pretty much agree at this time, although I want to play a
little
> bit with StringScanner.

Of course, new toys have to be played with! :-)

Kind regards

    robert