[ruby-core:18532] Ruby 1.9 string performance

I would like to submit the attached patch to string.c which substantially  
improves Ruby 1.9's string performance when operating on ASCII data,  
especially when working in a UTF-8 locale.

As a result of these changes the previously very slow benchmark  
"bm_so_count_words.rb" now runs about 15 times faster on my system (UTF-8  
locale) - about the same speed as it did on 1.8. It now runs about 7 times  
faster than it did under an ASCII locale (eg: LANG=C). And it now runs at  
the same speed whether you use UTF-8 or ASCII locale.

The main things I did are:

1) A number of methods failed to set the 7-bit coderange flag when they  
should have. I actually think that these may be bugs. In fact I think the  
usage of the coderange flags is a bit strange. "rb_str_modify()" clears  
the flags, so you really should make sure that they are set again  
correctly after every rb_str_modify(). This was not happenning in several  
places. Even little old String#clear was clearing these flags. As an  
example, if you say:

	s = "abc".force_encoding("UTF-8")
	s.clear
	s << "xyz"

Before the "clear" the 7-bit flag is set, so ruby knows that "s" is all  
ascii, and can do faster operations. But the "clear" was clearing the  
7-bit flag, causing subsequent use of this string to be slow, as from then  
on ruby thinks it is a multi-byte-char string. In this case the "append"  
thinks it is appending a string of ascii chars to a multi-byte-char  
string, so it leaves the 7-bit flag clear.

Note: I have NOT checked methods outside class String which return strings  
for other occurrences of this problem - it may also be in other places.

2) I added "single-byte" optimization to a number of methods that didn't  
already have it. Perhaps some of them aren't used that much and don't need  
the optimization, but I was in the mood! They are (all String#):

	casecmp
	strip/lstrip/rstrip
	ljust/rjust/center
	upcase/downcase/swapcase
	count
	delete/squeeze

Certainly I use strip & casecmp quite a bit in my apps.

3) I did a separate optimization in String#split when the parameter is a  
string. Now s.split(":") is about twice as fast as s.split(/:/), so  
libraries using regexps like /:/ because they used to be slightly faster  
than strings, should be changed to strings.

4) I also added "each_codepoint" as described previously. If you don't  
think it should go into 1.9, please feel free to remove it, though I feel  
it is useful.


I haven't tested these changes exhaustively, but it passes the "rubytest".  
I am not sure of the procedure for further testing. I hope I haven't  
broken anything!

There is something I'd like Matz or another developer to check: In  
"lstrip" & "rstrip" I made the assumption that all "space" characters in a  
multi-byte ascii-compatible character set (eg: UTF-8) are single-byte. I  
am not 100% sure if this assumption is correct.


I would also like to propose an optimization, which though minor, involves  
lots of changes:

When processing strings sequentially, there is a lot of code something  
like:
	c = rb_enc_codepoint(s, end, enc);
	s += rb_enc_mbclen(s, end, enc);

In other words, there are 2 function calls - one to get the next  
codepoint, and one to get the length of the character in bytes to advance  
the pointer. The first function has to calculate the length of the  
character anyhow, so why not change it to something like:
	c = rb_enc_codepoint(s, end, enc, &len);
	s += len;

I would think that this is faster, and also easier to understand.

Cheers
Mike