From: Daniel DeLorme <dan-ml@...42.com>
Date: 2007-12-06T14:29:41+09:00
Subject: Re: Unicode in Regex

MonkeeSage wrote:
> Heh, if the topic at hand is only that indexing into a string is
> slower with native utf-8 strings (don't disagree), then I guess it's
> irrelevant. ;) Regarding the idea that you can do everything just as
> efficiently with regexps that you can do with native utf-8
> encoding...it seems relevant.

How so? These methods work just as well in ruby1.8 which does *not* have
native utf8 encoding embedded in the strings. Of course, comparing a
string with a string is more efficient than comparing a string with a
regexp, but that is irrelevant of whether the string has "native" utf8
encoding or not:

$ ruby1.8 -rbenchmark -KU
puts Benchmark.measure{100000.times{ "日本語".index("本") }}.real
puts Benchmark.measure{100000.times{ "日本語".index(/[本]/) }}.real
puts Benchmark.measure{100000.times{ "日本語".index(/[本]/u) }}.real
^D
0.225839138031006
0.304145097732544
0.313494920730591

$ ruby1.9 -rbenchmark -KU
puts Benchmark.measure{100000.times{ "日本語".index("本") }}.real
puts Benchmark.measure{100000.times{ "日本語".index(/[本]/) }}.real
puts Benchmark.measure{100000.times{ "日本語".index(/[本]/u) }}.real
^D
0.183344841003418
0.255104064941406
0.263553857803345

1.9 is more performant (one would hope so!) but the performance ratio
between string comparison and regex comparison does not seem affected by
the encoding at all.

> Someone just posted a question today about how to printf("%20s ...",
> a, ...) when "a" contains unicode (it screws up the alignment since
> printf only counts byte width, not character width). There is no
> *elegant* solution in 1.8., regexps or otherwise.

It's not perfect in 1.9 either. "%20s" % "日本語" results in a string of
20 characters... that uses 23 columns of terminal space because the font
for Japanese uses double width. In other words neither bytes nor
characters have an intrinsic "width" :-/

Daniel