From: Ken Bloom <kbloom@...>
Date: 2008-10-16T10:09:42+09:00
Subject: Re: regexp with accent insensitive ??

On Mon, 13 Oct 2008 15:23:23 -0500, Davi Barbosa wrote:

> Thank you for your answer, but I'm working with a lot of languages, so I
> don't know where someone can put an accent.
> 
> For the moment, I just discovered that I can't remove the accents with
> Iconv like I said before. Here, it works only under irb.. I described
> this problem here: http://www.ruby-forum.com/topic/70827#738081 Another
> problem with utf-8 under ruby is that ruby can't index correctly the
> string. For example: '叩b'[2..2] gives the second half of '叩'. I
> discovered how to workaround using the unicode version of regexp: $KCODE
> = 'u'
> '叩b'.split(//m) == ["叩", "b"]
> 
> Without these problems, I think that I know how to make it without false
> matchs with an ugly loop.
> If str and regexp are the versions without accents, str =~ regexp gives
> the position of the match and str[regexp].length the length. With these
> two numbers, It's possible to make the highlight in the original string.
> It's something like:
> ascii_string = Iconv.conv('US-ASCII//TRANSLIT','UTF-8',string)
> ascii_search = Iconv.conv('US-ASCII//TRANSLIT','UTF-8',search) regexp =
> Regexp.new(Regexp.escape(ascii_search),true) position = (ascii_string =~
> regexp)
> size = ascii_string[regexp].length
> highlighted = ascii_string[0..(position-1)]+'<span
> class="highlight">'+ascii_string[position..(position+size-1)]+'</
span>'+ascii_string[(position+size)..-1]
> 
> Of course, it need some modifications to put this in a loop (and I need
> to use the vector version of the string to index correctly the string).

You can use a StringScanner (require 'strscan') to properly do this in a 
loop, because StringScanner#pos will tell you the starting position of 
the match, where String#scan will not.

Consider whether Ruby 1.9.0 is stable enough for your purposes because it 
handles Unicode natively and should save you from needing to have a 
vector version of the string.

-- 
Chanoch (Ken) Bloom. PhD candidate. Linguistic Cognition Laboratory.
Department of Computer Science. Illinois Institute of Technology.
http://www.iit.edu/~kbloom1/