From: Vincent Isambart <vincent.isambart@...>
Date: 2007-06-17T18:58:09+09:00
Subject: Re: Ruby Changes Its Mind About Non-Word Characters

> > It is because the � and � characters are not in ISO-8859-1.
>
> The characters were in UTF-8, which is why I asked iconv to convert
> them.  Don't take my word for it though, I gave you the source so you
> could double-check me.

Sorry, my mistake, � and � are indeed in ISO 8859-1. It's just that
they were displayed as question mark which is the default character
used by iconv when a characters is not in the destination encoding.
But here it seems it's the terminal that displays it, not iconv.

> > The ? character has the same code in both UTF-8 and ISO-8859-1 as it
> > is an ASCII character...
>
> My text editor and terminal don't agree
> Firefly:~/Desktop$ cat saved_as_utf8.txt saved_as_latin1.txt
> ��
> ??
> Firefly:~/Desktop$ ruby -e 'p ARGF.read' saved_as_utf8.txt
> saved_as_latin1.txt
> "\302\253\302\273\n\253\273\n"

In fact nobody is wrong. I was talking about the question mark (?)
character that is indeed an ASCII character and so the has the same
code in both UTF-8 and ISO 8859-1.
The "question mark" you are refering to is not a question mark but an
invalid UTF-8 character that is displayed by the terminal as a
question mark.

> I'd view � and � as the same kind of punctuation as ". So it's not a
> letter. And \b should match at letter <> non-letter boundaries.

I agree that the � and � are logically quotes (after all I'm French
and they are used a lot in French), but what I said is that the
Unicode support for Ruby 1.8 is poor (in fact except split(//) you
can't do much) so I am not that suprised by the fact it does not work.