From: Ken Bloom <kbloom@...>
Date: 2007-05-12T02:10:50+09:00
Subject: Re: regex \w allows non english characters

On Fri, 11 May 2007 04:25:42 +0900, Kyle Schmitt wrote:

> I'm making a guess here, but ruby is probably looking at the Hebrew
> characters as a normal range of chars, with a character encoding.  Now
> what encoding Hebrew uses I'm not sure, but for instance the ascii code
> for 'a' is 97.  The code for one of the Hebrew characters is probably 97
> also.  Since ruby doesn't really do UTF, it just sees two characters,
> both with a code of 97, and lets them through.

Unless you're using special fonts that do a special mapping (which is 
generally no longer done these days), non-English characters are always 
found in characters 128-255. Different encodings are simply different 
ways of mapping these characters to different languages. 0-127 are always 
the same English ASCII characters.

שבת שלום
--Ken Bloom

-- 
Ken Bloom. PhD candidate. Linguistic Cognition Laboratory.
Department of Computer Science. Illinois Institute of Technology.
http://www.iit.edu/~kbloom1/