From: peterjohannsen@... (pj) Date: 2003-03-10T08:41:51+09:00 Subject: Re: What character sets are available in Ruby ? "MikkelFJ" wrote in message news:<3e6b1d39$0$137$edfadb0f@dtext01.news.tele.dk>... > "pj" wrote in message > news:127ce4a9.0303081532.2011b08f@posting.google.com... > > There is a Ruby FAQ which I read that said that Ruby only supports > > ASCII. That is, AFAIK, a 7 bit character encoding scheme for a very > > old character set (one with less than 128 characters). > ... > > I'm frankly hoping that UTF-8 is available, because I would happily > > stick to UTF-8 -- well, assuming there is something like iconv > ... > Here is what we did: First, the input can be in Unicode (You obviously mean UTF-8 encoded Unicode, as you make clear below.) > This is not entirely exact according to Unicode specs, but it is simply and > works quite well. (The actual identifiers are slightly different as we allow > @ and a few other symbols in ASCII as well, but that is beside the point). > > identifier = /[A-Za-z\x80-\xff][A-Za-z0-9\-\x0080-\xffff]*/ > > Highly interestly an 8-bit characterstream in UTF-8 or Latin-1 can detect > identfiers with the following expression, oblivious to the fact that the > stream: > > utf8ident = /[A-Za-z\x80-\xff][A-Za-z0-9\-\x80-\xff]*/ > > In Latin-1 it works as expected. > In UTF-8 the magic works because any non-ASCII character is made up of a > sequence of one or more characters in the \x80-\xff range and never in the > ASCII range. It is very important that ASCII characters never appear inside > a non-ASCII UTF-8 sequence. This makes it extremely easy to support UTF-8 in > a parser normally dealing with ASCII symbols for operators etc. (+'*\ etc.) > > The expression may think that one character is multiple characters, but that > is completely irrelevant - it works. > > Therefore, I would guess that Ruby could instantly support UTF-8 and Latin-1 > identifiers by a slight change to the lexer. > But, isn't there a string#length -- does it give you character length or byte length ? I am pretty ignorant of Ruby -- does it provide uppercase, lowercase, titlecase functions ? If so, I would guess that they only support UTF-8 if they were designed for UTF-8 ? > UTF-8 has an optional leading 3-character sequence called BOM (Byte Order > Marker) that should be stripped. > > > available to project down to CP-1252 for example when going to US > > English MS-Windows (or down to UCS-2 when going to MS-Windows NT, > > because, I don't think I'd risk believing scattered claims that (some > > of) NT handles UTF-16). > > UTF-16 text has a big endian or little endian BOM 3-character sequence. > Ruby could easily support these by detecting the BOM and converting the > script to UTF-8 before parsing. The conversion is completely mechanical > using a few shift and logical operations per character. > > By converting to UTF-8, the Lexer need not be able to handle 16 bit > characters. Ok, you're talking about how Ruby could be altered to do things, right ? I'm wondering what Ruby out of the box does, altho, it is pretty interesting to hear what Ruby could do too. > > In the script I'm working with, it currently only reads ASCII, Latin-1 and > UTF-8 directly from disk, but does a UCS-2 conversion when receiving script > in-memory from other components. > > It may be the process is not 100% bullet proof, but it seems to work well > enough. > > Mikkel I've not been following, but I read at least a rumor that China put out a new standard (GB18030 ? I probably have the numbers wrong) that cannot be represented in UCS-2, but can be represented in Unicode 3.1 (ie, UTF-8, UTF-16, etc). I dunno if that matters to anyone outside of China though.