From: peterjohannsen@... (pj) Date: 2003-03-10T21:04:31+09:00 Subject: Re: What character sets are available in Ruby ? "MikkelFJ" wrote in message news:<3e6bde55$0$155$edfadb0f@dtext01.news.tele.dk>... > "pj" wrote in message > news:127ce4a9.0303091534.7c5e7d8d@posting.google.com... > > > > Here is what we did: First, the input can be in Unicode > > > > (You obviously mean UTF-8 encoded Unicode, as you make clear below.) > > No, I meant Unicode as in UCS-2 - that is why the expression was > \x0080-\xffff and not \x80-\xff (although I forget it in the lead > character): > Oh yes, I was mislead by not reading very carefully. > > > identifier = /[A-Za-z\x80-\xff][A-Za-z0-9\-\x0080-\xffff]*/ > > I then went on to argue that it also works in UTF-8 representation. > Right, I see. > > > > But, isn't there a string#length -- does it give you character length > > or byte length ? > > As I also answered in another posting, it is a separate issue. I'm talking > about the lexer process and symbol table lookup. String classes can easily > have 16 or 32 bit representation internally. The main point is that it is > easy to use an 8bit lexer. I think you're addressing identifier parsing -- am I getting that right ? > > > I am pretty ignorant of Ruby -- does it provide uppercase, lowercase, > > titlecase functions ? If so, I would guess that they only support > > UTF-8 if they were designed for UTF-8 ? > > I'm not sure I understand your question. Ruby is case sensitive. > Generally case sensitive languages are great because you completely avoid > the problem of what is an upper case letter (a moving target in Unicode i.e. > large table that must be updated frequently). > However, Ruby uses uppercase to identify constants. This means that Ruby > does not avoid the upper case problem after all. > This problem is not properly addressed by my expression, except if you only > allow A-Z upper case constant. In the end you have use a large table to do > it properly. > Yes, I suppose the "constants begin with upper case letters" is a good rule for an ASCII language, but not so good for a modern one -- or at least, not so easy :) > > > > By converting to UTF-8, the Lexer need not be able to handle 16 bit > > > characters. > > > > Ok, you're talking about how Ruby could be altered to do things, right > > ? > > Well yes - I'm partly talking about how I thought Ruby did (and apparently > does it using the proper -K switch), partly I'm talking about my own > experiences with how UTF-8 support can easily be retrofitted a parser. I > have already used it several times so I exploit every opportunity to spread > the merry message ;-) Actually I picked the expression almost directly from > a parser I'm hacking in Ruby. > > > I've not been following, but I read at least a rumor that China put > > out a new standard (GB18030 ? I probably have the numbers wrong) that > > cannot be represented in UCS-2, but can be represented in Unicode 3.1 > > (ie, UTF-8, UTF-16, etc). I dunno if that matters to anyone outside of > > China though. > > Is this the Big-5? Here is a url that talks about it: http://www.anycities.com/gb18030/introduce.htm My thought is that China has a very difficult fight. I think UCS-2 was an unfortunate, short-sighted mistake that would probably be dead and gone, except that Microsoft invested heavily in it, so heavily that I cannot imagine Windows having a 32bit wchar_t. Because MS-Windows is such a force, I think therefore the UCS-2 mistake will be strong for a long-time. AFAIK, this is fine for almost everyone outside of China, so they are the sole big loser, and isolated, and have a rather difficult road to try to fight it. > Anyway - I think this is one reason that matz refuses to settle for Unicode > only. Initially I though this was wrong, but now I realize how important it > is to not settle for any single format. Indeed I sent an email to Paul > Graham that he considered this point in his new Arc language. > > Mikkel What I wish for, is some language where characters are a primitive, and I would not have to muck about with them myself. But I fear that maybe most languages use UCS-2 at best, so that eventually they will (may ?) all require the programmers to muck about with surrogate pairs and whatnot. I'm lazy, and I want a language that does my work for me (altho of course that is rather utopian dream) :) But I'll grant you that if you use 16 bit characters, your life is much easier with UCS-2 than UTF-16; UTF-16 will only bring you heartache (what do you do with string slices that cut through surrogate characters -- ugh).