From: Caleb Clausen <caleb@...>
Date: 2008-09-22T17:47:14+09:00
Subject: [ruby-core:18788] MultiEncodingString

Now I'm hardly an expert on any character set but ascii, but I've been 
watching the recent discussions on m17n with everyone gnashing their 
teeth over how to make multiple scripts within the ruby interpreter 
really work smoothly. In the absence of a single character set to rule 
them all, it occurs to me that maybe something that could help would be 
the ability to have strings that actually use multiple different 
encodings in different sections.

At first I was thinking that core lib String should be modified to 
support this, but obviously it would be a pretty big change. Now I'm 
thinking that a class could be written in ruby that emulates String's 
interface, but with multiple encodings.

If it were me writing it, I'd re-use some code I already wrote: 
Sequence::List, (from Sequence, an external iterator library). It keeps 
an array of strings internally, which are logically catenated together. 
It doesn't emulate String's interface all that closely at the moment, 
but it does support regexp stuff, which is the hardest part. Still and 
all, Sequence::List should be 60-70% of the way to being a 
MultiEncodingString already.

Getting Regexps to work across multiple character sets in the hard part, 
but I think I see a way to do it now, without hacking at the guts of 
oniguruma. The interesting case is when there's two adjacent buffers in 
different encodings and a regexp in yet a third. To deal with this, 
transcode both data buffers into the encoding of the regexp, then match 
against the catenation of them. If there are characters in one of the 
buffers which can't be represented in the encoding of the regexp, then 
convert them to some character which you know does not occur in the 
regexp. Should be good enough for matching purposes. It won't be 
terribly fast; Sequence::List is already slower than I would like. But 
it will probably be acceptable.

I think this is a good idea, but I don't want to start a new project at 
this point.....