[ruby-core:18716] Re: Encoding Safe Regexp.escape()

On Sep 18, 2008, at 6:07 PM, Michael Selig wrote:

> On Fri, 19 Sep 2008 06:48:41 +1000, James Gray <[e-mail] 
> > wrote:
>
>> As part of my ongoing process to make CSV m17n savvy, I'm needing  
>> an encoding safe Regexp.escape().
> I could be wrong, but I think this may now be working - at least it  
> appears to be here on UTF-16LE & ruby revision 19423

Looks like you are right, yes:

$ ruby_dev -e 'p Regexp.escape("|".encode("UTF-16BE").tap { |c| p c })'
"\x00|"
"\x00\\\x00|"

Good.

I tried to remove my specialty escape_re() routine in favor for the  
new Regexp.escape().  Unfortunately, it still throws several errors  
with my tests:

Failed to support Big5.
Failed to support Emacs-Mule.
Failed to support EUC-TW.
Failed to support GB18030.
Failed to support GBK.
Failed to support ISO-8859-16.
Failed to support KOI8-R.
Failed to support KOI8-U.
Failed to support Windows-1251.
Failed to support IBM437.
Failed to support IBM737.
Failed to support IBM775.
Failed to support CP850.
Failed to support IBM852.
Failed to support CP852.
Failed to support IBM855.
Failed to support CP855.
Failed to support IBM857.
Failed to support IBM860.
Failed to support IBM861.
Failed to support IBM862.
Failed to support IBM863.
Failed to support IBM864.
Failed to support IBM865.
Failed to support IBM866.
Failed to support IBM869.
Failed to support Windows-1258.
Failed to support GB1988.
Failed to support macCentEuro.
Failed to support macCroatian.
Failed to support macCyrillic.
Failed to support macGreek.
Failed to support macIceland.
Failed to support macRoman.
Failed to support macRomania.
Failed to support macThai.
Failed to support macTurkish.
Failed to support macUkraine.
Failed to support GB2312.
Failed to support GB12345.
Failed to support Windows-1252.
Failed to support Windows-1250.
Failed to support Windows-1256.
Failed to support Windows-1253.
Failed to support Windows-1255.
Failed to support Windows-1254.
Failed to support TIS-620.
Failed to support Windows-874.
Failed to support Windows-1257.
Failed to support MacJapanese.
Failed to support UTF8-MAC.

You guys can pick apart the logic errors in my implementation, but it  
seems to work on all of the above to at least some extent.

>> Cannot convert to ISO-2022-JP-2
>> Cannot convert to UTF-7
> These encodings are marked as "dummy" by Ruby (see Encoding::list  
> output). I think it means that they are not properly supported  
> though I don't know it what ways. So you may very well be able to  
> ignore them.

Yes, and Encoding has a new dummy?() method to support this.  Here are  
the docs:

$ ri_dev -T  
Encoding 
#dummy?--------------------------------------------------------  
Encoding#dummy?
      enc.dummy? => true or false
------------------------------------------------------------------------
      Returns true for dummy encodings. A dummy encoding is an encoding
      for which character handling is not properly implemented. It is
      used for stateful encodings.

        Encoding::ISO_2022_JP.dummy?       #=> true
        Encoding::UTF_8.dummy?             #=> false

After filtering these dummy encodings out of my tests, I'm getting a  
100% support ratio for all of Ruby's encodings.  I'm still flushing  
some minor issues, but it seems I'm past the hard part.  I'll commit  
my changes soon and let everyone critique them.

Thanks again for all the help in this matter and do let me know if you  
find flaws in anything I've posted so far.  I really appreciate the  
feedback so far.

There's another change in the latest build by the way.   
Encoding::NoConverter became Encoding::NoConverterError.

James Edward Gray II