Re: Win32 Non-ASCII Filename Access

On Thu, 10 Mar 2005 07:18:37 +0900, Berger, Daniel
<[e-mail]> wrote:
>> -----Original Message-----
>> From: Austin Ziegler [mailto:[e-mail] ]
>> Sent: Wednesday, March 09, 2005 2:13 PM
>> To: [e-mail] 
>> Subject: Re: Win32 Non-ASCII Filename Access
> Ok, I think I see your point now. I also noticed that the -U
> option hasn't been taken yet, btw. :)

;) Still, 

>>>> I don't have the Ruby code in front of me, but a lot of things
>>>> probably wouldn't work quite the same if we used the UNICODE
>>>> macro. String#each_byte, anyone?
>>> It would be a case of caveat programmor in the case of String
>>> methods and the like. If you're using unicode, then something
>>> like String#each_byte would either return 2 bytes per char, or
>>> would yield two separate 8-bit chars.
>> But then we have two *different* versions of Ruby simply based on
>> how Ruby was compiled. If someone doesn't #define UNICODE,
>> they'll get one set of semantics; if they do, they'll get
>> another. BAD BAD BAD language.
> Even if you call _A or _W at runtime based on a global variable
> like $KCODE, how do you keep the String semantics identical for 1
> byte versus 2 byte characters for a method like String#each_byte?

Well, to a point, you don't. But I think you missed the point of
what I meant to implement:

  1. In Win32, always call _W with the wchar_t version.
  2. Immediately convert the wchar_t to char with WideToMultiByte
     using CP_UTF8 ($KCODE == 'UTF-8') or CP_ACP ($KCODE == anything
     else). In theory, I could support EUC-JP and SJIS, but I'm not
     sure whether that's the case or not.
  3. Pass the char version around. That means that #each_byte is
     still not *quite* going to work, but it's going to work better
     and more predictably than with wchar_t. Additionally, it *will*
     work 100% with regular expressions.
  4. When working with the file, convert it back from char to
     wchar_t. Potential problem: $KCODE may have changed in the
     intervening period. This will definitely be solved when the
     m17n strings are ready, but can either be documented as a
     limitation, or we can lobby Matz to add an encoding field to a
     String now (defaulting to +nil+).

Callers to Win32 APIs would be encouraged to do the same; there
could be some helper functions or macros to accomplish the
conversion process both directions.

This works very well, by and large, at least with what we're doing
at work (and we can't afford to miss files). The only other thing
that would happen is that when working with files that are fully
anchored (e.g., C:/foo/bar/baz), the "unipath" marker (\\?\) would
be added to the front and all forward slashes would be converted to
backslashes (\\?\C:\foo\bar\baz).

This particular item hasn't really been discussed to death, at least
not its relationship with Win32. There is a definite need for this
in Windows.

Matz? What do you say? Should I make a patch? If so, should I make
it for 1.9 or 1.8 as well? Can I use $KCODE for this, or do I need
something else? How do we solve the problem mentioned in #4?

-austin
-- 
Austin Ziegler * [e-mail]
               * Alternate: [e-mail]