[ruby-core:121316] [Ruby master Bug#20025] Parsing identifiers/constants is case-folding dependent

Issue #20025 has been updated by hsbt (Hiroshi SHIBATA).

Backport changed from 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED to 3.0: REQUIRED, 3.1: REQUIRED, 3.2: DONE

ruby_3_2 commit:6c24731837f88d67517cfc590cb496daed7a0ef5 merged revision(s) commit:79eb75a8dd64848f23e9efc465f06326b5d4b680.

----------------------------------------
Bug #20025: Parsing identifiers/constants is case-folding dependent
https://bugs.ruby-lang.org/issues/20025#change-112281

* Author: kddnewton (Kevin Newton)
* Status: Closed
* Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: DONE
----------------------------------------
When CRuby parses identifiers, it is encoding-dependent. Once the identifier is found, it determines if it starts with a uppercase or lowercase codepoint. This determines if the identifier is a constant or not.

The function is charge of this is `rb_sym_constant_char_p`. For non-unicode encodings where the leading byte has the top-bit set, this relies on onigmo's `mbc_case_fold` to determine if it is a constant or not (as opposed to `is_code_ctype`).

This works for almost every single codepoint in every encoding, but has one very weird edge case. In the Windows-1253 encoding for the 0xB5 byte, it's the micro sign. The micro sign, when case folded, becomes the uppercase mu character, and then the lowercase mu character, or 0xEC. This means that even though 0xB5 reports itself as being a lowercase codepoint, it gets parsed as a constant. This example might make this more clear:

``` ruby
class Context < BasicObject
  def method_missing(name, *) = :identifier
  def self.const_missing(name) = :constant
end

encoding = Encoding::Windows_1253
character = 0xB5.chr(encoding)

source = "# encoding: #{encoding.name}\n#{character}\n"
result = Context.new.instance_eval(source)

puts "#{encoding.name} encoding of 0x#{character.ord.to_s(16).upcase}"
puts "  [[:alpha:]] => #{character.match?(/[[:alpha:]]/)}"
puts "  [[:alnum:]] => #{character.match?(/[[:alnum:]]/)}"
puts "  [[:upper:]] => #{character.match?(/[[:upper:]]/)}"
puts "  [[:lower:]] => #{character.match?(/[[:lower:]]/)}"
puts "  parsed as #{result}"
```

this results in the output of:

```
Windows-1253 encoding of 0xB5
  [[:alpha:]] => true
  [[:alnum:]] => true
  [[:upper:]] => false
  [[:lower:]] => true
  parsed as constant
```

To be clear, I don't think the case-folding is incorrect here (and @duerst confirms that it is correct). I believe instead that it is incorrect to use case-folding here to determine if a codepoint is uppercase or not.

Note that this only impacts this one codepoint in this one encoding, so I don't believe this is actually a large-scale problem. But I found it surprising, and think we should change it.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- [e-mail]
 To unsubscribe send an email to [e-mail]
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/