From: "noraj (Alexandre ZANNI) via ruby-core" <ruby-core@...>
Date: 2023-01-08T00:45:31+00:00
Subject: [ruby-core:111731] [Ruby master Feature#19317] Unicode ICU Full case mapping

Issue #19317 has been updated by noraj (Alexandre ZANNI).





duerst (Martin D��rst) wrote in #note-3:

> Something like this was discussed. My recollection was that it was rejected because it was overkill for the case at hand, and there was no other functionality in core Ruby that needed it.



Maybe but that would be clearer if other options need to be passed as well, more standard (it could plug well with [RDoc::I18n::Locale](https://ruby-doc.org/3.2.0/stdlibs/rdoc/RDoc/I18n/Locale.html) that already uses [ IETF BCP 47 language tag](https://en.wikipedia.org/wiki/IETF_language_tag) or with [IRB::Locale](https://ruby-doc.org/3.2.0/stdlibs/irb/IRB/Locale.html) rather than having to customly map tr_TR and az_AZ with turkic and lt_LT with lithuanian), this would also map better with locales from the system (eg. `/etc/locale.conf`, `LANGUAGE` environment variable).



----------------------------------------

Feature #19317: Unicode ICU Full case mapping

https://bugs.ruby-lang.org/issues/19317#change-101134



* Author: noraj (Alexandre ZANNI)

* Status: Assigned

* Priority: Normal

* Assignee: duerst (Martin D��rst)

----------------------------------------

As announced in [Case Mapping](https://docs.ruby-lang.org/en/master/case_mapping_rdoc.html#label-Default+Case+Mapping), Ruby support for Unicode case mapping is not complete yet.



Unicode supports in Ruby is pretty awesome, it works by default nearly everywhere, things are implemented the right way and works as expected by the UTRs.



But some features are still missing.



To reach [ICU Full Case Mapping support](https://unicode-org.github.io/icu/userguide/transforms/casemappings.html#full-language-specific-case-mapping), a few points need to be enhanced.



### context-sensitive case mapping



* [ ] cf. [Table 3-17 (Context Specification for Casing) of the Unicode standard](https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf) and [ucd/SpecialCasing.txt](https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt).



```ruby

"����".downcase # returns ���� instead of ����

```



Output examples in ECMAScript:



```

��    ������ ��

��a   ������ ��a

a��   ������ a��

a��a  ������ a��a

��A   ������ ��a

a�� a ������ a�� a

��1   ������ ��1

a��1  ������ a��1

����   ������ ����

```



## language-sensitive case mapping



* [ ] Lithuanian rules

* [x] Turkish and Azeri



```ruby

"I".downcase # => "i"

"I".downcase(:turkic) # => "��"

"I\u0307".upcase # => "I��"

"I\u0307".upcase(:lithuanian) # => "I��" instead of "I"

```



* [ ] using some standard locale / language codes



Also, it's true that for now there are only a few language-sensitive rules (for Lithuanian, Turkish and Azeri) but why: 



- adding a `:turkic` symbol and not a `:azeri`?

- using full english arbitrary (why `turkic` and not `turkish`?) language name rather than some [ICU locale IDs](https://unicode-org.github.io/icu/userguide/locale/)?

  - Language code ISO-639 standard

  - Script code Unicode ISO 15924 Registry

  - country code ISO-3166 standard



So I would rather see something like that



```ruby

"placeholder".upcase(locale: :tr_TR)

"placeholder".upcase(lang: :tr)

```











-- 

https://bugs.ruby-lang.org/

 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/