From: "camertron (Cameron Dutro) via ruby-core" <ruby-core@...>
Date: 2026-10-16T17:06:20+00:00
Subject: [ruby-core:127143] [Ruby Bug#21559] Unicode normalization nfd -> nfc -> nfd is not reversible

Issue #21559 has been updated by camertron (Cameron Dutro).


Hi there @duerst ����

I found this ticket after running into a normalization problem while working on an implementation of the Unicode Collation Algorithm. It seems there is at least one other case where the identity `toNFD(x) == toNFD(toNFC(x))` is not maintained by the current Ruby normalization implementation:

```ruby
# ruby 4.1 (master)
str = "\u0FB2\u0591\u0F81\u0061"
p str.unicode_normalize(:nfd) == str.unicode_normalize(:nfc).unicode_normalize(:nfd)
#=> false
```

In this example, the 0F81 decomposes into 0F71 0F80, which have combining classes of 129 and 130 respectively. The 0591 that comes logically earlier in the codepoint sequence has a combining class of 220. Because Ruby's NFD normalization algorithm decomposes subsections of the string rather than the entire string at once, the 0FB2 and 0591 are processed in isolation, which generates a string with combining classes 0 220 129 130 0, which is not the correct canonical order; 0591 (combining class of 220) should be ordered after 0F80 (combining class of 130).

One possible solution is to augment the NFD regex to include characters that decompose into entirely "accent" characters (i.e. characters that all have non-zero combining classes). That way, such characters are decomposed _and_ reordered along with any accent characters that precede them.

I've created a PR demonstrating this solution: https://github.com/ruby/ruby/pull/19393

Curious to hear your thoughts.

----------------------------------------
Bug #21559: Unicode normalization nfd -> nfc -> nfd is not reversible
https://bugs.ruby-lang.org/issues/21559#change-119548

* Author: tompng (tomoya ishida)
* Status: Closed
* Assignee: duerst (Martin D��rst)
* Backport: 3.2: DONTNEED, 3.3: DONTNEED, 3.4: DONTNEED
----------------------------------------
I expect `nfd(nfc(str)) == nfd(str)` but found a string that doesn't.

~~~ruby
# Ruby 3.1 - 3.5
str = "s\u{11930}\u{323}\u{11930}\u{307}"
p str.unicode_normalize(:nfd) == str.unicode_normalize(:nfc).unicode_normalize(:nfd)
#=> false
~~~

~~~ruby
# ruby 3.5.0dev
str = "s\u{1611e}\u{323}\u{1611e}\u{307}\u{1611f}"
p str.unicode_normalize(:nfd) == str.unicode_normalize(:nfc).unicode_normalize(:nfd)
#=> false
~~~




-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/