From: "Mrlaura (QUANG NGUYEN) via ruby-core" <ruby-core@...>
Date: 2026-10-12T06:56:01+00:00
Subject: [ruby-core:127078] [Ruby Feature#12831] /\X/ (extended grapheme cluster) can't pass unicode.org's GraphemeBreakTest

Issue #12831 has been updated by Mrlaura (QUANG NGUYEN).


Thanks to everyone who investigated and fixed this grapheme segmentation issue.

One related detail worth highlighting is that correct grapheme segmentation doesn't automatically guarantee Unicode-equivalent string comparisons.

For example, "�" (U+00E9) and "e" + U+0301 are both single grapheme clusters, but their underlying code points differ.

For text matching or Levenshtein distance over user-perceived characters, it's often useful to normalize both strings to NFC before splitting them into grapheme clusters.

It's also important to use a GraphemeBreakTest version matching the Unicode rules implemented by the Ruby version being tested.

A related technical discussion is available here:

https://www.levenshtein.net/unicode-and-grapheme-safe-levenshtein

For transparency, I contribute to that reference.

Thanks again for improving Ruby's Unicode support!

----------------------------------------
Feature #12831: /\X/ (extended grapheme cluster) can't pass unicode.org's GraphemeBreakTest
https://bugs.ruby-lang.org/issues/12831#change-119477

* Author: mtsmfm (Fumiaki Matsushima)
* Status: Closed
* Assignee: naruse (Yui NARUSE)
----------------------------------------
I'm trying to replace Rails's grapheme implementation (http://api.rubyonrails.org/classes/ActiveSupport/Multibyte/Unicode.html#method-i-unpack_graphemes) with Ruby's extended grapheme cluster (/X/).
https://github.com/rails/rails/issues/26743

I noticed that Ruby's grapheme cluster can't pass unicode.org's GraphemeBreakTest.

Following test script will fail on Ruby 2.2/2.3

~~~ ruby
require 'rubygems'
require 'open-uri'
require 'test/unit'

UNICODE_VERSION =
  if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("2.3.0")
    "8.0.0"
  else
    "7.0.0"
  end

class TestGrapheme < Test::Unit::TestCase
  # https://github.com/rails/rails/blob/v5.0.0.1/activesupport/test/multibyte_grapheme_break_conformance_test.rb#L37
  def test_breaks
    each_line_of_break_tests do |*cols|
      *clusters, comment = *cols
      string = clusters.map {|c| c.pack("U*") }.join
      assert_equal clusters, string.scan(/\X/).map(&:codepoints), comment
    end
  end

  def each_line_of_break_tests(&block)
    lines = 0
    max_test_lines = 0 # Don't limit below 21, because that's the header of the testfile
    URI.parse("http://www.unicode.org/Public/#{UNICODE_VERSION}/ucd/auxiliary/GraphemeBreakTest.txt").open do |f|
      until f.eof? || (max_test_lines > 21 && lines > max_test_lines)
        lines += 1
        line = f.gets.chomp!
        next if line.empty? || line.start_with?("#")

        cols, comment = line.split("#")
        # Cluster breaks are represented by �
        clusters = cols.split("�").map { |e| e.strip }.reject { |e| e.empty? }
        clusters = clusters.map do |cluster|
          # Codepoints within each cluster are separated by �
          codepoints = cluster.split("�").map { |e| e.strip }.reject { |e| e.empty? }
          # codepoints are in hex in the test suite, pack wants them as integers
          codepoints.map { |codepoint| codepoint.to_i(16) }
        end

        # The tests contain a solitary U+D800 <Non Private Use High
        # Surrogate, First> character, which Ruby does not allow to stand
        # alone in a UTF-8 string. So we'll just skip it.
        next if clusters.flatten.include?(0xd800)

        clusters << comment.strip

        yield(*clusters)
      end
    end
  end
end
~~~

https://gist.github.com/mtsmfm/38f46882c3d4ccde35c269594fc24ebc

I found an issue on Onigmo (https://github.com/k-takata/Onigmo/issues/46)
but I couldn't on bugs.ruby-lang.org so I created this ticket.

I'm unfamiliar with grapheme so please tell me if I get something wrong.



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/