From: "Martin J. Dürst" Date: 2018-10-16T12:38:02+00:00 Subject: [ruby-core:89418] Re: Fwd: [ruby-cvs:72191] duerst:r65091 (trunk): update to Unicode 11.0.0 (basic step, not complete yet) Hello Nobu, Many thanks for looking into this. Your commits make the segmentation fault go away, that's great! I can reproduce the problem here, in a simple one-liner that doesn't even rely on the \X / grapheme_clusters logic: ./ruby -e 'puts "abc" =~ /\p{Grapheme_Cluster_Break=E_Modifier}/' Before upgrading to Unicode 11.0.0, this works without errors. After upgrading, it produces the following error: -e:1: invalid character property name {Grapheme_Cluster_Break=E_Modifier}: /\p{Grapheme_Cluster_Break=E_Modifier}/ Looking for all the appearances of "Extend" in enc/unicode/1[01].0.0/name2ctype.h, I have found only the following differences: >>>>>>>> 11c11,12 < #define CR_Grapheme_Cluster_Break_Extend CR_Grapheme_Extend --- > static const OnigCodePoint CR_Grapheme_Cluster_Break_Extend[] = { > }; /* CR_Grapheme_Cluster_Break_Extend */ 29a31,33 > /* 'In_Georgian_Extended': Block */ > static const OnigCodePoint CR_In_Georgian_Extended[] = { > }; /* CR_In_Georgian_Extended */ 90a95 > CR_In_Georgian_Extended, >>>>>>>> The Georgian stuff is new, a new block of characters for an already existing script is often called _Extended. The first few changes may be more relevant. As far as I understand, character properties are preprocessed by removing spaces/underscores and lowercasing everything. The problem seems to be related to the following text at http://www.unicode.org/versions/Unicode11.0.0/: >>>>>>>> Segmentation-related Changes Four Grapheme_Cluster_Break and Word_Break classes have become obsolete and are no longer used: E_Base, E_Modifier, Glue_After_Zwj, and E_Base_GAZ. Those values are still part of the enumeration of the property values, because stability constraints prevent removal of enumerated property values, even if obsolete; however, these are no longer assigned to any characters, and are no longer referred to explicitly by any rules in the algorithms. >>>>>>>> That suggests that we may have some kinds of problems with properties that exist but don't have any characters assigned. Also, the current problem is related to the fact that I'm trying to do the update from Unicode 10.0.0 to Unicode 11.0.0 in incremental steps; it is possible that there are some dependencies that make that impossible, but I'll try to find a way maybe to temporarily 'cheat' a bit to allow upgrading in steps. I'll investigate more, but if you find something, that would also be great. Regards, Martin. On 2018/10/16 18:16, ������������ wrote: > > Hi, > > On 2018/10/16 16:31, Martin J. D��rst wrote: >> The commit below fails on travis-ci >> (see https://travis-ci.org/ruby/ruby/jobs/442027489). >> >> It produces a Ruby segmentation fault when running the tests in >> test/lib/minitest/unit.rb. I have tried to look at that file and find >> any kinds of dependencies to Unicode 11.0.0, but I haven't found any. >> >> I have also tried to run that file independently, but >> ./ruby test/runner.rb test/lib/minitest/unit.rb >> just says >> Finished tests in 0.006213s, 0.0000 tests/s, 0.0000 assertions/s. >> 0 tests, 0 assertions, 0 failures, 0 errors, 0 skips >> >> Any help is appreciated, thanks very much in advance. > > Added missing checks and error details at r65094-65096. > Seems some property names gone. > > ``` > $ ./ruby -e '"\u{80}"[/\X/]' > Traceback (most recent call last): > ������ 1: from -e:1:in `
' > -e:1:in `[]': invalid character property name > {Grapheme_Cluster_Break=E_Modifier}: /\X/ (RegexpError) > ``` > ``` > $ ./ruby -e '"\u{80}".grapheme_clusters' > Traceback (most recent call last): > ������ 1: from -e:1:in `
' > -e:1:in `grapheme_clusters': cannot compile grapheme cluster regexp: > invalid character property name {Grapheme_Cluster_Break=E_Modifier} (fatal) > ``` Unsubscribe: