[ruby-core:116548] [Ruby master Bug#20203] `TestEnumerable` test failures with GCC 14

Issue #20203 has been updated by alanwu (Alan Wu).


So there are currently 2 issues with using `qsort_r` that I see.

1. As noticed in this issue, if the comparison function uses coroutine/fiber to reenter the middle of `qsort_r`, that results in heap corruption. This happens on older glibc too. Valgrind shows this issue:

```ruby
require 'continuation'
c = nil
o = Object.new
class << o; self; end.class_eval do
  define_method(:<=>) do |x|
    callcc {|c2| c ||= c2 }
    0
  end
end
Array.new(1000, o).sort_by {|x| x }
c.call
```

```
==8321== Invalid read of size 8
==8321==    at 0x484DE5E: memmove (vg_replace_strmem.c:1410)
==8321==    by 0x4E42270: msort_with_tmp (msort.c:44)
==8321==    by 0x4E42270: msort_with_tmp.part.0 (msort.c:53)
<snip>
==8321==    by 0x4E427B5: qsort_r (msort.c:296)
==8321==    by 0x492BFEA: enum_sort_by (enum.c:1293)
==8321==  Address 0x9bb55a0 is 0 bytes inside a block of size 16,000 free'd
==8321==    at 0x484488F: free (vg_replace_malloc.c:985)
==8321==    by 0x4E427C2: qsort_r (msort.c:298)
<snip>
==8321==  Block was alloc'd at
==8321==    at 0x4841828: malloc (vg_replace_malloc.c:442)
==8321==    by 0x4E42641: qsort_r (msort.c:221)
==8321==    by 0x492BFEA: enum_sort_by (enum.c:1293)
```

2. With GC compaction, the GC can update references inside the comparison function. This is undefined behavior:
    > The application shall ensure that the comparison function pointed to by _compar_ does not alter the contents of the array.


    https://pubs.opengroup.org/onlinepubs/9699919799/functions/qsort.html
    In practice, I've seen this causing issue with `GC.auto_compact` in our app's CI, where sorting leaving moved objects in the array.

It seems that to fix these issues we need to stop using `qsort_r` and use our own `ruby_qsort` implementation that doesn't malloc.

----------------------------------------
Bug #20203: `TestEnumerable` test failures with GCC 14
https://bugs.ruby-lang.org/issues/20203#change-106565

* Author: vo.x (Vit Ondruch)
* Status: Open
* Priority: Normal
* ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
There is ongoing mass rebuild in Fedora and that is first time GCC 14 is used and we observe test failures in `TestEnumerable`. Here are a few examples:

~~~
[ 3000/26419] TestEnumerable#test_transient_heap_sort_bymalloc_consolidate(): unaligned fastbin chunk detected
~~~

~~~
[ 2455/26535] TestEnumerable#test_transient_heap_sort_bycorrupted size vs. prev_size in fastbins
~~~

~~~
[ 9716/26532] TestEnumerable#test_any_with_unused_blockdouble free or corruption (fasttop)
~~~

The full logs are accessible [here](https://koji.fedoraproject.org/koji/taskinfo?taskID=112176941). Please drill through `Descendants` and `build.log`



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- [e-mail]
 To unsubscribe send an email to [e-mail]
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/