[ruby-core:81389] [Ruby trunk Bug#13595] rb_alloc_tmp_buffer2 broken when: elsize % sizeof(VALUE) == 0

Issue #13595 has been updated by nobu (Nobuyoshi Nakada).

Description updated

Maybe like this?

```diff
diff --git a/include/ruby/ruby.h b/include/ruby/ruby.h
index 0b277dce19..95dab1bf3f 100644
--- a/include/ruby/ruby.h
+++ b/include/ruby/ruby.h
@@ -1612,9 +1612,10 @@ rb_alloc_tmp_buffer2(volatile VALUE *store, long count, size_t elsize)
 {
     size_t cnt = (size_t)count;
     if (elsize % sizeof(VALUE) == 0) {
-	if (RB_UNLIKELY(cnt > LONG_MAX / sizeof(VALUE))) {
+	if (RB_UNLIKELY(cnt > LONG_MAX / elsize)) {
 	    ruby_malloc_size_overflow(cnt, elsize);
 	}
+	cnt *= elsize / sizeof(VALUE);
     }
     else {
 	size_t size, max = LONG_MAX - sizeof(VALUE) + 1;
```

----------------------------------------
Bug #13595: rb_alloc_tmp_buffer2 broken when: elsize % sizeof(VALUE) == 0
https://bugs.ruby-lang.org/issues/13595#change-65098

* Author: normalperson (Eric Wong)
* Status: Open
* Priority: Normal
* Assignee: naruse (Yui NARUSE)
* Target version: 
* ruby -v: 
* Backport: 2.2: UNKNOWN, 2.3: UNKNOWN, 2.4: REQUIRED
----------------------------------------
Here is the function in full as of current trunk (r58863):

~~~c
static inline void *
rb_alloc_tmp_buffer2(volatile VALUE *store, long count, size_t elsize)
{
    size_t cnt = (size_t)count;
    if (elsize % sizeof(VALUE) == 0) {
	if (RB_UNLIKELY(cnt > LONG_MAX / sizeof(VALUE))) {
	    ruby_malloc_size_overflow(cnt, elsize);
	}
    }
    else {
	size_t size, max = LONG_MAX - sizeof(VALUE) + 1;
	if (RB_UNLIKELY(rb_mul_size_overflow(cnt, elsize, max, &size))) {
	    ruby_malloc_size_overflow(cnt, elsize);
	}
	cnt = (size + sizeof(VALUE) - 1) / sizeof(VALUE);
    }
    return rb_alloc_tmp_buffer_with_count(store, cnt * sizeof(VALUE), cnt);
}
~~~
Notice that elsize is completely ignored in the top branch when
"`(elsize % sizeof(VALUE) == 0)`" is true; this gives me problems
when attempting to use `ALLOCV_N`.

I am terrible at arithmetic and this function is too complicated for me,
so I'll let naruse or someone else fix this.  But please do.  Thanks




-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>