From: "mame (Yusuke Endoh) via ruby-core" <ruby-core@...>
Date: 2026-10-13T03:14:38+00:00
Subject: [ruby-core:127085] [Ruby Feature#22431] Add `IO#unpack` and `IO#unpack1`

Issue #22431 has been updated by mame (Yusuke Endoh).


I understand the motivation and have wanted this myself, but there seem to be many things to consider. `unpack` has specifiers like `X` and `@` that do not fit streams. We would also need to decide how to handle specifiers with unbounded length, such as `a*`.

Also, the protobuf use case looks risky to me. Reading LEB128 from an untrusted stream without a limit can make us read endlessly and build a huge Integer. Protobuf itself avoids this by imposing its own limit of 10 bytes when reading a varint. But LEB128 itself has no length limit, so it would be unnatural for `unpack` to impose one. So a protobuf implementation would likely need its own limit-aware code anyway, and could not use this method.

----------------------------------------
Feature #22431: Add `IO#unpack` and `IO#unpack1`
https://bugs.ruby-lang.org/issues/22431#change-119480

* Author: ntkme (Natsuki Natsume)
* Status: Open
----------------------------------------
I would like to have ruby add `IO#unpack` and `IO#unpack1` that effectively does both `IO#read(...)` and `String#unpacking(...)` in one call.

This can significantly improve the performance of consuming data in a variable-length encoding format (E.g. LEB128) from a non-peekable `IO` stream (e.g. network/stdio).

In a streaming `IO` with variable-length encoding, today it requires the consumer to scan the stream byte-by-byte to precisely finding the ending of a variable-length segment (E.g. LEB128). If we can have `IO#unpack`/`IO#unpack1`, it will be able to precisely consume a variable-length segment by itself, without needing a two-step solution that requires lots of intermediate ruby object allocation. 

---

Here is a read world example:

LEB128 pack/unpack was recently added by: https://bugs.ruby-lang.org/issues/21785

A very common use case for this is LEB128 delimited payload streaming. In real world it can be a LEB128 delimited protobuf message stream.

For example:

```
length, payload, length, payload, length, payload...
```

In pseudo code, reading and writing such stream would look like:

```ruby
def write(payload, io)
  io.write([payload.length].pack('R'))
  io.write(payload)
end

def read(io)
  length = io.unpack1('R')
            # ^ This method does not exist, which makes the real world implementation slower and more complicated.
            #   See attached benchmark.rb for working implementations.
  io.read(length)
end
```

---

See attached `benchmark.rb`, which demonstrates the limitation of native unpacking for LEB128:

```
ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27]
Warming up --------------------------------------
write varint (pure ruby)    12.167k i/100ms
write leb128 (native)       15.710k i/100ms
Calculating -------------------------------------
write varint (pure ruby)    124.060k (�� 1.6%) i/s    (8.06 ��s/i) -    620.517k in   5.001746s
write leb128 (native)       157.036k (�� 1.1%) i/s    (6.37 ��s/i) -    785.500k in   5.002040s

Comparison:
write leb128 (native)   :   157035.9 i/s
write varint (pure ruby):   124060.1 i/s - 1.27x  slower

ruby 4.1.0dev (2026-10-12T17:01:34Z master 98fd04f76c) +YJIT +PRISM [arm64-darwin27]
Warming up --------------------------------------
read varint (pure ruby)    42.025k i/100ms
read leb128 (native)       20.246k i/100ms
Calculating -------------------------------------
read varint (pure ruby)    423.775k (�� 0.8%) i/s    (2.36 ��s/i) -      2.143M in   5.057579s
read leb128 (native)       201.781k (�� 1.4%) i/s    (4.96 ��s/i) -      1.012M in   5.016818s

Comparison:
read varint (pure ruby):   423774.9 i/s
read leb128 (native)   :   201781.3 i/s - 2.10x  slower
```

On the writing path, LEB128 packing outperforms the pure ruby implementation, which is expected.

However, on the reading path, LEB128 unpacking does not help the performance due to the "variable length" nature of LEB128, which requires finding exact end of the LEB128 before it can be unpacked:

``` ruby
buffer = []
loop do
  byte = io.readbyte
  buffer << byte
  break if byte < 0x80 # this is the last byte of a LEB128 number
end
buffer.pack('C*').unpack1('R')
```

Because of the extra intermediate steps that locate the ending of LEB128 and build a buffer, using LEB128 unpacking in a streaming context ended up to be much slower than a pure ruby implementation that does both scanning and decoding in a single loop:

``` ruby
value = bits = 0
loop do
  byte = io.readbyte
  value |= (byte & 0x7f) << bits
  bits += 7
  break if byte < 0x80
end
value
```

Therefore, if we can have a native ruby method that does scanning and decoding from `IO` in one shot, it would be a huge improvement.

``` ruby
io.unpack1('R')
```

---Files--------------------------------
benchmark.rb (1.64 KB)


-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/