[ruby-core:80215] [Ruby trunk Bug#13321] String#codepoints for one-byte encodings

Issue #13321 has been updated by duerst (Martin D端rst).


InfraRuby (InfraRuby Vision) wrote:
> Please update the documentation for `String#codepoints` too.

That says "This is a shorthand for `str.each_codepoint.to_a`".

> `String#codepoints` does return (Unicode) codepoints for US-ASCII and ISO-8859-1 as those encodings are the basis of Unicode.

Well, yes, and for almost all encodings, the returned values are Unicode code points for the ASCII characters, and for some other encodings, there is a bit more of overlap. I don't think we need to go too much into details.

> Maybe add `Encoding#unicode_codepoints?` which returns `true` for these encodings: US-ASCII, ISO-8859-1, UTF-8, UTF-16(BE|LE), UTF-32(BE|LE).

There are quite a few other cases where behavior of String methods changes depending on the string's Encoding. I think it would be good to have access to this information, but methods with more general names may be needed.

Anyway, to get Unicode codepoints out of an arbitrary string, `string.encode('UTF-8').codepoints` will always do the job.

> (Also, there's an unrelated change in that revision.)

Yes, thanks for noticing, fixed.

----------------------------------------
Bug #13321: String#codepoints for one-byte encodings
https://bugs.ruby-lang.org/issues/13321#change-63650

* Author: InfraRuby (InfraRuby Vision)
* Status: Rejected
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: 
* Backport: 2.2: UNKNOWN, 2.3: UNKNOWN, 2.4: UNKNOWN
----------------------------------------
On many versions of Ruby, including 2.4.0:

    "\x80".force_encoding("WINDOWS-1252").codepoints.first # => 0x80

I expected 0x20AC: https://en.wikipedia.org/wiki/Windows-1252

See:
  https://github.com/ruby/ruby/blob/v2_4_0/string.c#L7817-L7818
  https://github.com/ruby/ruby/blob/v2_4_0/string.c#L422-L424




-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>