From: "Marvin Gülker" <redmine@...>
Date: 2010-06-26T01:29:06+09:00
Subject: [ruby-core:30903] [Bug #3482] StringScanner#pos returns wrong character position if used with multibyte chars

Bug #3482: StringScanner#pos returns wrong character position if used with multibyte chars
http://redmine.ruby-lang.org/issues/show/3482

Author: Marvin G��lker
Status: Open, Priority: Normal
Category: M17N, Target version: 1.9.x
ruby -v: ruby 1.9.2dev (2010-05-31 revision 28117) [x86_64-linux]

The StringScanner class from 1.9's stdlib works on bytes rather than on characters. That means, if you want to extract substrings from the original string by use of the return value of StringScanner#pos you get incorrect results: 

irb(main):001:0> require "strscan"
=> true
irb(main):002:0> str = "abc��def��ghi"
=> "abc��def��ghi"
irb(main):003:0> ss = StringScanner.new(str)
=> #<StringScanner 0/13 @ "abc\xC3\xA4...">
irb(main):004:0> ss.scan_until(/��/)
=> "abc��"
irb(main):005:0> ss.pos
=> 5
irb(main):006:0> ss.scan_until(/��/)
=> "def��"
irb(main):007:0> ss.pos
=> 10
irb(main):008:0> 

After the first scan_until I expected the position to be 4, after the second to be 8, which means we finally have an offset of 2 here. 

My Ruby version is ruby 1.9.1p378 (2010-01-10 revision 26273) [x86_64-linux], but I also get the same beaviour with the 1.9.2-preview3 (ruby 1.9.2dev (2010-05-31 revision 28117) [x86_64-linux]).


----------------------------------------
http://redmine.ruby-lang.org