[ruby-core:39021] [Ruby 1.9 - Bug #5202][Assigned] RubyLex very slow when lexing large files

Issue #5202 has been updated by Yui NARUSE.

Status changed from Open to Assigned
Assignee set to Keiju Ishitsuka


----------------------------------------
Bug #5202: RubyLex very slow when lexing large files
http://redmine.ruby-lang.org/issues/5202

Author: Ryan Melton
Status: Assigned
Priority: Normal
Assignee: Keiju Ishitsuka
Category: 
Target version: 
ruby -v: ruby 1.9.2p290 (2011-07-09) [i386-mingw32]


RubyLex exponentially slower when lexing large files based on the size of the file.  This is because of a call to @readed.reverse inside of the get_readed and ungetc methods.   The following updated versions of these methods greatly improve performance by using rindex instead of reverse.index.

  def get_readed
    if idx = @readed.rindex("\n")
      @base_char_no = @readed.size - (idx + 1)
    else
      @base_char_no += @readed.size
    end
    
    readed = @readed.join("")
    @readed = []
    readed
  end  
  
  def ungetc(c = nil)
    if @here_readed.empty?
      c2 = @readed.pop
    else
      c2 = @here_readed.pop
    end
    c = c2 unless c
    @rests.unshift c #c = 
    @seek -= 1
    if c == "\n"
      @line_no -= 1 
      if idx = @readed.rindex("\n")
	@char_no = idx + 1
      else
	@char_no = @base_char_no + @readed.size
      end
    else
      @char_no -= 1
    end
  end


-- 
http://redmine.ruby-lang.org