From: brabuhr@...
Date: 2010-08-10T23:52:40+09:00
Subject: Re: NEWBIE: Ruby & XML

On Tue, Aug 10, 2010 at 9:53 AM, Jerome David Sallinger
<imran.nazir@yahoo.co.uk> wrote:
> Hello.
>
> I am working with some XML logs coming from a network simulator.
> My aim is to strip out the transient information concerning any given
> variable.
>
> For example here is some example data:
>[...]
>
> For example I may want to strip out all the "CQI" and timing values to
> get:
>
> 23, 00:00:40.450
> 22, 00:00:40.460
> 22, 00:00:40.460
> 23, 00:00:40.460
> 22, 00:00:40.460
> 24, 00:00:40.460
> 21, 00:00:40.470
> 22, 00:00:40.470
>
> Question: These files can be very large and keeping the computer
> resource overhead is important. I've looked at other threads on this
> forum to decide which method of extracting data against timestamps would
> be the quickest but the information has been conflicting.

I make no claim about what might be best :) but, nokogiri seems to be
the leading Ruby XML library at the moment.  I quickly adapted an old
REXML pull parser to work with your sample data:

def parse(stream)
  raise "BlockRequired" unless block_given?

  parser = REXML::Parsers::PullParser.new(stream)

  row = {}

  while parser.has_next?
    event = parser.pull

    case event.event_type
    when :start_element
      case event[0]
      when 'primitive'
        row = event[1]; col = nil
      when 'parameter'
        col = event[1]["name"]
      end

      row[col] ||= "" if col

    when :end_element
      col = nil

      case event[0]
      when 'primitive'
        yield(row)
      else
        # ignore
      end

    when :text
      row[col] << event[0].chomp if col

    else
      #ignore
    end
  end
end

parse(string){|row|
  #p row
  puts "#{row["CQI"]}, #{row["time"]}"
}

> ruby x.rb
23, 00:00:40.450
22, 00:00:40.460
22, 00:00:40.460
23, 00:00:40.460
22, 00:00:40.460
24, 00:00:40.460
21, 00:00:40.470
22, 00:00:40.470

The original program I lifted that from was processing XML files up to
several gigabytes; particularly on the largest files we saw much
better performance running under JRuby over MRI (1.8.5 or so).