From: "RubyTalk@..." <rubytalk@...>
Date: 2006-12-18T02:06:06+09:00
Subject: Re: Grabbing data off a webpage

Its slow and messy, but i did it in 5 mins


require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'uri'
require 'pp'


module Hpricot
  module Traverse

    # Returns the node neighboring this node to the south: just below it.
    # This method includes text nodes and comments and such.
    def next_node(loop=1)
      sib = parent.children
      sib[sib.index(self) + loop] if parent
    end
  end
end





 class HTMLpage
  #We save some things for a single load of the page
  #and just because
  def initialize()

    #html of the whole page. only get this once
    @page_html=nil
    load_page
  end


  ##Complete html of page
  ##
  def page_html
    load_page.to_html
  end
  def row(location=0)
      doc=load_page
      return doc.search("tbody").collect{|x| x.search("tr")[location] }.compact
  end
  def to_struct()
      doc=load_page
      struct=[]
      doc.search("tbody").each{|x|
      arr=[]
      x.search("td").each{|xx|
              arr.push(xx.inner_html)
        }
      (0 .. arr.size/5).each{|index|
        struct.push(Thing.new(arr[(index*5)],arr[(index*5)+1],arr[(index*5)+2],arr[(index*5)+3],arr[(index*5)+4]))
      }

      }
      return struct
  end

    private

    #loads the page data
    def load_page
      #check if we have page html if so return
      if @page_html
        doc=Hpricot(@page_html)
      else
          doc=Hpricot(open('http://yorkcountyschools.org/mves/arlist/3-3.4.htm'))
          @page_html=doc.to_html
      end
      return doc
    end


  end
  class Thing
    attr_reader :quiz_id,:title,:author,:booklevel,:points
    def initialize(quizID,title,author,bookLevel,points)
            @quiz_id=quizID
            @title=title
            @author=author
            @booklevel=bookLevel
            @points=points
    end
  end


 page=HTMLpage.new

 stuff=page.to_struct
 pp stuff[0].title
 pp stuff[0].author

On 12/17/06, Bil Kleb <Bil.Kleb@nasa.gov> wrote:
> OK, so I haven't done this in years.
>
> What's the "modern" way of grabbing the data off
> a webpage, e.g.,
>
>   http://yorkcountyschools.org/mves/arlist/3-3.4.htm
>
> My initial attempt has been focused on Hpricot,
>
>   require 'rubygems'
>   require 'open-uri'
>   require 'hpricot'
>   doc = Hpricot(open('http://yorkcountyschools.org/mves/arlist/3-3.4.htm'))
>
> and I can find doc/"th" and doc/"tr", but what's
> the best way to cram them into an array of structs
> or something?
>
> Thanks,
> --
> Bil Kleb
> http://funit.rubyforge.org
>
>