From: jzakiya <jzakiya@...>
Date: 2009-11-13T01:55:07+09:00
Subject: Re: Using Nokogiri

On Nov 10, 10:29 pm, Mark Thomas <m...@thomaszone.com> wrote:
> > OK, when I put Mark's code in a file and ran it (versus entering it in
> > a irb session) it DOES work. However, it doesn't capture the website
> > url, which 7stud's approach does. I haven't figure out how to do it
> > with this approach, and merely adding more items in xpaths doesn't
> > work.
>
> > So Mark, how can your approach be used to capture the url add the end
> > of the data section?
>
> > Here's the file I used with Mark's approach:
>
> > File: scrape1.rb
> > ----------------------------
> > require 'rubygems'
> > require 'open-uri'
> > require 'nokogiri'
>
> > def scrape (id)
>
> >   id = id.to_s
> >   url = "http://www.xyz.org/../../..ID=#{id}"
> >   doc = Nokogiri::HTML.parse(open(url))
>
> >   prefix = '//div[@class="sectionHeaderText"]/following-sibling::'
> >   xpaths = {
> >    :name => "#{prefix}b/text()",
> >    :addr => "#{prefix}text()[2]",
> >    :citystzip => "#{prefix}text()[3]",
> >    :country => "#{prefix}text()[4]",
> >    :phone => "#{prefix}text()[5]",
> >    :web => "#{prefix}text()[6]",
> >    :url => "#{prefix}text()[7]"
>
> You'll need to modify that last line. Unlike the other items, the URL
> is not in a text node, it is the href attribute of the first <a>
> element. So try:
>
>     :url => "#{prefix}a[1]/@href"

Yes, this allows me to capture the url I want (and sometimes ones I
don't want), and I'm able to post-process xpaths to get everything I
need.

  xpaths = {
   :name => "#{prefix}b/text()",
   :addr => "#{prefix}text()[2]",
   :citystzip => "#{prefix}text()[3]",
   :country => "#{prefix}text()[4]",
   :phone => "#{prefix}text()[5]",
   :url => "#{prefix}a[1]/@href"
  }

Now, I just need to understand completely WHY/HOW it works. :-)

Jabari