From: Phlip <phlip2005@...>
Date: 2007-07-30T04:34:57+09:00
Subject: Re: hpricot and xpath doesn't work like they should ?!?

anansi wrote:

> Am I using hpricot in the wrong way? I thought it could handle xpaths?

Briefly, I suspect Hpricot uses an XPath subset invented on the fly to 
permit querying into the HTML node space.

(This isn't a bad thing; the alternative, REXML::XPath, cannot handle some 
well-formed XHTML [according to Tidy], and certainly can't handle 
traditional HTML.

(BTW: When I tried to install Hpricot 6 (ruby) on Kubuntu, the require 
'hpricot' refused to find it. This might indicate a broken .so file, so I 
switched to Windows.)

The best way to use XPath is to locate tags by unique id=''. (The page you 
used abuses the IDs, as CLASSes, so it's ill-formed. But that's not your 
problem here.)

Don't use long XPath chains (even if an XPath visualizer provides them), 
because these locate things by incidental features that could change when 
you hit the page again. Table elements could come and go on the fly.

When I installed that XPath Checker (thanks for pointing it out!) and hit 
that page, your XPath selects ZDF, so this implicates Hpricot.

Let's find a workaround. If I want to hit, say, "Hotel Zack und Cody", I use 
Firebug's Inspect Element context menu feature, and see that blurb has a <td 
title="19:45 Hotel Zack und Cody">. So if I XPath for things like that, we 
get:

    //td[ @title ]

That sweeps for every td with a title attribute. (The View XPath feature 
should have an option to find minimal and unique paths based on attributes, 
not long obsessive paths based on indices.)

And that works in Hpricot, too, to select every cell with a title. Further 
poking and parsing should get you the raw TV listings.

  tv.search("//td[ @title ]").each{     |a| p a}

BTW scraping TV guide listings is ... kind'a tacky. Aren't the actual data 
feeds available somewhere?

-- 
  Phlip
  http://www.oreilly.com/catalog/9780596510657/
  "Test Driven Ajax (on Rails)"
  assert_xpath, assert_javascript, & assert_ajax