From: James Britt Date: 2005-12-03T03:12:12+09:00 Subject: Re: Screen scraping via regex vs. htmltools (vs. REXML) Dan Kohn wrote: > I've finally reimplemented the screen scraper I mentioned on > > using regexes and no external libraries. It is, as Daz suggested, many > times faster than REXML. My question is whether it would be smarter > (faster?, easier to code?) to use htmltools or HTMLTree::Parser > instead. The code in your post seems to use Mechanize. If you are using agent.get to fetch the HTML then you've already parsed the html using htmltools & REXML. You can register callback objects that are invoked when the parsing process encounters matching nodes. Mechanize does this automatically for certain nodes (form stuff, I think), but you can use watch_for_set= {} to define a set of nodes to watch for. This is what I use to construct the product pages for rubystuff.com from the multiple CafePress pages that contain the images, prices, and product description. I tell Mechanize to watch for img, tr, and td elements, and it constructs sets of custom objects of just the parts of the source HTML matching certain criteria. Then I extract the data, create RSS feeds, and turn those into a set of aggregated HTML pages. What I like about this is that the parse process gives me business objects, with (hopefully) self-explanatory behavior. For example, I can ask one of these objects for 'product_id' or 'description'; the object encapsulates the assorted XPath/regex code needed to get that from the source HTML node, making the main part of the app easier to maintain. James Britt -- http://www.ruby-doc.org - Ruby Help & Documentation http://www.artima.com/rubycs/ - Ruby Code & Style: Writers wanted http://www.rubystuff.com - The Ruby Store for Ruby Stuff http://www.jamesbritt.com - Playing with Better Toys http://www.30secondrule.com - Building Better Tools