From: Dan Kohn Date: 2005-09-15T17:56:36+09:00 Subject: Re: REXML screen scraping questions Daz, thank you so much for taking the time to code that. I was also busy today, and got my code working with REXML. Could you please take a look at my code below and share your thoughts on whether you'd still switch to htmltools. The issue is that I'm creating a hundred different screen scrapers for every frequent flyer program. Any scraper is, of course, brittle, but it seemed to me like a DOM/XPath-based technique is both less likely to break from small tweaks to the page and is also generally far more concise to program. The downside, and it may be too big, is that my code is awfully inefficient, and also requires that tidy be run on the HTML before I start. Also, since you're taking a look, could you please tell me if there's any more concise way to initialize my arrays. (Ruby generally seems to figure out variables, but this would only run if I explicitly used Array.new.) require "rexml/document" include REXML string = < 9-Jan-05 OZ 0204 F Class ICN to LAX 5,968 2,984 8,952 19-Jan-05 MILEAGE PLUS UPGRADE AWARD 15,000 MILES -15,000 -15,000 EOF def remove_tag( rexml_array,tag) # Removes tag but leaves the text inside the tag as text inside # the parent of the now removed tag while rexml_array.elements["//#{tag}"] rexml_array.elements["//#{tag}"].replace_with( Text.new( rexml_array.elements["//#{tag}"].text.strip)) end end doc = Document.new( string.gsub!(/\n| /," "), { :compress_whitespace => :all } ) tablearray = Array.new XPath.each( doc,"//tr[count(td)=5]") { |row| rowarray = Array.new rowdom = Document.new( row.to_s) XPath.each( rowdom,"//td") { |cell| remove_tag( cell,"a") rowarray << cell.texts.to_s } tablearray << rowarray } tablearray.each {|el| print el.join(":"),"\n"} Even better is some other scraping I do on the same page, where in each case I only need a one-dimensional array: XPath.each( xml, "//td[@class='t3'][2]") { |cell| summaryarray << cell.texts.to_s } XPath.each( xml, "//td[@colspan='4']/child::*") { |cell| actsumarray << cell.text.to_s } Thanks again, Daz, for taking the time to look at my (first ever Ruby) code. Any other suggestions you could offer would be greatly appreciated. - dan -- Dan Kohn