From: daz Date: 2005-09-15T13:51:34+09:00 Subject: Re: REXML screen scraping questions Dan Kohn wrote: > My goal here is to take an HTML table and convert it into an array of > arrays, with each inner array representing the 5 columns of cells in a > given row and the outer array representing the whole table. > > I'm using REXML to parse the DOM tree. I would appreciate suggestions > on cleaning up the code below. I've run into the following problems: > > [snip problems] :-) Hiya Dan, I know you've tried other packages but I think REXML isn't what you want for sloppy old HTML work. Below is an "htmltools" implementation using your input. (If you've used Mechanize, html/tree may be installed, already.) If you just want to search the data with regexen, you might make a method which yields the data to its block at the point where it's been collated. (Post back if you need any help with that) #----------------------------------------------------------- require 'html/tree' # http://ruby-htmltools.rubyforge.org/ exa = HTMLTree::Parser.new(verbose=true, false) #exa.parse_file_named('xxx.html') exa.feed(string) # replacing '.parse_file_named' # Pick out all tags under the tag exa.html.children.select {|e0| e0.tag == 'tr'}.each do |tr| # p [:line, __LINE__, tr.to_s] tdn = 0 # Pick out all tags under successive tags tr.select {|e1| e1.tag == 'td'}.each do |td| data = ''; tdn += 1 # Inside the are untagged data or nested tags td.each do |item| if item.data? # p [:data, item.to_s] data << item.to_s else # p [:line, __LINE__, item.tag, item.attributes] end end data.gsub!(/\s+/, ' ') ### yield data # ? (from a method) puts '#--td%02d--> %s' % [tdn, data] end puts '#' << '='*55 end #--td01--> 9-Jan-05 #--td02--> OZ 0204 F Class ICN to LAX #--td03--> 5,968 #--td04--> 2,984 #--td05--> 8,952 #======================================================= #--td01--> 19-Jan-05 #--td02--> MILEAGE PLUS UPGRADE AWARD 15,000 MILES #--td03--> -15,000 #--td04--> #--td05--> -15,000 #======================================================= It's up to you to make the output _and_ the code look pretty. daz