From: Paul Lutus Date: 2006-11-21T02:35:05+09:00 Subject: Re: Ruby screen scraping James Edward Gray II wrote: > On Nov 20, 2006, at 2:45 AM, Paul Lutus wrote: > >> In this thread, the OP started out by examining the alternatives among >> specialized libraries meant to address the general problem, but >> apparently >> never considered writing code to solve the problem directly. > > Starting out by looking for a library that does the hard work for you > is a good first step, I would say. Do we really want to be > discouraging that? IMHO yes, when it doesn't solve the problem at hand. This is obviously a matter of personal taste, but I always try coding a solution first, or at least endeavor to understand what such a solution would entail, before going shopping for a library. It's based on KISS, and large libraries that can be relied on to solve any problem except the problem the adopter faces, fail the KISS principle. / ... > I've seen valid XHTML that wouldn't be much fun to parse. You still > need to worry about whitespace, namespaces, the kind of quoting used, > CDATA sections, ... These are all relatively easy to parse. Even the CDATA sections are clearly and consistently delimited, so can be reliably skipped over and encapsulated. That was the design goal of XHTML -- to be easy to parse, to be consistent -- assuming the syntax is followed. I just converted my 500-page Web site to XHTML, and at the end of the project I found that I could parse any page on the site using a very simple parser. This was about the time the pages also began passing XHTML validation tests. But this is all by the way. The point is the OP adopted a powerful library, only to discover he still couldn't solve his original problem, and if we assume (as I did) that the pages are machine-generated and meet reasonable syntax standards, the one-line solution I posted will meet his requirements. Your earlier point, that a Web page picked at random might be virtually unparseable, is certainly true, and solutions like we are discussing assume a high degree of cooperation between the page generator and the parser. -- Paul Lutus http://www.arachnoid.com