From: Paul Lutus Date: 2006-11-21T06:40:06+09:00 Subject: Re: Ruby screen scraping James Edward Gray II wrote: > On Nov 20, 2006, at 11:35 AM, Paul Lutus wrote: > >> James Edward Gray II wrote: >> >>> I've seen valid XHTML that wouldn't be much fun to parse. You still >>> need to worry about whitespace, namespaces, the kind of quoting used, >>> CDATA sections, ... >> >> These are all relatively easy to parse. Even the CDATA sections are >> clearly >> and consistently delimited, so can be reliably skipped over and >> encapsulated. That was the design goal of XHTML -- to be easy to >> parse, to >> be consistent -- assuming the syntax is followed. > > But if you use an already developed parser, you gain all their work > on edge cases, all their testing efforts, all their optimization > work, etc. Yes, all to the good, if the feature set is needed and if the target environment can support the library. And if the library actually solves the original problem. > I see what you are saying about knowing you can count on the data, > but your messages are filled with a lot "as long as you are sure" > conditions. Dropping a bunch of those conditions is just one more > advantage to using a library. Yes, unless the library serves no purpose and occupies memory and machine cycles better spent elsewhere. Without a library, you have to work out the problem directly. With a library, you have to work out the problems caused by the library. My personal favorite for this dichotomy is REXML, which apparently can do anything, unless you have something specific in mind, then IMHO you are better off writing your own code to parse XML data sets. It isn't as though XML is a dark and mysterious world that is beyond the reasoning powers of mere mortals. If it were, the designers of the scheme were wasting their time. In the beginning, we had all sorts of weak and limited dataset protocols. These weaknesses are well addressed by XML, but some think XML is too complicated to manipulate directly. So libraries like REXML get created. But the libraries often turn out to be so complex and difficult to put into service that in some cases one is better off writing one's own generator/parser for the simpler applications of XML. The complexity referenced above seems to arise from an irresistible tendency to put every feature into a library, with the side effect that important and trivial/esoteric features often get mixed up together in the documentation and the interface, and the library ends up too large to justify for simple processing tasks. Maybe now someone will write a library to bring REXML under control. Ad infinitum. > You say you are always surprised when people build up all this hefty > library code when a simple regex will do, No, not always, those are not my words. But in a case like this, where the library accomplishes everything except what the OP actually wanted, yes. Please note that I only made this plain-code argument after the OP explained that he had put the library in place, had run it through its paces, only to discover that he still couldn't solve the original problem. > but I'm always shocked when > I can replace hundreds of lines of code by loading and making use of > a library. If we have to err on one side of that, I would prefer it > be on the library using side. For myself, I prefer to know what is going on. As I said, it's just a personal preference. > That said, I guess we'll just have to agree to disagree. That's for > the intelligent and civil debate. You're welcome (if I read you correctly). Such an exchange is always possible, some might say likely, between two people who both want it that way. An aside with some small relevance. It's just possible that the Linux kernel maintainers' tendency to adopt existing libraries over laboriously writing fresh code will spawn a huge legal battle with Microsoft, who clearly intend to argue (and who are now arguing) that it is their intellectual property embedded in Linux, and therefore all those Linux users are actually Microsoft customers. I can see how this post may be interpreted, so I want to say I hope no one is misled. If I were really intent on avoiding libraries, I would write everyting in assembly. My disdain for libraries is fully constrained by reality and pragmatism, and there are plenty of libraries that I use with something that approaches reckless abandon. But ... when a specialized library can't solve a problem that is soluble with one line of tautological Ruby code, I'm more than willing to speak up. -- Paul Lutus http://www.arachnoid.com