Re: Ruby screen scraping

James Edward Gray II wrote:

> On Nov 20, 2006, at 11:35 AM, Paul Lutus wrote:
> 
>> James Edward Gray II wrote:
>>
>>> I've seen valid XHTML that wouldn't be much fun to parse.  You still
>>> need to worry about whitespace, namespaces, the kind of quoting used,
>>> CDATA sections, ...
>>
>> These are all relatively easy to parse. Even the CDATA sections are
>> clearly
>> and consistently delimited, so can be reliably skipped over and
>> encapsulated. That was the design goal of XHTML -- to be easy to
>> parse, to
>> be consistent -- assuming the syntax is followed.
> 
> But if you use an already developed parser, you gain all their work
> on edge cases, all their testing efforts, all their optimization
> work, etc.

Yes, all to the good, if the feature set is needed and if the target
environment can support the library. And if the library actually solves the
original problem.

> I see what you are saying about knowing you can count on the data,
> but your messages are filled with a lot "as long as you are sure"
> conditions.  Dropping a bunch of those conditions is just one more
> advantage to using a library.

Yes, unless the library serves no purpose and occupies memory and machine
cycles better spent elsewhere. Without a library, you have to work out the
problem directly. With a library, you have to work out the problems caused
by the library.

My personal favorite for this dichotomy is REXML, which apparently can do
anything, unless you have something specific in mind, then IMHO you are
better off writing your own code to parse XML data sets. It isn't as though
XML is a dark and mysterious world that is beyond the reasoning powers of
mere mortals. If it were, the designers of the scheme were wasting their
time.

In the beginning, we had all sorts of weak and limited dataset protocols.
These weaknesses are well addressed by XML, but some think XML is too
complicated to manipulate directly. So libraries like REXML get created.
But the libraries often turn out to be so complex and difficult to put into
service that in some cases one is better off writing one's own
generator/parser for the simpler applications of XML.

The complexity referenced above seems to arise from an irresistible tendency
to put every feature into a library, with the side effect that important
and trivial/esoteric features often get mixed up together in the
documentation and the interface, and the library ends up too large to
justify for simple processing tasks.

Maybe now someone will write a library to bring REXML under control. Ad
infinitum.

> You say you are always surprised when people build up all this hefty
> library code when a simple regex will do,

No, not always, those are not my words. But in a case like this, where the
library accomplishes everything except what the OP actually wanted, yes.
Please note that I only made this plain-code argument after the OP
explained that he had put the library in place, had run it through its
paces, only to discover that he still couldn't solve the original problem.

> but I'm always shocked when 
> I can replace hundreds of lines of code by loading and making use of
> a library.  If we have to err on one side of that, I would prefer it
> be on the library using side.

For myself, I prefer to know what is going on. As I said, it's just a
personal preference.

> That said, I guess we'll just have to agree to disagree.  That's for
> the intelligent and civil debate.

You're welcome (if I read you correctly). Such an exchange is always
possible, some might say likely, between two people who both want it that
way.

An aside with some small relevance. It's just possible that the Linux kernel
maintainers' tendency to adopt existing libraries over laboriously writing
fresh code will spawn a huge legal battle with Microsoft, who clearly
intend to argue (and who are now arguing) that it is their intellectual
property embedded in Linux, and therefore all those Linux users are
actually Microsoft customers.

I can see how this post may be interpreted, so I want to say I hope no one
is misled. If I were really intent on avoiding libraries, I would write
everyting in assembly. My disdain for libraries is fully constrained by
reality and pragmatism, and there are plenty of libraries that I use with
something that approaches reckless abandon.

But ... when a specialized library can't solve a problem that is soluble
with one line of tautological Ruby code, I'm more than willing to speak up.

-- 
Paul Lutus
http://www.arachnoid.com