From: Bob Aman <vacindak@...>
Date: 2006-07-28T22:59:46+09:00
Subject: Re: FeedTools

> Hi Bob,
>
> I was referring to this:
> http://feedvalidator.rubyforge.org/
>
> I'm not sure exactly how to use auto-discovery since I use my own
> crawler for http retrieval.  Can you point me at the method that does
> the auto-discovery?

If FeedTools fails to parse it checks the mime-type of the thing it
retrieved and if it was an (x)html mime-type it looks for the
auto-discovery stuff.  This logic mostly happens in the update!
method, but also partly in the HtmlHelper module.

That said, I'd recommend against using your own crawler for dealing
with feeds unless you're only pulling the feed once.  FeedTools has a
lot of functionality built in that's designed to prevent feeds from
being unnecessarily polled, thus wasting people's bandwidth.  I
suppose you might have actually implemented all that functionality
yourself, but I'm going to go out on a limb and guess that you haven't
added ETag/Last-Modified/If-None-Match/feed time-to-live support into
your crawler.  Just be aware that failing to implement that stuff can
and will cost other people hundreds of dollars in bandwidth if you
poll too often without having that code in place.  It's not exactly
polite.

> Perhaps the http://torrentspy.com/rss.asp case was broken for me since
> my crawler returned a different output than what would have been gotten
> using feedtools internals.

Very possible.

Cheers,
Bob Aman
--
AIM: sporkmonger
Jabber: sporkmonger@jabber.org