From: Bill Pennington <bill@...>
Date: 2005-07-01T17:24:09+09:00
Subject: Re: spidering a website to build a sitemap

When detecting 404's watch out for the servers that return a 200 code  
with a pretty "Not found" page. Those can throw a real curve ball  
depending on what your are trying to do.


On Jun 30, 2005, at 10:10 AM, Gene Tani wrote:

> I was looking for somehting to trap 404-type errors,  kind of like
> Mertz' code (but in ruby):
>
> http://gnosis.cx/TPiP/069.code
>
> does this sound familiar to anybody?
>
>
>



- Bill