, <META>, <BODY> for phrases like "not found", "not available", "does not exist" when the HTTP/FTP lib gives you a 200. But at this point I'd settle for responses different from "timed out" or Errno::ECONNREFUSED. I have 2 apps, one is simply validating personal bookmarks, one is commercial. For the commercial one, I'd be happy to register a spider per O'reilly's "Spidering Hacks". For my bookmarks, I figured this would be easy...

From: Gene Tani Date: 2005-07-01T23:15:47+09:00 Subject: Re: spidering a website to build a sitemap Right, the point of Mertz' code is to parse , <META>, <BODY> for phrases like "not found", "not available", "does not exist" when the HTTP/FTP lib gives you a 200. But at this point I'd settle for responses different from "timed out" or Errno::ECONNREFUSED. I have 2 apps, one is simply validating personal bookmarks, one is commercial. For the commercial one, I'd be happy to register a spider per O'reilly's "Spidering Hacks". For my bookmarks, I figured this would be easy...