From: Thufir <hawat.thufir@...>
Date: 2009-11-14T12:50:30+09:00
Subject: Re: scrape html gives partial result

On Nov 13, 7:10 pm, brab...@gmail.com wrote:
> On Fri, Nov 13, 2009 at 3:27 AM, Thufir <hawat.thu...@gmail.com> wrote:
> > Why does scrape.rb just result in a few lines of html, rather than an
> > entire document?  The html can be printed in its entirety, but how to
> > persist it?
>
> mysql> describe rss2mysql.items;
>
> http://dizzy.co.uk/ruby_on_rails/cheatsheets/rails-migrations#databas...

Ok, thanks, that problem is solved by using text instead of string.

In terms of design, I'm considering the pros/cons for breaking of the
html to another table.  Perhaps the scraped data should be with the
html in its own table?  I doubt I'd see a performance differential for
the amount of data I'll be working with, but does it matter whether a
relatively large text field, and perhaps about five string fields are
added onto existing table, or whether there's a 1:1 relation to
another table?  Also, perhaps there would be a 1:many relation between
the raw html and scraped data, but I'm not sure about that.

Also, I don't want to accidentally re-fetch the html and end up with a
bunch of 404 error pages, so would it make sense to add a boolean
indicating whether html had been fetched?  Or, just restrict fetching
(?) the html to when feeds are grabbed?


thanks,

Thufir