From: Josh Cheek <josh.cheek@...>
Date: 2010-11-25T11:48:54+09:00
Subject: Re: Dynamically creating webpages via Ruby

--0016364273ad642ecf0495d7a5f7
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Nov 24, 2010 at 4:39 PM, Phil H. <henleyphil@yahoo.com> wrote:

> I'm just getting started with Ruby and have very little programming
> background, so hopefully this is a simple problem to solve.
>
> I'm writing a program that takes a number of url's from all ready
> existing webpages as an input (the number of these URL's can vary),
> extracts some data from the these pages (such as links to pictures,
> google maps data, etc...) then builds 2 new webpages based off this
> extracted data.  One page is the index with a list of links to the
> second page which contains all my extracted data.
>
> The way it works now is basically in one big loop.  So my input URL's
> are stored in an array, I pop the last element out of the array, send it
> to the loop to extract data and generate HTML and so on until my array
> of input URL's is empty.
>
> During this loop I am only generating HTML for the Body of the webpages.
> The beginning of each webpage is written prior to starting the loop and
> the end of the webpages are written after the loop finishes.
>
> This was working because the pre-body and post-body parts of my webpages
> didn't need any of the extracted data from my input URL's - it was just
> really basic HTML code.  And the body of the webpages ONLY needed the
> extracted data.  This isn't the case anymore.
>
> Long story short, I need to separate the data extraction from the web
> page building and this is turning out to be harder than I thought since
> the number of input URL's can change at any time.
>
> What I want to do is extract all the data for all the input URL's first
> and store that info somehow.  Then use a webpage building method of some
> kind to reference the extracted data and generate my webpages.  That
> seems simple enough, but because my list of input URL's is always
> changing I'm not sure how to dynamically create the number of objects I
> need to store the extracted info.
>
> Any tips?  Thanks in advance.
>
> --
> Posted via http://www.ruby-forum.com/.
>
>
Hi, I don't really see why the number of URLs changing is causing you
problems. This should be in a loop, as you said, and a loop will iterate
over all of them regardless of their size.

I can understand the appeal of separating extraction of data from building
of data, but in this case, I think that storing it in an intermediate form
is unnecessary. I would suggest simply doing these steps one after the
other. First extract all data, then build the page. Then you don't need to
save it in a file and go run a second script to read it in and do stuff with
it.

I don't know what you are trying to do with this data, but here is an
example https://gist.github.com/714817 It iterates over an array of URLs,
opens those pages, pulls all the links out of them, then builds an html
document where each page is displayed in a paragraph with a link to the page
followed by an unordered list of all the links that page contains. No
storing in files necessary.

--0016364273ad642ecf0495d7a5f7--