From: John Joyce <dangerwillrobinsondanger@...>
Date: 2007-08-15T08:26:47+09:00
Subject: Re: How to parse the "next" button on yahoo resultpages?

>
> Ps: I read about hpricot and scrubbyt but wanna solve this without
> external modules.
>
> bye
> --  
> Posted via http://www.ruby-forum.com/.
>

At some point you simply have to look at the html generated by Yahoo  
or whatever site you're scraping and determine what to do.
Even if you want to do it without using 3rd party gems or modules,  
you might find some good ideas to borrow from those!
In the words of Pablo Picasso, "A good artist knows when to  
copy" (paraphrased)

Web scraping is a lot of work to develop. You're more or less  
creating some of the functionality of what a web browser does, but  
tougher because you're looking for and expecting specific content,  
and it is tougher because a site can change its formatting any time.  
There may be easier search APIs to use than scraping. Check google  
and yahoo carefully for developer tools/APIs that allow accessing  
their search engines in a simpler/different format.