From: Benedict Wong <lists@...>
Date: 2012-09-05T09:32:14+09:00
Subject: Re: Parsing Newb Help

7stud -- wrote in post #1074725:
> Benedict Wong wrote in post #1074721:
>> I have successfully scraped the source code, but when it comes to
>> parsing, I'm having trouble inputting the right parameters.
>>
>
> What have you tried?

require 'open-uri'
         BASE_LIST_URL =
 'http://www.yellowpages.com/san-francisco-ca/restaurants?page='

         LAST_PAGE_NUMBER = 157

        LIST_PAGES_SUBDIR = 'yp-list-pages'

         Dir.mkdir(LIST_PAGES_SUBDIR) unless
 File.exists?(LIST_PAGES_SUBDIR)

         for page_number in 1..LAST_PAGE_NUMBER
                 page = open("#{BASE_LIST_URL}#{page_number}")

                file =
 File.open("#{LIST_PAGES_SUBDIR}/yp-list-page-#{page_number}.html",
 'w')

                 file.write(page.readlines)

                 file.close

                 puts "Copied page #{page_number}"

         sleep 4
      end




> This copied all over the web pages onto my hard drive into a .html format.
>

Then, downloaded and installed Nokogiri gem

 Next lines of code:
        require 'rubygems'
         require 'nokogiri'
         require 'open-uri'

         url =
 <view-source:http://www.yellowpages.com/san-francisco-ca/restaurants?p
 age=1>
 'http://www.yellowpages.com/san-francisco-ca/restaurants?page=1'
         page = Nokogiri::HTML(open(url))

         links = page.css('a')

 puts links.length

 (this printed out the number 982)

 Then typed:

 Hrefs = links.map{ |link|

 Link['href'] }

 doc_hrefs = hrefs.select{ |href|
                 href.match('title') != nil
         }
         doc_hrefs = doc_hrefs.uniq


 After this point, I got kind of lost.

-- 
Posted via http://www.ruby-forum.com/.