From: Ruby Quiz <james@...>
Date: 2006-03-02T22:57:36+09:00
Subject: [SUMMARY] Current Temperature (#68)

Obviously, this quiz is about fetching data from the Internet.  People went
about this in two different ways.  The most popular solution was to "scrape" the
answer from popular Internet weather sites.  Here is one such solution from
David Tran:

	require 'uri'
	require 'open-uri'
	require 'rexml/document'
	
	class Weather
	  attr_reader :location, :temperature, :unit
	
	  def initialize(zip_or_city, unit='f')
	    raise "Error: Unit must be 'C' or 'F'." unless unit =~ /^[cf]$/i
	    id = get_id(zip_or_city)
	    url = "http://xml.weather.yahoo.com/" +
	          "forecastrss/#{id}_#{unit.downcase}.xml"
	    xml = open(url) { |f| f.read }
	    doc = REXML::Document.new(xml)
	    @temperature = doc.elements['/rss/channel/item/yweather:condition/@temp'].
	                       to_s.to_i
	    @unit = unit.upcase
	  end
	
	  private
	  def get_id(location)
	    location = URI.escape(location)
	    url = "http://xoap.weather.com/search/search?where=#{location}"
	    xml = open(url) { |f| f.read }
	    doc = REXML::Document.new(xml)
	    locations = doc.elements.to_a("/search/loc")
	    raise "Cannot find the location." if locations.size <= 0
	    @location = locations[0].text.sub(/\s*\(\d+\)\s*$/, '')
	    locations[0].attributes['id']
	  end
	end
	
	if __FILE__ == $0
	  if ARGV.size <= 0 || (ARGV[1] && ARGV[1] !~ /^[cf]$/i)
	    puts "Usage:  #$0  city_or_zip_code  [c|f]"
	    exit(1)
	  end
	
	  begin
	    w = Weather.new(ARGV[0], ARGV[1] || 'f')
	    puts "The temperature in #{w.location} is " +
	         "#{w.temperature} degress #{w.unit}."
	  rescue
	    puts "Information for #{ARGV[0]} was not found or unavailable."
	  end
	end

Start at the bottom if statement.  You can see in here that arguments are
checked and a usage statement is printed, if needed.  Next we can see that a
Weather object is constructed and the temperature information is pulled from its
methods.  If anything goes wrong in this process, the rescue clause prints a
suitable error message.

Now we need to check out the Weather object.

In initialize(), the zip_or_city is somehow turned into an "id".  Using that id,
a url is constructed for an RSS feed of weather information for the location. 
Next the terrific open-uri library is used to slurp the feed, just as we would a
normal file.  The xml is then dropped into Ruby's standard REXML library and an
XPath statement is used to extract the temperature from the feed.

The other method is the magic "id" conversion I mentioned earlier.  It really
works pretty similar to the method we just examined.  Again a url is
constructed, the page slurped, the xml handed to REXML, and XPath used to find a
matching location that was listed on the page.  This time a Regexp is also
needed to clean up the location name.

Unfortunately, scraping has some down sides.  First, this use of REXML and
Regexps to grab the information is pretty fragile.  As soon as these pages
change in some way the author didn't expect, the application is broken and will
need maintenance.

The other big problem with scraping is legal issues.  Many sites do not allow
access to their content like this (because it bypasses their ads).  Google is a
famous example of a company that does not allow scraping content.  Be sure you
check the usage policies of a site before you construct or use software like
this.

If you want to get around these issues, you can use a web service.  Web services
represent a predefined communication protocol.  You pass the information the
service expects in, and it will return a promised response.  Here's a
temperature solution using a web service by Rudolfs Osins:

	#!/usr/local/bin/ruby
	require 'soap/wsdlDriver'
	require 'rexml/document'
	
	URL = 'http://www.webservicex.net/globalweather.asmx?WSDL'
	
	# process the comandline arguments
	if ARGV[0] == nil
	  abort("Usage: weather.rb city")
	else
	  city = ARGV.join(' ')
	end
	
	soap = SOAP::WSDLDriverFactory.new(URL).create_rpc_driver
	begin
	  weather = soap.GetWeather(:CityName => city, :CountryName => "")
	
	  # strip the first line with <? ?> stuff, else REXML wont parse
	  xml = weather.getWeatherResult.gsub(/(<\?.*?>\n)/, '')
	  data = REXML::Document.new(xml)
	
	  # celsius degrees are in parentheses
	  data.elements["//Temperature"].text[/\((.*)\)/]; temp = $1
	  data.elements["//Location"].text[/^(.*),/]; loc = $1
	
	  # show the gathered data
	  puts "The temperature in " + loc + " is " + temp
	rescue
	  puts "Could not find data for your supplied city: " + city
	end

Here again, we see that a URL is built, but this time the URL points to the
document describing the web service (a Web Service Description Language, or
WSDL, document).  Another standard library, SOAP, is used to read and parse that
document.  In doing so, it will build a custom object that has the methods
provided by the service, accepting the arguments they expect.

The rest of the solution will look pretty familiar, since this service provides
an xml answer.  Again REXML is used, with XPath, to find the data we are
interested in and again Regexps are used to clean it up for display.  In this
case we didn't get too far away from the fragile scraping technique it seems,
though web services typically see less change than web pages.

Not that the above solution doesn't work with zip codes, because the service
expects a city name.  Supporting a zip code might be possible using another
service to lookup the city name, if one could be found.

My thanks to all the submitters for another great quiz.  Pulling information
from the web is pretty common and these guys did a great job of showing off how
painless it can be.

Tomorrow's quiz is an easy, but unique, problem.  If you have yet to complete a
quiz solution, this one is for you...