From: Shashank Date <ADATE@...>
Date: 2002-06-26T10:13:26+09:00
Subject: Re: Q: download html file and parse it

I had to modify the html-parser a little to get it to work.
For example:
--------------------------------------
#! /usr/local/bin/ruby
require "net/http"
require "html-parser"
require "formatter"

def htmltest(data)
  w = DumbWriter.new
  f = AbstractFormatter.new(w)
  p = HTMLParser.new(f)
  p.feed(data)
  p.close
end

domain = 'www.rubycentral.com'
file   = '/book/rubyworld.html'

h = Net::HTTP.new(domain, 80)
resp, data = h.get(file, nil )
puts domain + file if $DEBUG

htmltest(data)
-------------------------------------

This program generated the following error:

c:/ruby/lib/ruby/site_ruby/html-parser.rb:409:in `Integer': invalid value
for Integer: ""1"" (ArgumentError)
        from c:/ruby/lib/ruby/site_ruby/html-parser.rb:409:in `do_img'
        from c:/ruby/lib/ruby/site_ruby/html-parser.rb:395:in `each'
        from c:/ruby/lib/ruby/site_ruby/html-parser.rb:395:in `do_img'
        from c:/ruby/lib/ruby/site_ruby/sgml-parser.rb:281:in `send'
        from c:/ruby/lib/ruby/site_ruby/sgml-parser.rb:281:in
`handle_starttag'
        from c:/ruby/lib/ruby/site_ruby/sgml-parser.rb:233:in
`finish_starttag'
        from c:/ruby/lib/ruby/site_ruby/sgml-parser.rb:208:in
`parse_starttag'
        from c:/ruby/lib/ruby/site_ruby/sgml-parser.rb:89:in `goahead'
        from c:/ruby/lib/ruby/site_ruby/sgml-parser.rb:58:in `feed'
        from htmltest00.rb:10:in `htmltest'
        from htmltest00.rb:21

To fix it, I modified  "do_img" method in file (at line 409) html-parser.rb
where it was:

      if attrname == 'width'
        width = Integer(value)
      end
      if attrname == 'height'
        height = Integer(value)
      end

changed to

      if attrname == 'width'
        width = Integer(value.gsub(/[\'\"/,''))  # replace all double-quotes
" and single quotes ' with nothing
      end
      if attrname == 'height'
        height = Integer(value.gsub(/[\'\"/,''))
      end

And then it worked.

I am not sure if this is the best way to do it ;-) but i thought I should
share it with you.

Also, here are some changes I did to the sgml-parser.rb at line 57:

  def feed(data)
    @rawdata << data
    goahead(false)
  end

changed to :

  def feed(data)
    @rawdata << data if data  # make sure that data is not nil
    goahead(false)
  end

HTH,
-- Shanko

"Philip Mak" <pmak@animeglobe.com> wrote in message
news:20020625073957.GR9237@trapezoid.interserver.net...
> On Tue, Jun 25, 2002 at 04:32:50PM +0900, kwatch wrote:
> > ..1) connect http server and get a html file
>
> You can use Net::HTTP. Some documentation for it can be found here:
>
> http://www.rubycentral.com/book/lib_network.html
>
> Search for "Net::HTTP" in that page (it's about halfway down).
>
> > ..2) parse a http file to retrieve infromation from it
>
> Here's a Ruby module that parses HTML. There may be others (look in
> the Ruby Application Archive):
>
> http://www.ruby-lang.org/en/raa-list.rhtml?name=html-parser
>
> > # I want to automate my routine work.
>
> Assuming you're on a UNIX system, make a cron job that periodically
> runs your ruby script.
>