From: Hugh Sasse <hgs@...>
Date: 2006-11-03T23:54:00+09:00
Subject: Re: Using another program (Lynx) from within Ruby

On Fri, 3 Nov 2006, z wrote:

> I'm trying to write a script to read a list of URLS, get the HTTP response
> headers and <title> (if there is a page there) from each URL, and output to
> a CSV file in this format:
> URL, header, <title>

You could do all that within ruby of course, but it's almost certainly 
quicker like you're doing it....
> 
> I've started with something like this, using Lynx to get the headers.  The
> part that doesn't seem to work is this:
> `lynx -dump -head "#{line}"`  -- it doesn't want to put the url into the
> #{line} within the backticks.

I don't think that's the problem from the code.
> 
> How do you insert a variable from Ruby into the shell command?  I'm ordering
> 4 Ruby books by mail today... I haven't seen anything like this in the ones
> that I've browsed though.
> 
> 
> Here is a larger section of the script:
> 
> print "Enter the location of the input file: "
>     infile = gets.chomp
> 
>     # open file
>     File.open(infile, "r") do |f|
>       # get HTTP headers with Lynx
>       output = f.each_line { |line| `lynx -dump -head "#{line}" |
> grep "HTTP"` }
        # each line won't return anything from the block
=begin ri output
----------------------------------------------------------- IO#each_line
     ios.each(sep_string=$/)      {|line| block }  => ios
     ios.each_line(sep_string=$/) {|line| block }  => ios
------------------------------------------------------------------------
     Executes the block for every line in _ios_, where lines are
     separated by _sep_string_. _ios_ must be opened for reading or an
     +IOError+ will be raised.

        f = File.new("testfile")
        f.each {|line| puts "#{f.lineno}: #{line}" }

     _produces:_

        1: This is line one
        2: This is line two
        3: This is line three
        4: And so on...
=end


        output = []
        f.each_line {|line| output << `lynx -dump -head "#{line}" | grep "HTTP"` }
> #      puts output to CVS file
        # See if you have anything that makes sense...
        $stdout.puts output.join("\n")
> # TODO
> 
> 
        Hugh