From: Alexander Bokovoy Date: 2002-10-08T18:54:50+09:00 Subject: Re: reading from a URL --LQksG6bCIzRHxTLp Content-Type: text/plain; charset=windows-1251 Content-Disposition: inline On Tue, Oct 08, 2002 at 03:22:07AM +0900, GOTO Kentaro wrote: > At Tue, 8 Oct 2002 02:33:52 +0900, > Volkmann, Mark wrote: > > I need to read the content of an HTTP URL. I can break the URL up into > > server, port and path and use Net::HTTP to read it, but it seems like there > > should be an easier way. I'd really like to pass the URL string to a method > > of some class that would return an object from a class that inherits from > > IO. Does something like that exist? If not, is there an easier way than > > using Net::HTTP? > > If you have Ruby 1.7, try > > require "net/http" > require "uri" > content = Net::HTTP.get_print(URI.parse("http://www.example.com/index.html")) > > If not, > > require "net/http" > require "uri" > uri = URI.parse("http://www.example.com/index.html") > content = Net::HTTP.get(uri.host, uri.request_uri, uri.port) > > Read net/http.rb if you would need http auth, redirection or proxy server. > This library includes the documentation of itself. BTW, current Net::HTTP code does not support proxies with authentication enabled which makes it rather unusable in corporate networks. I've sent a patch to fix it to Matz and he forwarded it to Minero Anoki (net/http.rb maintainer) but fix still isn't in CVS. I'm attaching it here in meantime. -- / Alexander Bokovoy --- Q: What is the sound of one cat napping? A: Mu. --LQksG6bCIzRHxTLp Content-Type: text/plain; charset=windows-1251 Content-Disposition: attachment; filename="ruby-net-http-alt.patch" --- ruby/lib/net/http.rb.orig 2002-07-12 00:33:38 +0300 +++ ruby/lib/net/http.rb 2002-09-19 17:30:22 +0300 @@ -67,6 +67,26 @@ Since Net::HTTP.Proxy() returns Net::HTTP itself when $proxy_addr is nil, there's no need to change code if there's proxy or not. +There are two additional parameters in Net::HTTP.Proxy() which allow to specify +proxy user name and password: + + Net::HTTP::Proxy(proxy_addr, proxy_port, proxy_name = nil, proxy_pass = nil) + +You may use them to work with authorization-enabled proxies: + + require 'net/http' + require 'uri' + + proxy_info = URI.parse(ENV['http_proxy']) + + proxy_name, proxy_pass = proxy_info.userinfo.split(":") if proxy_info.userinfo + + Net::HTTP::Proxy($proxy_addr, $proxy_port, proxy_name, proxy_pass).start('some.www.server') {|http| + # always connect to your.proxy.addr:8080 using specified username and password + : + } + + === Following Redirection require 'net/http' @@ -214,9 +234,10 @@ res = Net::HTTP.get_response(URI.parse('http://www.example.com')) print res.body -: Proxy( address, port = 80 ) +: Proxy( address, port = 80, username = nil, password = nil ) creates a HTTP proxy class. - Arguments are address/port of proxy host. + Arguments are address/port of proxy host and username/password if authorization + on proxy server is required. You can replace HTTP class with created proxy class. If ADDRESS is nil, this method returns self (Net::HTTP). @@ -279,6 +300,12 @@ : proxy_port port number of proxy host. If self does not use a proxy, nil. +: proxy_name + user name for accessing proxy. If self does not use a proxy, nil + +: proxy_pass + user password for accessing proxy. If self does not use a proxy, nil + : get( path, header = nil ) : get( path, header = nil ) {|str| .... } gets data from PATH on the connecting host. @@ -602,14 +629,14 @@ protocol_param :socket_type, '::Net::InternetMessageIO' class << HTTP - def start( address, port = nil, p_addr = nil, p_port = nil, &block ) - new( address, port, p_addr, p_port ).start( &block ) + def start( address, port = nil, p_addr = nil, p_port = nil, p_name = nil, p_pass = nil, &block ) + new( address, port, p_addr, p_port, p_name, p_pass ).start( &block ) end alias newobj new - def new( address, port = nil, p_addr = nil, p_port = nil ) - obj = Proxy(p_addr, p_port).newobj(address, port) + def new( address, port = nil, p_addr = nil, p_port = nil, p_name = nil, p_pass = nil ) + obj = Proxy(p_addr, p_port, p_name, p_pass ).newobj(address, port) setimplversion obj obj end @@ -645,8 +672,10 @@ @is_proxy_class = false @proxy_addr = nil @proxy_port = nil + @proxy_name = nil + @proxy_pass = nil - def HTTP.Proxy( p_addr, p_port = nil ) + def HTTP.Proxy( p_addr, p_port = nil, p_name = nil, p_pass = nil ) p_addr or return self p_port ||= port() @@ -658,6 +687,8 @@ @is_proxy_class = true @proxy_address = p_addr @proxy_port = p_port + @proxy_name = p_name + @proxy_pass = p_pass } proxyclass end @@ -669,6 +700,8 @@ attr_reader :proxy_address attr_reader :proxy_port + attr_reader :proxy_name + attr_reader :proxy_pass end def proxy? @@ -697,10 +730,16 @@ def conn_port port end + + # Empty, void + def authorization(header) + return header + end def edit_path( path ) path end + module ProxyDelta private @@ -718,6 +757,14 @@ def edit_path( path ) 'http://' + addr_port() + path end + + def authorization(header) + if self.class.proxy_name then + header = Hash.new unless header + header['Proxy-Authorization'] = "Basic " + ["#{self.class.proxy_name}:#{self.class.proxy_pass}"].pack('m').strip + end + return header + end end @@ -729,6 +776,7 @@ def get( path, initheader = nil, dest = nil, &block ) res = nil + initheader = authorization(initheader) request( Get.new(path,initheader) ) {|res| res.read_body dest, &block } @@ -741,6 +789,7 @@ end def head( path, initheader = nil ) + initheader = authorization(initheader) res = request( Head.new(path,initheader) ) @newimpl or res.value res @@ -748,6 +797,7 @@ def post( path, data, initheader = nil, dest = nil, &block ) res = nil + initheader = authorization(initheader) request( Post.new(path,initheader), data ) {|res| res.read_body dest, &block } @@ -760,6 +810,7 @@ end def put( path, data, initheader = nil ) + initheader = authorization(initheader) res = request( Put.new(path,initheader), data ) @newimpl or res.value res @@ -767,18 +818,22 @@ def request_get( path, initheader = nil, &block ) + initheader = authorization(initheader) request Get.new(path,initheader), &block end def request_head( path, initheader = nil, &block ) + initheader = authorization(initheader) request Head.new(path,initheader), &block end def request_post( path, data, initheader = nil, &block ) + initheader = authorization(initheader) request Post.new(path,initheader), data, &block end def request_put( path, data, initheader = nil, &block ) + initheader = authorization(initheader) request Put.new(path,initheader), data, &block end @@ -789,6 +844,7 @@ def send_request( name, path, body = nil, header = nil ) + header = authorization(header) r = HTTPGenericRequest.new( name, (body ? true : false), true, path, header ) request r, body --LQksG6bCIzRHxTLp--