Re: patch for lib/net/http.rb, self['User-Agent'] ||= 'Ruby'

At 1:54 AM +0900 11/5/07, Tanaka Akira wrote:
>In article <[e-mail]>,
>  Minero Aoki <[e-mail]> writes:
>
>>> After applying this patch the 'User-Agent:' header is set to 'Ruby' 
>>> unless it is already defined.
>>
>>> +      self['User-Agent'] ||= 'Ruby'
>>
>> OK, I have commited your patch in trunk (Ruby 1.9).
>>
>
>I guess it will be forbidden by someone, like libwww-perl.
>
>% ruby -rnet/http -e 'Net::HTTP.start("www.google.com") {|h| p h.get("/search?q=foo", "User-Agent"=>"libwww") }'
>#<Net::HTTPForbidden 403 Forbidden readbody=true>

Yes, perhaps but then just do:

$ ruby -rnet/http -e 'Net::HTTP.start("www.google.com") {|h| p h.get("/search?q=foo", "User-Agent"=>"") }'
#<Net::HTTPOK 200 OK readbody=true>

Of course both the problem I experienced:

1) a tomcat server running a webstart servlet needing a User-Agent header

and the problem you describe:

2) some servers forbidding access to certain User-Agents

are unusual.

At least the response "403 Forbidden" from Google was what Google intended to deliver. I'm sure that an "500 server error" response is not what the tomcat/webstart server authors intended.

The Google 403 response gives just a bit of a hint about where to look, the Tomcat 500 error is a bit harder to discover the problem and solution.

Of course now that I've written about the problem in a number of messages google searchers will have an easier time finding the solution for net/http in the 1.8 std library.

Perhaps the latest version of net/http should also have a way of suppressing the production of the "User-Agent" header entirely.  I have not checked yet what is produced when this option is passed:

  "User-Agent"=>""

I assume the User-Agent header is produced with no value.