From: Daniel Lichtenberger <daniel_usenet@...>
Date: 2004-01-31T09:39:51+09:00
Subject: Re: Net::http.get has a 50K limit?

Hi!

Meihua Liang wrote:

> Yes the server completes the document. Browser serves it nicely, and I
> also verified via wget, which give the complete copy. I still don't know
> why Net::http prematurely cuts off the document.

I played around with your script a bit, and noticed something strange: When
trying to fetch the file via telnet, it is also cut off early. However, as
you said, wget correctly retrieves the whole document. Why? wget sends a
user-agent header field, and only in this case the whole document is
served. So, by adding a user-agent header field to your request, it works
for me (with Ruby 1.8):

response = http.get("/wl/jobs/JS_JobSearch?TS=1012409733026",   
        {"user-agent" => "blub"})

returns around 60000 bytes in response.body.
When writing www spiders, you sometimes have to outsmart the webservers ;).

Hth,
Daniel