From: Adam Bender <abender@...>
Date: 2009-04-30T16:54:36+09:00
Subject: Emulating a web browser

--001485f6d538b855e10468c10199
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I am looking for a library to help me emulate a web browser, at least at the
network level.  By this I mean I would like to run a program that, from the
point of view of a web server, behaves just like, say, Firefox, but I don't
care about actually displaying text or images or anything like that.  What I
would like it to do is speak HTTP, store and send cookies, automatically
fetch embedded content like images and style sheets, and so forth.  I
thought Mechanize was what I wanted, but it doesn't fetch embedded content.
It doesn't even recognize it.  I could perhaps tell Nokogiri to find all the
images and have Mechanize fetch them, but I've never used Nokogiri before, I
don't know an exhaustive list of types of embedded content Firefox loads
automatically (images, JavaScript, Flash, anything else?), and it seems like
getting Mechanize to emulate FF's HTTP request for these objects is
difficult.

Are there libraries that are meant for this type of interaction with
websites?  Perhaps I'm better off abandoning Ruby and making a Firefox
extension.

Thanks,

Adam

--001485f6d538b855e10468c10199--