From: Daniel DeLorme Date: 2010-05-19T20:29:05+09:00 Subject: Re: Asynchronous HTTP request Daniel N wrote: > On 19 May 2010 11:56, Daniel DeLorme wrote: >> Ok, here's the context. I didn't put this in my OP because I figured >> it would just bore everybody to tears. >> >> This is inside a rack request. The idea is that I'm assembling a web >> page by doing a bunch of sub-requests for the various parts of the >> page. So I'll have something like: >> >> action "index" do >> @news = subreq("http://news.server") >> @ad = subreq("http://ad.server") >> @blog = subreq("http://blog.server") >> @forum = subreq("http://forum.server") >> end >> >> All these sub-requests are launched asynchronously and, while they are >> executing, the app generates the layout within which the output of the >> subrequests will be embedded. So I'll have something like: >> >> response = ['', >> '

',@ad,'

', >> '

',@news,'

', >> '

',@blog,'

', >> '

',@forum,'

', >> ''] >> >> And when rack finally outputs the response to the client it will block >> on the various subrequests unless/until they have completed. >> >> What I can't figure out with EventMachine is how to have the "main >> thread" generate the layout while the subrequests are executing. >> >> > Ok now we're talking. So with rack you can't do true async with a callback. Ah, but I never really wanted callbacks; that would be evented IO. The various approaches to asynchronous IO are not terribly well defined, but what I meant by nonblocking was simply: resource.send_request #=> doesn't block waiting for response! resource.get_response This is not possible with Net::HTTP because those two phases of the http request are bound into one monolithic get(url) operation. > callback based async actually needs to block in order for the rack > application you're in to return it's result. You _could_ do it by returning > a custom object in the rack response that renders as much as possible while > it waits for the response, and then renders that when it can, but that > option may not be available depending on what framework you're using. I guess I was not clear enough, but this approach is exactly what I tried to explain above. > You can use ESI. There's an esi for rack project on github by Joshua Hull > which could be useful to you. http://github.com/joshbuddy/esi-for-rack You > can also use esi outside of the rack request in apache or nginx, by > responding first with a layout file containing esi tags pointing to the > content to use. Ngins, Apache or the esi rack project can then assemble the > page for you using the resources specified. Oh wow this was *really* interesting. This sent me on an hours-long exploration of Varnish+ESI and Nginx+SSI. It turns out that Nginx will fetch the subrequests in parallel but Varnish (caching proxy) will not. This is probably fine if most of the subrequests are already cached (which admittedly is the point of a caching proxy) but if not... Nginx is the winner. This opens a lot of possibilities. For example I can imagine serving up simple "skeleton" pages with heavy caching and then generate all the user-specific parts through SSI.