Re: Web APIs

On Fri, May 28, 2004 at 05:28:30AM +0900, Kirk Haines wrote:
> On Fri, 28 May 2004 03:59:55 +0900, Aredridel wrote
> 
> >   * URL portability -- if a script lives in a server's namespace at
> >   /application/feed, and you need to move it to /apllication2/feed,
> >  it  shouldn't have to refer to everything using absolute paths nor  
> > relative ones when at all possible. It should, at the least, be able 
> >  to ask the API for a URL like:
> > 
> >     #{thisscript}/path
> >     #{thisserver}/path
> >     #{thisrequest}/path
> >   to facilitate more aware applications. In addition, where possible,
> >  it  would be nice to have rewrite support be bidirectional, so if there's
> >   a rewrite for /foo/bar to /baz/bar, if the app requests a path that
> >   would ordinarily return /baz/bar, the server could rewrite to /foo/bar
> >   so the user sees only clean URLs.
> 
> On the surface, this seems like a pretty simple thing, although my brain 
> isn't coughing up any good real life examples of why this would be useful.  
> Can you help my poor brain with one?

Just pragmatism: It's easier to do with the app cooperating than with a
proxy solution. Proxies might understand HTML, but I use a lot of RDF
and things like that... it's easy to get wrong.

It's good for URL migration, and for filtering namespaces -- if I
implement an app that, say, is a web-based editor, I can see it being
very useful to "reparent" the entire namespace transparently into a
sub-space -- 

if /foo.html and /bar.html are the normal namespace, it would be nice to
implement a filter so that /baz/foo.html maps to the original foo.html,
it would be nice to not have to recode any absolute links in the files.

More complicated scenarios affect relative URLs, too.

> >   * Registry or inspection of namespace -- I'd love an application 
> > to be  able to find out what other valid paths are within the server,
> >  so that  there's less guesswork in making a non-brittle way of linking
> >   applications together. Being able to say
> >   "server.application['wiki'].url" would be really cool, though perhaps
> >   too simplistic.
> 
> These seem somewhat related to the URL portability, and is a cool idea.  In 
> fact, I can see enough reason in my own work for this one right now that I 
> might build this into Iowa.  I'd love to be able to register components in 
> some centralized sort of registry that would exist outside of the umbrella 
> of any one application process.
> 
> So, to borrow from your example, maybe I could do something like this:
> 
> 
> new_page = registry.application('wiki')
> new_page.page = 'HtmlTemplates'
> yield new_page

Ooh, nice.

> or, if I wanted to embed a direct link to the wiki within my output, in the 
> template I could do something like this:
> 
> <a href="@wikilink">Jump To The HTML Templates Wiki Page</a>
> 
> --- (in the code file)
> 
> def wikilink
>   new_page = registry.application('wiki')
>   new_page.page = 'HtmlTemplates'
>   new_page.url
> end
> 
> That would be very, very neat if it would work regardless of whether the 
> wiki and the application that wanted to connect to it was being managed 
> under the same overarching process or a different one.
> 
> The wiki application, when it registered itself, would have to tell the 
> registry what parameters (such as 'page') it accepts, and could provide a 
> method to call to determine the URL to call to invoke it, if it was more 
> complicated than the superclass method could handle (such as if it takes a 
> parameter on the query string to define what wiki page to display).  That 
> could be a very neat way to link applications that would work even if one 
> changed the location of an application.  After changing the location, the 
> application would just reregister itself in the new place, and everything 
> that cared about it could still find it.  That would be very, very, very 
> cool for me.  Very very very cool.

Yeah... that's what I was thinking.

Making it RPC-based isn't a bad idea, so it would work across servers
(Aha! Web-enabling web applications! A novel idea!)

> >   * Streamability -- templating has to be optional, and preferably
> >   separate, so that one could, say, write a streaming video-over-http
> >   server, or a realtime chat system.
> 
> This is interesting.  I don't think it is necessarily related specifically 
> to templating, though, but rather more generally to a buffered versus an 
> unbuffered response.  For example, with Iowa one can skip using a template 
> simply by making the "template" be a single call to a method that returns 
> the actual content.  Currently there is no option to have that content be 
> returned in an unbuffered way, however.  It all gets collected into a buffer 
> and is then returned in one large chunk.

Bingo. I just associate it with templating, because template engines
-all- buffer as far as I can tell.

Even for non-realtime apps, it's important, because it's nice to give
the browser something to munch on while the rest is generating. Users
like feedback.

> If I were to implement a mechanism that allowed the content to be sent back 
> to the front end as it was generated, one could then stream that content, 
> regardless of whether there was any sort of a "template" involved in its 
> creation or not.
> 
> Not that one would probably _want_ to do this under Iowa until we either 
> have native threads or I implement some sort of a multiple process backend 
> so that one wasn't trying to send 15 streams out of seperate threads sharing 
> the same process, but....

Fifteen wouldn't be so bad even with the green threads.

Some of the fastest web servers out there use non-blocking IO instead of
threads (which is what Ruby threads do internally.)

> >   * Support for more than GET and POST methods. It should be 
> > possible to  write a WebDAV server using the API, without having to 
> > modify the  webserver configuration.
> 
> This seems like it would mostly be a function of whatever front end is 
> receiving the HTTP requests.  Whatever API/system is being used to generate 
> content shouldn't have any restrictions at all regarding what kind of 
> request that it is.  GET or POST of COPY or MKCOL or STRAIGHTONTILMORNING 
> should all work if the frontend accepts them.

Yeah, it should. Making that a requirement is all I'm doing.

> >   * Filtering of requests, so I could define a handler to intercept all
> >   text/http documents served by the server, and apply some
> >   transformation to them.  This would aid writing applications by
> >   aggregation rather than making applications monolithic. It brings 
> > to  mind the useful unix concepts of filters and pipes.
> 
> That's a powerful tool.  The one thing that a command line has going for it 
> is that it is easy to specify the order of what pipes into what.  Handling 
> the interactions of ordering with a system that intercepts responses adds a 
> wrinkle that would need to be thought about carefully in order to find a 
> solution that is flexible without being difficult.

Yeah -- However, it can largely be set up by the admin. Let the tools be
linked together in ways not thought of beforehand.  Let apps do it
internally, too, with the same mechanism, but present a unified front.

I do it internally in my php-based Wiki. It's been a great tool. The
Wiki spits out very plain XHTML. My filters transform that into content.
It's XSLT as web-templating. Just one useful thing.

> > There's probably a lot more, but I just want to start the conversation
> > (and if neccesary, we can form a mailing list just for such a thing) 
> > so we can stop re-inventing the wheel and make something exceptional.
> 
> I'd be interested in this, regardless of where the discussion takes place.  
> This is a fascinating topic, and I know that I can definitely use some of 
> this to make Iowa a more powerful tool.

Excellent. I aim to keep persuing it.

What triggered the idea in my head as something that needed to be fixed
is how I re-invented the wheel to implement webrick-fcgi -- I like the
webrick API, as opposed to the CGI one. Internally, the library takes
the request parsed into variables by the webserver, shipped to the app
via FCGI protocol, and re-assembles it into an HTTP request that's
hopefully similar to the original, then feed that into WEBrick and let
it parse it into different variables.  Hardly optimal.

Then I realized that if mod_proxy could speak over a unix-domain socket,
I could just use WEBrick directly with a little hacking.

That made me realize that CGI variables is a poor, and webrick is a
somewhat better abstraction of HTTP.

Then I started thinking of improvements to HTTP and just to web-app
semantics that would make implementing robust programs easier, and
started off on this tangent.

Ari