From: Luke Kanies <luke@...>
Date: 2005-07-23T05:24:42+09:00
Subject: Threading help with testing webrick and xmlrpc

Hi all,

Well, as so often is the case, I'm finding that my testing was the
source of a bug that's been bothering me for a while (I was able to
ignore it because I had other priorities).

I'm writing an xmlrpc server (it was previously a SOAP server, but I
changed to XMLRPC to see if it would fix this bug -- of course not) to
pass files around.  It works fine if I start the client and server in
entirely different processes, but I get a hang on larger files if I try
to use threads for it to work.  Of course, my test code is the primary
place I want to run both client and server in the same process, so
that's where I'm encountering the problem.

Here's basically what my test code looks like:

    port = 8088

    files = %w{... a list of files ...}

    server = FileBucket::BucketWebserver.new(
        :Bucket => @bucket,
        :Debug => true,
        :Port => port
    )
    trap(:INT) { server.shutdown }
    serverthread = Thread.new {
        server.start
    }
    client = FileBucket::Dipper.new(
        :Server => "localhost",
        :Port => port
    )

    # i'm obviously using asserts here and such, but this is the basic
    # idea
    clientthread = Thread.new {
        files.each { |f|
            sum = client.backup(file)
            File.unlink(file)
            client.restore(sum)
        }
    }
    clienthread.join
    server.shutdown
    serverthread.join

There's a good bit more going on than that (you can find the full code
linked to, below), but it gets the basic idea across.

I want to start a server, then create a client, and pass that client a
bunch of file names and have it send the server those files.

As I said, this works just fine if the server is in a separate process,
and for some reason it works fine for smaller files (around 20k), but it
hangs indefinitely -- and then the server refuses to even shut down --
for files around 1MB.  This seems to be a threading problem, then, but
(as I alluded to in an earlier thread) I don't know much about threads.
It looks fine to me, but obviously isn't.

Could this be an issue within the webrick or xmlrpc library?  All of my
classes are entirely independent, with no shared state or shared
variables, but could it still be a problem there?

Any help would be much appreciated.  It's currently acceptable if the
only solution is to put the code in different processes, but in the long
term I'm going to need to be able to perform both functions in the same
thread.

Server:
https://reductivelabs.com/cgi-bin/puppet.cgi/file/language/trunk/bin/filebucketd

Library:
https://reductivelabs.com/cgi-bin/puppet.cgi/file/language/trunk/lib/puppet/filebucket.rb

Test code:
https://reductivelabs.com/cgi-bin/puppet.cgi/file/language/trunk/test/bucket/tc_bucket.rb

-- 
The surest sign that intelligent life exists elsewhere in the universe
is that it has never tried to contact us.
                --Calvin and Hobbes (Bill Watterson)
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://config.sage.org