From: Chuck Remes <cremes.devlist@...>
Date: 2009-09-10T06:07:34+09:00
Subject: Re: EventMachine.defer and ActiveRecord connection pool?


On Sep 9, 2009, at 3:37 PM, Artūras Šlajus wrote:

> Hello,
>
> I'm writing a server with EM. I'm having some long-running tasks, so  
> I'm
> deferring those requests using EM.defer.
>
> However, after 5 such deferred requests AR suddently stops working...
> I'm using AR 2.3.4, so concurrency shouldn't be a problem. Gems
> configuration:
>
> REQUIRED_GEMS = [
>  ['mysql', '>=2.7.0'],
>  ['activerecord', '=2.3.4'],
>  ['activesupport', '=2.3.4'],
>  ['eventmachine', '=0.12.6'],
>  ['json', '>=1.1.6'],
>  "daemons"
> ]
>
> Anyway, as you can see it happens after 5 requests (just simulated
> action, one select query):
>
> [2009-09-09 23:32:35|main|info ] Starting server...
> [2009-09-09 23:32:36|main|info ] [Resources manager]
> [Resources manager END] Time taken: 0.031 seconds
> [2009-09-09 23:32:37|main|info ] [Resources manager]
> [Resources manager END] Time taken: 0.016 seconds
> [2009-09-09 23:32:38|main|info ] [Resources manager]
> [Resources manager END] Time taken: 0.000 seconds
> [2009-09-09 23:32:39|main|info ] [Resources manager]
> [Resources manager END] Time taken: 0.016 seconds
> [2009-09-09 23:32:40|main|info ] [Resources manager]
> [Resources manager END] Time taken: 0.000 seconds
> [2009-09-09 23:32:41|main|info ] [Resources manager]
> [Resources manager END with EXCEPTION]
> #<ActiveRecord::ConnectionTimeoutError: c
> ould not obtain a database connection within 5 seconds.  The max pool
> size is cu
> rrently 5; consider increasing it.>
>

I have a guess. I guess that your CONFIG['resources_manager.period']  
is set to 0 or quite close to it. If so, the PeriodicTimer is probably  
firing on every "crank" of the EM reactor which is deferring more and  
more of your singleton blocks. There isn't any time left over to clean  
up connections to the ActiveRecord connection pool and you are running  
out.

Trying making your 'resources_manager.period' 1 second and see if it  
still blows up.

BTW, EM#defer uses a threadpool (size of 20). It enqueues each #defer  
request to a Queue (thread safe). As each thread in the pool completes  
its task it pops the next one off the queue.

If you flood the queue (which I suspect) then the newest tasks are  
going to starve while waiting for the previous ones to complete. As  
each task completes, try outputting the number of tasks in your  
@@running hash and see if it is a large number.

cr