From: Avdi Grimm Date: 2012-09-12T11:03:43+09:00 Subject: Re: Long calculation & time limit --047d7b6dbfd48fe90504c9779978 Content-Type: text/plain; charset=ISO-8859-1 Yeah, I've used Ubuntu for years and I've never heard of a process being killed because it took too long (how would daemons work otherwise?) However, the kernel's OOM (Out Of Memory) killer will definitely kill processes that are starving the system of memory. If you're loading all the records into memory instead of processing them in a streaming or chunked fashion you could easily be running afoul of the OOM killer. On Sep 11, 2012 5:11 PM, "Douglas Seifert" wrote: > On Sun, Sep 9, 2012 at 12:51 PM, Brian Candler wrote: > >> toto tartemolle wrote in post #1075217: >> >>ulimit -a >> > core file size (blocks, -c) 0 >> > data seg size (kbytes, -d) unlimited >> > scheduling priority (-e) 20 >> > file size (blocks, -f) unlimited >> > pending signals (-i) 16382 >> > max locked memory (kbytes, -l) unlimited >> > max memory size (kbytes, -m) unlimited >> > open files (-n) 1024 >> > pipe size (512 bytes, -p) 8 >> > POSIX message queues (bytes, -q) 819200 >> > real-time priority (-r) 99 >> > stack size (kbytes, -s) 8192 >> > cpu time (seconds, -t) unlimited >> > max user processes (-u) unlimited >> > virtual memory (kbytes, -v) unlimited >> > file locks (-x) unlimited >> >> Well it was a long shot, but I just wanted to check that cpu time was >> unlimited (which it is). >> >> >>Message at full >> > >> > litterally: "Killed, returned error 137" >> >> Nothing else? No ruby backtrace, nothing? >> >> Actually that makes sense. As has been pointed out, 137 = 128 (killed) >> +9 (SIGKILL). This means that some other process sent a kill signal, >> e.g. using "kill -9 ", which ruby cannot trap; it dies instantly. >> >> However it is up to you to find out which other process sent the signal. >> Have you or someone on this system implemented some sort of watchdog >> process? This is *not* default behaviour. >> >> Try running a simple infinite loop: >> >> ruby -e 'loop {}' >> >> Does this also die? >> >> Another suggestion: keep monitoring your process using 'top' while it >> runs. Keep a note of the memory usage. See if it dies at a consistent >> point. >> >> I don't *think* that simply running out of memory will cause the process >> to be killed -9; malloc should fail and ruby should be able to print >> something. >> >> -- >> Posted via http://www.ruby-forum.com/. >> >> > Ubuntu (Linux) will definitely kill off processes consuming large amounts > of memory if all the system RAM has been used up and the box is swapping > heavily. > > OP, check your script for memory leaks. > > -Doug Seifert > > --047d7b6dbfd48fe90504c9779978 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Yeah, I've used Ubuntu for years and I've never heard of a proce= ss being killed because it took too long (how would daemons work otherwise?= ) However, the kernel's OOM (Out Of Memory) killer will definitely kill= processes that are starving the system of memory. If you're loading al= l the records into memory instead of processing them in a streaming or chun= ked fashion you could easily be running afoul of the OOM killer.

On Sep 11, 2012 5:11 PM, "Douglas Seifert&q= uot; <doug@dseifert.net> wro= te:
On Sun, Sep 9, 2012 at 12:51 PM, Brian Candler <lists@ruby-forum.com> wrote:
toto tartemolle wrote in post #1075217:
>>ulimit -a
> core file size =A0 =A0 =A0 =A0 =A0(blocks, -c) 0
> data seg size =A0 =A0 =A0 =A0 =A0 (kbytes, -d) unlimited
> scheduling priority =A0 =A0 =A0 =A0 =A0 =A0 (-e) 20
> file size =A0 =A0 =A0 =A0 =A0 =A0 =A0 (blocks, -f) unlimited
> pending signals =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 (-i) 16382
> max locked memory =A0 =A0 =A0 (kbytes, -l) unlimited
> max memory size =A0 =A0 =A0 =A0 (kbytes, -m) unlimited
> open files =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0(-n) 1024
> pipe size =A0 =A0 =A0 =A0 =A0 =A0(512 bytes, -p) 8
> POSIX message queues =A0 =A0 (bytes, -q) 819200
> real-time priority =A0 =A0 =A0 =A0 =A0 =A0 =A0(-r) 99
> stack size =A0 =A0 =A0 =A0 =A0 =A0 =A0(kbytes, -s) 8192
> cpu time =A0 =A0 =A0 =A0 =A0 =A0 =A0 (seconds, -t) unlimited
> max user processes =A0 =A0 =A0 =A0 =A0 =A0 =A0(-u) unlimited
> virtual memory =A0 =A0 =A0 =A0 =A0(kbytes, -v) unlimited
> file locks =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0(-x) unlimited
Well it was a long shot, but I just wanted to check that cpu time was=
unlimited (which it is).

>>Message at full
>
> litterally: "Killed, returned error 137"

Nothing else? No ruby backtrace, nothing?

Actually that makes sense. As has been pointed out, 137 =3D 128 (killed) +9 (SIGKILL). This means that some other process sent a kill signal,
e.g. using "kill -9 <pid>", which ruby cannot trap; it dies= instantly.

However it is up to you to find out which other process sent the signal. Have you or someone on this system implemented some sort of watchdog
process? This is *not* default behaviour.

Try running a simple infinite loop:

=A0 =A0 ruby -e 'loop {}'

Does this also die?

Another suggestion: keep monitoring your process using 'top' while = it
runs. Keep a note of the memory usage. See if it dies at a consistent
point.

I don't *think* that simply running out of memory will cause the proces= s
to be killed -9; malloc should fail and ruby should be able to print
something.

Ubuntu (Linux) will definitely = kill off processes consuming large amounts of memory if all the system RAM = has been used up and the box is swapping heavily.

OP, ch= eck your script for memory leaks.

-Doug Seifert=A0

--047d7b6dbfd48fe90504c9779978--