Re: POLS - exception comparisons

On Tue, 24 Aug 2004, Robert Klemme wrote:

> Note also, that because of the recursion backtraces will never compare
> equal.

right you are (and above) - i took your code.

> Ah, I see.  Why don't you just use a general retry as in:
>
> def exec_retry(tries = 3)
>  begin
>    yield
>  rescue Exception
>    tries -= 1
>    raise if tries == 0
>    retry
>  end
> end


it's long and terrible,  here's a snippet from my post to the nfs list, it's
what i'm implementing:

From [e-mail] Mon Aug 23 09:24:34 2004
Date: Mon, 23 Aug 2004 09:18:11 -0600 (MDT)
From: Ara.T.Howard <[e-mail]>
To: [e-mail]
Subject: [NFS] client side - application level 'dead' lock detection


nfs gurus-

i have an application which does alot of byte range locking on an nfs mounted
file.  twice now, i've seen a 'dead' lock appear on our nfs server - a lock
held by a non-existent pid.  none of the processes in question have been dying
unexpectedly (or exiting at all actually) and also have not been shut down
uncleanly using kill -9, for instance.  basically a dead lock simply appears
after some length of time during long periods of successful locking and file
useage - at this point after about 3 weeks of 24/7 lock usage.  the file is
never corrupt and i have found that i can clear the dead locks using

    mv locked locked.tmp
    mv locked.tmp locked

of course, this still leaves a lock on SOME file on the server, but not my
file.

discovering this i've been thinking of a way to determine via my application
WHEN this situation has arisen and to automatically recover from it.  here is
my algorithim so far, it attempts to make the determination that

    "i cannot get the lock AND no one else has it"

upon finding this, automatic recovery of the system is attempted.

i'd appreciate any insight as to it's effectiveness taking into account the
subtleties of real world nfs behaviour.  the following
pre-conditions/assumptions/definitions apply:

    - a lockd impl that works, excepting for the occasional 'dead' lock existing
      on the server which prevents all clients from obtaining any new locks

    - non-blocking lock types do not block, even in the presence of such locks

    - moving a file out of the way, and back again, clears such 'dead' locks
      (they may still exist on the server, but clients will be able to lock the
      'new' file)

    - the 'file in question' is the file under heavy usage/locking

    - a monitor file will be used along side the file in question.  this file is
      simply a zero length file that all applications will use in addition to
      the file in question.

    - a recovery file is simply an empty file using mark the time of lock
      recovery

    - a 'refresher' thread is one which simply loops touching a file and
      sleeping.  technically this could be either a thread or process so long as
      it does not go to sleep when it's process issues a blocking operation
      (like fcntl).

    - an auto-recovering lockfile library which uses link(2), it is atomic, safe
      over nfs, and contains no bugs ( ;-) ) - exists (i have an impl).


here is the algorithim

    0.
       attempt to apply a non-blocking lock of type write/read to the monitor
       file

    1.
      a. monitor lock success

        start a refresher thread for the monitor file and attempt to apply a
        non-blocking lock of type write/read (same lock as the one on the
        monitor file) to the file in question - this should always succeed; iff
        it does not raise an error, the algorithim, or something else, has
        failed.

        having aquired both locks, call the callback for the file in question
        and, when it is complete,  kill the refresher thread, unlock the file in
        question, and unlock the monitor file.

        if estale is encoutered during 1.a.  sleep and retry on 0.

      b. monitor lock failure

        iff the monitor file is stale (mtime < now - max_age)

          some process must have died uncleanly while holding the lock (or the
          network/cpu has become very slow for that client) - we attempt
          recovery:

            mark recovery_start_time

            create an nfs safe lockfile to serialize recovery among clients.
            this is a blocking operation.

              iff recovery file exists and is newer than recovery_start_time

                someone else has recovered, sleep and retry on 0.

              else recovery file is older than recovery_start_time or does not
              exist

                recover:

                  for file in (monitor file_in_question)
                    mv file file.tmp && mv file.tmp file
                  end

                  (note that this could cause estale in some other client - but
                  they are prepared to deal with this condition)

                touch recovery file

              rm lockfile

              sleep and retry on 0.

        else monitor file is not stale

          some other process must have the lock (it's refresher thread is
          running) sleep and retry on 0.

thanks in advance for any inputs/critiques.





so all the error catching is for applications that happen to be using the
'file in question' at the time of recovery.  basically i'm trying to overcome
an error in lockd - it's not easy ;-(


kind regards.

-a
--
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: [phone]
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it. 
|   --Dogen
===============================================================================


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  [e-mail]
https://lists.sourceforge.net/lists/listinfo/nfs


thanks for the help.

-a
--
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: [phone]
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it. 
|   --Dogen
===============================================================================