From: David Balmain <dbalmain.ml@...>
Date: 2005-12-20T10:45:57+09:00
Subject: Re: Writing to ferret index from multiple processes

Hi Andreas,

This is what I would expect to happen. What machine where you running
it on the first time. Whatever it was, Ferret's locking mechanism must
not work.

Anyway, to avoid this problem you need to make sure the batch process
doesn't keep the lock for too long (about 5 seconds). I would change
the rebuild index method to use an IndexWriter or switch auto_flush to
false. This should speed the reindexing up. I'd also add a pause in
there so other processes can get a hold of the lock if they need to.
Since you are flushing explicitly you may as well set auto_flush to
false anyway.

  def index
    @index ||= Index::Index.new(:path => @path,
                                #:auto_flush =>true <= don't use this anymore
                                :default_search_field => ['subject'],
                                :key => ['id', 'class'])
  end

  # update will continue to work, handling the flushing explicitly
  def update(post)
    index << create_doc(post)
    index.flush
  end

  # batch_update will keep the IndexWriter open between updates
  # so it will run much faster
  def batch_update(post)
    index << create_doc(post)
  end

  # define a flush method for use with the batch_update method
  def flush
    index.flush
  end

Then in your process that is doing the reindex I'd use the
batch_update method and I might even add some pauses in there.
Something like this;
  MAX_ADDS_BEFORE_FLUSH = 10
  def rebuild_index
    i = 0
    Post.find_all_by_deleted(0).each do |post|
      self.update(post)
      i += 1
      if (i % MAX_ADDS_BEFORE_FLUSH) == 0
        self.flush
        sleep(0.5)
      end
    end
  end

These are just ideas. You'll probably come up with something better. I
think the best solution is just to keep the Ferret index in sync with
the database so that you don't need to reindex everything.

Let me know what kind of system you were running it on the first time
to get the documents out of order error. I'll see if I can find out
why the locking wasn't working.

Cheers,
Dave

On 12/20/05, Andreas S. <f@andreas-s.net> wrote:
> Andreas S. wrote:
> > David Balmain wrote:
> >> Hey Andreas,
> >>
> >> The latest version of RForum still has :create => true so I'm guessing
> >> you haven't checked in your latest changes. Could you let me know when
> >> you have?
> >
> > I have checked it in.
>
> Btw, I tried it again on another machine, and couldn't reproduce the
> "docs out of order" exception, but instead I got
> RuntimeError: could not obtain lock:
> ./ferret_index/ferret-f62496686e637eca67e933a9cdc5eb21write.lock
>
> --
> Posted via http://www.ruby-forum.com/.
>
>