From: Francis Cianfrocca <garbagecat10@...>
Date: 2006-10-23T13:13:58+09:00
Subject: Re: Fast portable storage for queues

------=_Part_89392_12677447.1161576836012
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

On 10/22/06, snacktime <snacktime@gmail.com> wrote:
>
> I've tested out a couple of ways of storing a queue structure and
> wondered if others had some better ideas.  I tried a berkeleydb based
> queue using it's native queue structure, which gives the best
> performance so far but it's for unix only.  I also have a file based
> queue where every message is just stored in a sequentially numbered
> file.  Almost as fast as berkeleydb and works anywhere.  Just for
> giggles I tried sqlite, and then immediately deleted it.  6-8 tps with
> sqlite compared to around 1000 with berkeleydb/flat files.
>
> Another alternative I was thinking of is a single file storage of some
> type, maybe fixed length or even variable length records.  Something
> that's portable.  But I'm thinking that could get complicated fairly
> quickly.
>
> Any other ideas?
>
>

Kirk Haines was involved in a thread here a couple of months back in which
he made the case that it's perfectly fast and perfectly scalable to just use
the filesystem for moderate requirements. His scheme used filenames that
were actually hashes of the contents, and as I recall he had a mechanism for
automatically generating a directory hierarchy based on the content hashes
to keep the inode sizes under control.

Having implemented all kinds of crazy single-file storage systems for
high-scalability message-queueing systems, I initially thought Kirk's idea
was just loopy. But sometimes the simplest solutions are the best. I
remember trying it out with some quick tests and it worked great.

------=_Part_89392_12677447.1161576836012--