From: Oliver Cromm <lispamateur@...>
Date: 2005-05-26T06:40:18+09:00
Subject: Re: [ANN] KirbyBase 2.2

* Oliver Cromm wrote:

> Jamey Cribbs wrote:
> 
>> Oliver Cromm wrote:
>> 
>>>So for a start, I plugged KirbyBase in just as a cache - where before, I
>>>was reading header data from a news server each time, in the new version
>>>I save the raw data to a KirbyBase, add only recent messages, then read
>>>the part of the data I want (by date) from the KirbyBase.
>>>
>> This might be the source of the slowness.  Is this field that you are 
>> reading by date defined as a Date field in the KirbyBase table?  [...]
>> 
>> Here is an alternative to try:  define this field in the table as a 
>> String field instead of a Date field.  Select's will still work pretty 
>> much the same way because, for example:
>> 
>>     2005-05-25 > 2005-05-24
> 
> I left the Date field as a string in the format I originally receive
> them, e.g. "Wed, 18 May 2005 10:29:44 +0900". Then, for each message, I
> use ParseDate. This is overhead for sure, but the point is that it is
> the same thing I do for the non-caching version (receive a specified
> number of Dates and decide which are within my limits).
> 
> But I'll go ahead and try a version where I parse at read-in time and
> store the result, which would be a number (or two numbers, as I'd want
> to keep the time zone separate).

I found some time now for further experiments, and stored time as an
integer. And yes, it is significantly faster this way, even slightly
faster than my first attempt to do the same with SQLite. 

Times from some test with similar, not exactly equal tasks, so read with
spoons of salt:
- reading data fresh from News server: 50s
- reading from KirbyBase with original format (rfc2822) Date field: 45s
- reading from KirbyBase with Date as Integer: 12s
- reading from SQLite with Date as Integer: 16s

I have to do quite a number of calculations on that field; for every
record selected (and in my simple experiments, that is nearly all of
them), I need to extract at least the day of the week and the day
number. But apparently, that doesn't take nearly as much time as a
KirbyBase "select" based on ParseDate(aField). I'm not quite clear about
what is going on with the select, but I know how to circumvent the
problem.
-- 
Oliver C.
45n31, 73w34
Temperatur: 14.9�C (25 May 2005 11:00 AM EDT)