From: David Masover <ninja@...>
Date: 2008-09-26T12:34:16+09:00
Subject: Re: How to make Ruby _THE_ scripting language of choice, fold in SQLite

On Thursday 25 September 2008 21:19:37 Clifford Heath wrote:
> David Masover wrote:
> > SQL databases aren't the only kind of database, nor is a database the only 
way 
> > to store data. Nor is SQLite the only SQL database, or necessarily the 
best 
> > to optimize for.
> 
> However, most of the non-SQL things that are included in such definitions
> of "database" are not actually transactional, and hence don't qualify
> as databases in any true sense of the word.

However, many things that I would consider are, in fact, quite transactional. 
Not as much as they could be, perhaps, but close enough.

First example: Filesystems. With ordered write mode, you can simply write to a 
temporary file, then rename said temporary file into place. With unordered 
writes, add an fsync between those two steps.

Any modern, journaled filesystem will make sure that the rename either 
succeeds or doesn't, and you've made sure all data is successfully on disk 
before you rename.

Implementation: Maildir.



Second example: Amazon's Dynamo. This powers S3 and SimpleDB, among other 
things. They've got a paper on it. Some implementations can, indeed, be 
considered purely-ACID.

What sets it apart from (most) SQL is, conflicts are expected as part of 
normal operation, and are left to the application to sort out. That is: 
Rather than preventing anyone else from modifying a record while I update it, 
simply allow two versions of a record to exist, and provide common algorithms 
for either choosing which version "wins", or for combining the two versions 
into a third.

Thus, inconsistency is allowed, but only temporarily. It is never exposed to 
the end-user, or even to (most of) the application.



Third example: Distributed version control, like Git. I mention this mostly 
because it reflects the same philosophy as Dynamo above, albeit with more 
human intervention -- but also because this is the one most developers are 
likely to be intimately familiar with (or should be).



> If by "transactions" you mean ActiveRecord::Base.transaction, possibly 
no-one.

Precisely so.

I remember that one of the first things I wrote, when developing a new Rails 
app, was this method:

def double_save
  transaction do
    save(false)
    yield
    save!
  end
end

This to allow the creation of circular record structures -- for example, every 
domain must have an SOA record, and every record (of any kind) must have a 
domain. Since I'd made these fields NOT NULL on the database, I would often 
do this hack with new records of this kind:

d = Domain.new(...)
d.soa_id = -1
d.double_save do
  d.soa = Soa.create!(...)
end

That -1, of course, would've killed my validations.

> Attempting rollback on exceptions from a user-mode process without using
> a two-phase locking protocol (as many/most of AR's adapters do) is utterly
> flawed and not transactional at all.

I actually don't see a lot of that -- more simple assumptions that nothing 
will go wrong, or that dangling records aren't a problem.

Quite a lot of defensive programming, too, based on the idea that the data 
WILL get corrupted somehow, someday, and being able to handle corrupt data is 
more important than avoiding the corruption in the first place.