From: Robert Klemme <shortcutter@...>
Date: 2008-04-02T04:35:28+09:00
Subject: Re: should I use a database or a flat file?

2008/4/1, Todd Benson <caduceass@gmail.com>:
> On Tue, Apr 1, 2008 at 11:55 AM, Lionel Bouton
>  <lionel-subscription@bouton.name> wrote:
>  > Todd Benson wrote:
>  >
>  >  > I'm going to slightly disagree with Lionel -- and also Robert -- on
>  >  > this one.  First of all, a database is not necessarily just for
>  >  > concurrency.  It's for data integrity
>  >
>  >  Yes I agree (as explained below concurrency is what I consider the main
>  >  problem to solve to enforce data integrity). That said if you write your
>  >  data in one pass as the OP, you don't need data integrity in the storage
>  >  layer... rename is atomic : you either renamed the temp file to its
>  >  final position before a crash or not.

Exactly.  With regard to all that we've learned about the issue at
hand a DB seems overkill here.  KISS

>  >  The problem are partial updates where you need to maintain consistancy.
>  >  And on the top of my head the only problems with partial updates are :
>  >  - concurrent accesses (most common, counting both concurrent read and
>  >  write accesses),
>  >  - crashes (fortunately less common and can even be adressed by backups
>  >  in many cases).
>  >
>  >  These are why I disagree with people wanting to push all the consistency
>  >  logic into the applicaltion layer on database-backed applications with
>  >  concurrent access (like often advocated for Rails). It's simply not
>  >  doable without recoding the whole concurrent access manager  and
>  >  log-based/MVCC/... crash resistance of the database in the application
>  >  layer (good luck with that).

Totally agree - but this is another story.

> Maybe we are talking about different things.  By data integrity, I
>  mean you can be certain not just that the data was entered correctly,
>  but also that it coincides with the relationships present.  In a
>  modified version of the OP's model, for example...

>  Now, surely, you can say, "Well, the application logic will take care
>  of that ambiguity."  But I say we should continue to separate
>  application logic from data logic.

But the consistency needs to be /somewhere/ and if no database is
needed then enforcing it in app logic is certainly ok.

>  I'm no CS guy, so I don't know the correct terms for this, but I do
>  see the potential pratfalls.
>
>  There certainly is a time and place for this, but I've found it's
>  usefulness generally not that beneficial.

What is "this" in this paragraph?

Generally I do not think we're far away - if at all.  Given the scale
of the problem and the apparent lack of future extension with regard
to size, complexity and concurrency a simple solution suffices IMHO.
Of course it's good to know the options - that's why we discuss here.

Kind regards

robert


A modified version of the script since the other posting did not seem
to make it into usenet. This one has consistency check as originally
required:

#!/bin/env ruby

require 'set'
require 'yaml'

class CatNames
  def self.load(file_name)
    File.open(file_name) {|io| YAML.load(io)}
  end

  def save(file_name)
    File.open(file_name, "w") {|io| YAML.dump(self, io)}
  end

  def initialize
    @cat = {}
    @all = {}
  end

  def add(cat, name)
    raise "Consistency Error" if @all[name]
    s = (@cat[cat] ||= Set.new)
    s << name
    @all[name] = s
  end

  def remove(cat, name)
    c = @cat[cat] and c.delete name
    @all.delete name
  end

  def clear
    @cat.clear
    @all.clear
  end

  def size
    @cat.inject(0) {|sum,(name,set)| sum + set.size}
  end
end

t = Time.now

d = CatNames.new

1000.times do |i|
  d.add("cat#{i % 10}", "name#{i}")
end

puts d.size

tt = Time.now
printf "%6.3f %s\n", tt-t, "create"
t = tt

d.save "test.yaml"

tt = Time.now
printf "%6.3f %s\n", tt-t, "write"
t = tt

d2 = CatNames.load "test.yaml"

tt = Time.now
printf "%6.3f %s\n", tt-t, "load"
t = tt

begin
  d2.add "foo", "name0"
rescue Exception => e
  puts e
end


-- 
use.inject do |as, often| as.you_can - without end