From: Thomas Chust <chust@...>
Date: 2009-07-07T03:21:55+09:00
Subject: Re: file.seek and unused bytes

2009/7/6 Greg Willits <lists@gregwillits.ws>:
> [...]
> -- So, this particular step I've been talking about is where a raw CSV
> file undergoes normalization by reorgnizing the fields of each record
> into a common structure for that given data topic, each field undergoes
> some scrubbing (character case, packing phones, normalizing date
> formats, translation of codes into strings, etc).
> [...]

Hello,

reading this I wonder why you don't dump the records into a database
at this point. For example SQLite3 would be lightweight, fast and
flexible. You could probably save a lot of effort not having to build
index structures manually and you could query your dataset using SQL
or the ORM of your choice afterwards...

cu,
Thomas