From: "sean.swolfe@..." <sean.swolfe@...>
Date: 2005-12-24T06:27:51+09:00
Subject: Re: character safe CSV parser.

Ahh there we go! I guess what I was trying to ask is what you just sent
me. But in the README and FasterCSV file I see this:

README (line 49 -51):
"== Documentation

See FasterCSV for documentation."

and this in faster_csv.rb (Line 8)
"# See FasterCSV for documentation. "

Because of that I was assuming that there should have been a file
called "FasterCSV" like the "README" file in the project package. I
didn't see any URL anywhere.

But regardless, I get the following error:
Unquoted fields do not allow \r or \n.

RAILS_ROOT: ./script/../config/..
Application Trace | Framework Trace | Full Trace

c:/dev/ruby/lib/ruby/gems/1.8/gems/fastercsv-0.1.4/lib/faster_csv.rb:408:in
`shift'
#{RAILS_ROOT}/app/controllers/import_controller.rb:41:in `join'
#{RAILS_ROOT}/app/controllers/import_controller.rb:41:in `upload'


This is strange since all fields are quoted. It could be because of the
Vertical tab characters that are in the file.

I called the library like so....
        faster_csv = FasterCSV.new(file.read, :row_sep => "\r")
        faster_csv.each do |csv_row|
            table_row = {}
            for index in 0...csv_row.length
                table_row[columns[index]] = csv_row[index].sub("\v",
"\n") if csv_row[index]
            end
            table[table_row[id_symbol]] = table_row
        end


The CSV file is an export from a user's FileMaker database that will
then get uploaded to this rails app to be parsed into a MySql database.
Here is a little exerpt from FileMaker about their CSV export format (I
hope this is of help):

>APPLICABLE TO
>FileMaker Pro 6
>
>If you are trying to reconstruct data from a recovered file, it might be helpful to know which characters are 'legal' in FileMaker Pro data. Knowing this, you may be better able to clean up your data by purging the text file of bad characters.
>
>Following is a list of expected embedded characters in FileMaker Pro text exports. Be aware that any of these listed characters may be a problem if they are not used correctly in your data file. Rebuilding a file is labor intensive and generally a trial-and-error process.
>
>Expected embedded characters in FileMaker Pro text exports:
>
>1. ASCII 29 for Repeating fields.
>
>2. Repetitions are separated by Group Separator character $1D (decimal 29) when exported.
>
>3. Embedded return $0D (decimal 13) or $0A (decimal 10) (hard wrap, not soft wrap) is exported as Vertical Tab character $0B (decimal 11).
>
>4. If you are performing an export to Tab Separated text, the embedded tab $09 (decimal 9) is exported as Space character $20 (decimal 32.) For all other export formats, the tab character is exported as itself (Horizontal Tab $09 (decimal 9).)
>
>5. Records are separated by EndOfLine character(s) usual for the platform:
>CR $0D(decimal 13) for Macintosh
>CRLF $0D$0A (decimal 13 10, concatenated) for PC, or
>LF $0A (decimal 10) for Unix.
>
>6. No other control characters (<=$1F (less than or equal to decimal 31)) are generated during export, but embedded control characters are exported as themselves excepting as specified in #2 and #3 above.
>
>7. Accented characters are exported as themselves without remapping from the platform's normal character set: ' $8C (140) is exported as $8C (decimal 140) on the Mac - it is NOT remapped to $86 (decimal 134) which is the equivalent ASCII character.



Sean