From: Paul Lutus <nospam@...>
Date: 2006-11-02T11:10:08+09:00
Subject: Re: Input file, change data, write to file

Paul Br wrote:

> I'm a ruby newbie trying to read data from a file, make a few changes,
> and write the output to a file so it can be imported into a MySQL
> database.
> 
> I found a partial solution on page 138 in Maik Schmidt's “Enterprise
> Integration with Ruby” book but it lacks a means to write the output to
> a file.
> 
> How can I write the output to a file using the below code?
> 
> For what it's worth, I'll be working with files that contain between
> 20,000 – 60,000 rows.
> 
> Below is a data sample:
> 
> 01234567890123456789012345678901234567890123456789012
> 
> 00123 random text   3.0010/20/200610/21/2006   -3.45
> 00253 more   text 275.0007/01/200606/12/2006   12.45
> 
> Here's what I want the file to look like with tabs between each section:
> 
> 01234567890123456789012345678901234567890123456789012
> 123 random text    3.00 2006-10-20 2006-10-21  -3.45
> 253 more   text  275.00 2006-07-01 2006-06-12  12.45
> 
> Filename: fixtest1.rb
> 
> class FixedLengthRecordFile
>   def FixedLengthRecordFile.open(filename, field_sizes)
> 
>     if field_sizes.nil? or field_sizes.empty?
>       raise ArgumentError, "Empty field sizes not allowed!"
>     end
> 
>     field_pattern = 'a' + field_sizes.join('a')
>     IO.foreach(filename) do |line|
>       record = line.chomp.unpack(field_pattern)
>       record.map { |f| f.strip! }
>       yield record
>     end
>   end
> end
> 
> Filename: rw1.rb
> 
> require 'fixtest1'
> 
> FixedLengthRecordFile.open('test1.abc', [2, 3, 12, 7, 2, 1, 2, 1, 4, 2,
> 1, 2, 1, 4, 10]) do |row|
>   puts
>
"#{row[1]}\t#{row[2]}\t#{row[3]}\t#{row[8]}-#{row[4]}-#{row[6]}\t#{row[13]}-#{row[9]}-#{row[11]}\t#{row[14]}"
> 
> Any feedback is greatly appreciated!

It is very important in a case like this to define the problem clearly.

For example, it would greatly improve the code if you were to clearly say
what the field sizes are. A list of field sizes would be a first step
toward a much more elegant and understandable program. In my solution
below, I guess about some of the field sizes.

Also, you do not want fixed width fields in your output file, as in your
diagram. The first step in the project is to understand that modern
database files use variable width fields, separated by delimiters like
tabs. In your example, you refer to tabs as delimiters, but you still show
the output format with a column scale as though fixed withs were in force.
It's not clear from your diagram that you understand that the output
record's fields won't fall on specific columns, and don't need to.

Sample code:

------------------------------------------

#!/usr/bin/ruby -w

data = [
   "00123 random text   3.1210/20/200610/21/2006   -3.45",
   "00253 more   text 275.8707/01/200606/12/2006   13.46",
   "00254 more   text 777.3407/01/200606/12/2006   14.47",
   "00255 more   text 555.2107/01/200606/12/2006   15.48"
]

out_file = File.open("outfile.txt","w")

data.each do |record|
   fields = [ record[0 .. 4],record[5 .. 17],record[18 .. 23],
   record[24 .. 33],record[34 .. 43],record[44 .. 51] ]
   fields[3 .. 4].each do |field|
      field.gsub!(%r{/},"-")
   end
   out_record = fields.join("\t") + "\n"
   out_file.write out_record
end

out_file.close

----------------------------------------

Output (may wrap when posted):

00123    random text      3.12  10-20-2006      10-21-2006         -3.45
00253    more   text    275.87  07-01-2006      06-12-2006         13.46
00254    more   text    777.34  07-01-2006      06-12-2006         14.47
00255    more   text    555.21  07-01-2006      06-12-2006         15.48

-- 
Paul Lutus
http://www.arachnoid.com