From: Reid Thompson Date: 2006-11-30T05:55:24+09:00 Subject: Re: Splitting a CSV file into 40,000 line chunks On Thu, 2006-11-30 at 04:04 +0900, James Edward Gray II wrote: > On Nov 29, 2006, at 12:56 PM, Drew Olson wrote: > > > Here's another question: in some cases I need to sort the > > file before splitting it (in this case sorting by the 4th cell in each > > row). However, the current file I'm trying to sort and split is around > > 76 MB and ruby fails when trying to store the CSV as an array. The > > code > > and output are below. How else can I go about this? > > Hmm, that's a good question. > > Perhaps you can collect just the key values in an Array and then use > those to reorder the lines bit by bit. That's not going to be fast > with any library helping you, but I don't have a better idea. > > James Edward Gray II sort -n +4 out or sort +4 out rthompso@jhereg:~$ head -50 out this is a test 10202 this is a test 23990 this is a test 11056 this is a test 9606 this is a test 28590 this is a test 18264 this is a test 12902 this is a test 12856 this is a test 27571 this is a test 5495 this is a test 15965 this is a test 22229 this is a test 18865 this is a test 31339 this is a test 21913 this is a test 406 this is a test 8602 this is a test 5329 this is a test 10048 this is a test 6458 this is a test 20069 this is a test 19771 this is a test 21844 this is a test 24719 this is a test 30894 this is a test 25239 this is a test 9900 this is a test 1727 this is a test 12042 this is a test 20832 this is a test 23735 this is a test 28768 this is a test 10283 this is a test 390 this is a test 10480 this is a test 1337 this is a test 2745 this is a test 26398 this is a test 32288 this is a test 3797 this is a test 22251 this is a test 458 this is a test 14679 this is a test 29642 this is a test 19943 this is a test 26342 this is a test 24232 rthompso@jhereg:~$ sort +4 out this is a test 0 this is a test 10 this is a test 10 this is a test 100 this is a test 100 this is a test 100 this is a test 1000 this is a test 1000 this is a test 10001 this is a test 10001 this is a test 10005 this is a test 10006 this is a test 10006 this is a test 10007 this is a test 10007 this is a test 10008 this is a test 10008 this is a test 10009 this is a test 10009 this is a test 1001 this is a test 10010 this is a test 10011 this is a test 10012 this is a test 10012 this is a test 10012 this is a test 10012 this is a test 10012 this is a test 10013 this is a test 10014 this is a test 10014 this is a test 10015 this is a test 10017 this is a test 10017 this is a test 10017 this is a test 10018 this is a test 10019 this is a test 1002 this is a test 1002 this is a test 10021 this is a test 10026 this is a test 10026 -------------------- OR rthompso@jhereg:~$ sort -n +4 out this is a test 0 this is a test 2 this is a test 3 this is a test 3 this is a test 4 this is a test 7 this is a test 10 this is a test 10 this is a test 12 this is a test 13 this is a test 14 this is a test 14 this is a test 15 this is a test 16 this is a test 16 this is a test 20 this is a test 21 this is a test 24 this is a test 25 this is a test 28 this is a test 29 this is a test 29 this is a test 29 this is a test 30 this is a test 30 this is a test 31 this is a test 32 this is a test 32 this is a test 34 this is a test 35 this is a test 35 this is a test 35 this is a test 36 this is a test 36 this is a test 37 this is a test 37 this is a test 38 this is a test 38 this is a test 41 this is a test 41 this is a test 41 this is a test 41 this is a test 41 this is a test 41 this is a test 42 this is a test 42 this is a test 42 this is a test 43 this is a test 45 this is a test 46 this is a test 46 this is a test 47 this is a test 47