From: "Saji N. Hameed" <saji@...21.net>
Date: 2008-11-10T17:28:02+09:00
Subject: Re: Moving large amount of files, 1.750.000+

* Sebastian Newstream <abeansits@gmail.com> [2008-11-10 17:11:08 +0900]:

> Randy Kramer wrote:
> > On Sunday 09 November 2008 01:35 pm, Randy Kramer wrote:
> >> I wouldn't be too worried about the 13 hours, you've got a lot of data 
> >> to move.
> > 
> Your probably right, I will start the job on a friday evening and let it 
> take it's time.
> 
> > PS: I wish I had added: Since all you're doing is copying files, do it
> > from the CLI (as Robert suggested)--no need to involve any programming
> > language which is just added overhead.  Then let us know how many hours
> > it takes that way, for comparison.
> > 
> > Randy Kramer
> 
> Your probably right about this as well, but I can't backout of the Ruby 
> corner now. I already opened my mouth about Ruby to much now, if I 
> change my method now  it will make Ruby look realy bad. =(
> 
> This is what I succeded with:
> * I removed all of the console prints for each file. (This lowered the 
> time with about 20s! I had no idea that output was so demanding.).
> * I kept the filehandle open for writing to the process.log.
> * I also removed any line of unessesary code in the critical part of my 
> application.
> This lowered the time to around 17s. I will now try to run the test on 
> the right environment.
> 
> Of course I will post the results here for your guys to se.
> Thanks again for your time.
> //Sebastian


This may be a naive suggestion. It may be worthwhile to see if there
is a benefit in parallelize the process using Threads (split the transfer
jobs among multiple threads ??? ... ) 

saji
-- 
Saji N. Hameed

APEC Climate Center          				+82 51 668 7470
National Pension Corporation Busan Building 12F         
Yeonsan 2-dong, Yeonje-gu, BUSAN 611705			saji@apcc21.net
KOREA