From: Dan Fitzpatrick <dan@...>
Date: 2008-03-04T14:44:07+09:00
Subject: Re: Better performance than native unix commands?

Ma Sa wrote:
> I have an issue with nfs and listing a large amount of files that I am
> wondering if I can solve with ruby. I know that ruby has better
> performance in some instances than a bash shell script so I am wondering
> if anyone has experience or some examples of how this might work...
>
> I have a directory with a large amount of files that is mounted over
> nfs. The network is gigabit and so are the nic's. The servers are very
> high end so the hardware / network performance is not the bottleneck.
> However, sometimes it can take a very long time to list out the files
> contained within a directory.
>
> I will do an ls and it might take 5 minutes to retrieve a response.
>
> Would this be faster if I were to do the same ls function using the ruby
> api?
>   
What OS are you running? Something seems like it's not working. I run 
FreeBSD 6.2 and I have almost 200,000 files in one folder on an NFS 
mount and ls is pretty much an instant response. I doubt Ruby will be 
any faster than a compiled C app (ls). Could be a firewall issue (pf and 
nfs do not get along to well for example), locking issue, network 
congestion or misconfiguration. You could try to see where the 
bottleneck is by doing the following:

On the NFS server:

time ls > temp_file

Make sure the temp file is on the server too. This will tell you if the 
bottleneck is with NFS/network or the server's file system.

On the client (with temp_file on the server):

time ls
time cat temp_file

This will tell you if it is the ls command or the NFS/network time to 
send the data over the network.

Copy the temp_file to the client and see what the speed is on the client.

time cat temp_file


Here are my numbers for reference:

ls | wc -l
197621

time ls > /dev/null (local)
real    0m0.669s
user    0m0.550s
sys    0m0.114s

time ls > /dev/null (over nfs)
real    0m0.716s
user    0m0.606s
sys    0m0.070s


Dan