From: Vidar Hokstad <vidar.hokstad@...>
Date: 2008-03-07T01:30:03+09:00
Subject: Re: Better performance than native unix commands?

On Mar 4, 1:10 am, Ma Sa <ntw...@gmail.com> wrote:
> I have an issue with nfs and listing a large amount of files that I am
> wondering if I can solve with ruby. I know that ruby has better
> performance in some instances than a bash shell script so I am wondering
> if anyone has experience or some examples of how this might work...
>
> I have a directory with a large amount of files that is mounted over
> nfs. The network is gigabit and so are the nic's. The servers are very
> high end so the hardware / network performance is not the bottleneck.
> However, sometimes it can take a very long time to list out the files
> contained within a directory.
>
> I will do an ls and it might take 5 minutes to retrieve a response.

Have you tried doing the same ls locally on the server? How many files
are in the directory you try to run ls on?

If it's a huge number of files and it's slow on the server too, then
the problem is likely that you use a filesystem (such as ext2) that
handles large directories poorly. If so, using "ls -f" should _start_
returning results faster, though it'll still be slow. It's a good
indicator of the source of the problem.

If that's the problem, then switching filesystem or reorganizing the
file structure are your best bests - ext2fs is completely unsuitable
for solutions with large number of files per directory.

Second, "strace -c" is your friend. It'll show you what syscalls takes
up the time. If it's spending all it's time in readdir() then the
problem is on the remote server or the network.

Third, "tcpdump" is your friend. Use it to see if the network traffic
to the NFS server is going at reasonable rates.

Ruby is very clearly _not_ going to be faster than "ls" with the right
options - with "ls -f" for example, ls will mostly just do readdir()
calls and spit out the results. There aren't really many ways of
making it much faster other than connecting to the remote server and
running "ls" directly on it.

Vidar