From: Brian Candler <B.Candler@...>
Date: 2003-04-17T23:45:38+09:00
Subject: Re: Need IO Optimization help


--Dxnq1zWXvFF0Q93v
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Thu, Apr 17, 2003 at 11:22:33PM +0900, Jim Freeze wrote:
> It's the old trade-off between performance, maintainability and
> development time. With ruby and a C library to speed up the routine,
> we have lowered the cost in each category. A big win in my book.

Absolutely. But I liked the idea of "beating Perl" as being the underlying
objective, rather than meeting any other particular criteria :-)

If you're still interested in mmap vs read you could try running the
attached program, once using read_normal and once with read_mmap, against
your 260MB file, and take the difference (in seconds, not as a percentage),
which is similar to the kind of difference you could expect (in seconds) in
your main application.

However, once you run this repeatedly and you have enough RAM then your file
is likely to remain in cache anyway, so this may not be representative of
the case where you run the job once (where I/O is likely to dominate).

Calculating a sum of the contents is a simple way to prove that the main
loop actually reads the bytes and hasn't been optimised away...

Regards,

Brian.

--Dxnq1zWXvFF0Q93v
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="test1.c"

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>

static long read_normal(int fd)
{
  long sum = 0;
  char buf[65536];
  long chunk;
  while ((chunk = read(fd, buf, sizeof buf)) > 0) {
    char *p;
    for (p = buf; p < buf+chunk; p++) {
      sum += *p;
    }
  }
  return sum;
}

static long read_mmap(int fd)
{
  long sum = 0;
  long fs;
  struct stat sb;
  char *fp, *p;
  
  if (fstat(fd, &sb) < 0) {
    perror("fstat");
    exit(2);
  }
  fs = sb.st_size;
  fp = mmap(0, fs, PROT_READ, 0, fd, 0);
  for(p = fp; p < fp+fs; p++) {
    sum += *p;
  }
  return sum;  
}

int main(int argc, char *argv[])
{
  int fd;
  long sum;
    
  if (argc != 2) {
    fprintf(stderr,"Usage: %s filename\n",argv[0]);
    return 1;
  }
  fd = open(argv[1], O_RDONLY);
  if (fd < 0) {
    perror("open");
    return 2;
  }
  sum = read_normal(fd);   /* CHANGE HERE */
  fprintf(stderr,"sum %ld\n", sum);
  return 0;
}

--Dxnq1zWXvFF0Q93v--