From: David King Landrith <dave@...>
Date: 2003-04-17T19:36:48+09:00
Subject: Re: Need IO Optimization help

One thing I forgot to copy out of my extension:  After the loop (and 
within error handling routines as appropriate), you'll want to clean up 
by issuing a munmap command:

munmap(buff, len);

On Thursday, April 17, 2003, at 06:29 AM, David King Landrith wrote:

> In my experience, the fastest way to access files (by far) is mmap.  
> I've written some C extensions to Ruby that use mmap to read files, 
> and they is dramatically faster the versions that I wrote using 
> standard, buffered IO.  All of the stuff I have written is for a 
> specific purpose (e.g., a state machine to read character delimited 
> files).  Is there any generic Ruby Extension that gives provides 
> access within ruby to mmap?
>
> Following is some code that copied out of I out of one of my C 
> extensions to give you a quick idea of what it takes to read using 
> mmap:
>
> #include <fcntl.h>      /* read/write flags  */
> #include <errno.h>      /* error numbers */
> #include <unistd.h>     /* open, close functions */
> #include <sys/types.h>  /* typedefs */
> #include <sys/stat.h>   /* stat structures */
> #include <sys/mman.h>   /* mmap */
>
> ....snip...
>
> /* the csv is passed as a parameter in the function I yanked this from 
> */
> int len;
> char *buff;
> char* path = RSTRING(csv)->ptr;
> int fd = open(path, O_RDONLY, 0);
> struct stat buffStat;
> /* error control code */
> if (fd < 0)  {
>     if (errno == EMFILE || errno == ENFILE)  {
>         rb_gc();
>         fd = stat(path, &buffStat);
>     }
>     if (fd < 0)  {
>         close(fd);
>         rb_sys_fail(path);
>     }
> }
> if (stat(path, &buffStat) < 0)  {
>     if (errno == EMFILE || errno == ENFILE) rb_gc();
>     if (stat(path, &buffStat))  {
>         close(fd);
>         rb_sys_fail(path);
>     }
> }
> /* here we actually do the wor */
> len = buffStat.st_size;
> buff = (char*)mmap(NULL, len, PROT_READ, 0, fd, 0);
> close(fd); /* close the file, mmap doesn't need it open */
> if (buff == MAP_FAILED) rb_sys_fail(path); /* one last check for 
> validity */
>
> From here, you just proceed to rifle through the file at a blazing 
> speed; e.g., something like:
>
> while (position < len) {
> 	position++;
> 	....
> 	if (rb_block_given_p() && buff[position] == "\n") {
> 		rb_yield(...);
> 	}
> }
>
> etc.
>
> Pass the line terminator as a parameter in the function, and yield the 
> accumulated result in between lines.
>
> The only problem here is that mmap is part of the posix standard, so 
> that it will work fine under Unix, but I have no idea how well (or 
> even how, for that matter) it will work with Windows.
>
> I hope this was useful,
>
> Best,
>
> Dave
>
> On Thursday, April 10, 2003, at 12:03 PM, Jim Freeze wrote:

-------------------------------------------------------
David King Landrith
   (w) 617.227.4469x213
   (h) 617.696.7133

One useless man is a disgrace, two
are called a law firm, and three or more
become a congress   -- John Adams
-------------------------------------------------------
public key available upon request