From: Robert Klemme <shortcutter@...>
Date: 2009-09-19T20:55:05+09:00
Subject: Re: Determining if a file is binary or text

On 19.09.2009 01:14, James Masters wrote:
> Hi all,
> 
> I need to search text files for a given expression and flag a warning/
> error if that expression does not exist.  I'm going to search a large
> number of files using the Linux "find" command, so I won't know if
> they are binary or text.
> 
> I realize that this can be OS-dependent and can be tricky to
> determine.  I was going to use the Linux "file" command which works
> well in providing human-readable information about the file; however,
> due to a variety of possible file types, I cannot easily determine the
> file type without specifying every single possible text file format to
> consider.  For example, the "file" command can produce the following
> (all of which are ASCII):
> 
> ASCII text
> XML document text
> Lisp/Scheme program text
> ...
> 
> Is there an easy way to do this in Ruby?  After looking around quite a
> bit, I thought about looking at a few first lines of the file and
> matching against this regular expression:
> 
> # Character class:
> # [:print:] Any printable character, including space
> line.match(/^[[:print:]]+$/)
> 
> Which I believe could work.  Any comments?

Just using a single "+" seems too unsafe to me: you need only three 
matching bytes which does not seem too unlikely even for binary files.

Some more random thoughts: if you use Ruby to determine file types you 
can as well use Find.find to find all files removing the dependency to 
an external program.

A complete different approach would be to define classes of bytes and do 
statistics on the first n bytes from the file, e.g.

32-127, \r, \n, \t printable
0-31 without \n, \t, \r, 128-255 non printable

Then determine based on ratio of occurrences.  Of course, that approach 
can also be tricky...

Kind regards

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/