From: "Michael W. Ryder" <_mwryder@...>
Date: 2009-09-20T07:05:09+09:00
Subject: Re: Determining if a file is binary or text

Robert Klemme wrote:
> On 19.09.2009 01:14, James Masters wrote:
>> Hi all,
>>
>> I need to search text files for a given expression and flag a warning/
>> error if that expression does not exist.  I'm going to search a large
>> number of files using the Linux "find" command, so I won't know if
>> they are binary or text.
>>
>> I realize that this can be OS-dependent and can be tricky to
>> determine.  I was going to use the Linux "file" command which works
>> well in providing human-readable information about the file; however,
>> due to a variety of possible file types, I cannot easily determine the
>> file type without specifying every single possible text file format to
>> consider.  For example, the "file" command can produce the following
>> (all of which are ASCII):
>>
>> ASCII text
>> XML document text
>> Lisp/Scheme program text
>> ...
>>
>> Is there an easy way to do this in Ruby?  After looking around quite a
>> bit, I thought about looking at a few first lines of the file and
>> matching against this regular expression:
>>
>> # Character class:
>> # [:print:] Any printable character, including space
>> line.match(/^[[:print:]]+$/)
>>
>> Which I believe could work.  Any comments?
> 
> Just using a single "+" seems too unsafe to me: you need only three 
> matching bytes which does not seem too unlikely even for binary files.
> 
> Some more random thoughts: if you use Ruby to determine file types you 
> can as well use Find.find to find all files removing the dependency to 
> an external program.
> 
> A complete different approach would be to define classes of bytes and do 
> statistics on the first n bytes from the file, e.g.
> 
> 32-127, \r, \n, \t printable
> 0-31 without \n, \t, \r, 128-255 non printable
> 

I have a problem with considering 128-255 being non-printable.  A lot of 
these characters are printable, and can be part of text, much like I use 
Alt-0xxx keys in Pagemaker a lot.  The other problem with saying a file 
is not a text file is determining what is meant by a text file.  Is it 
strictly a file with only Ascii text like a log file, or does it include 
formated text like word processor file?  Word processing and spreadsheet 
files contain many characters that are considered non-printable but 
display as text with the correct program.


> Then determine based on ratio of occurrences.  Of course, that approach 
> can also be tricky...
>
> Kind regards
> 
>     robert
>