From: Robert Klemme <shortcutter@...>
Date: 2009-09-20T18:20:05+09:00
Subject: Re: Determining if a file is binary or text

On 20.09.2009 00:03, Michael W. Ryder wrote:
> Robert Klemme wrote:
>> On 19.09.2009 01:14, James Masters wrote:
>>> Hi all,
>>>
>>> I need to search text files for a given expression and flag a warning/
>>> error if that expression does not exist.  I'm going to search a large
>>> number of files using the Linux "find" command, so I won't know if
>>> they are binary or text.
>>>
>>> I realize that this can be OS-dependent and can be tricky to
>>> determine.  I was going to use the Linux "file" command which works
>>> well in providing human-readable information about the file; however,
>>> due to a variety of possible file types, I cannot easily determine the
>>> file type without specifying every single possible text file format to
>>> consider.  For example, the "file" command can produce the following
>>> (all of which are ASCII):
>>>
>>> ASCII text
>>> XML document text
>>> Lisp/Scheme program text
>>> ...
>>>
>>> Is there an easy way to do this in Ruby?  After looking around quite a
>>> bit, I thought about looking at a few first lines of the file and
>>> matching against this regular expression:
>>>
>>> # Character class:
>>> # [:print:] Any printable character, including space
>>> line.match(/^[[:print:]]+$/)
>>>
>>> Which I believe could work.  Any comments?
>>
>> Just using a single "+" seems too unsafe to me: you need only three 
>> matching bytes which does not seem too unlikely even for binary files.
>>
>> Some more random thoughts: if you use Ruby to determine file types you 
>> can as well use Find.find to find all files removing the dependency to 
>> an external program.
>>
>> A complete different approach would be to define classes of bytes and 
>> do statistics on the first n bytes from the file, e.g.
>>
>> 32-127, \r, \n, \t printable
>> 0-31 without \n, \t, \r, 128-255 non printable
> 
> I have a problem with considering 128-255 being non-printable.  A lot of 
> these characters are printable, and can be part of text, much like I use 
> Alt-0xxx keys in Pagemaker a lot.

That was just an example.  Of course you can use a different 
classification (for example, adding a third category for 127-255).  I 
assume those characters are comparatively rare in text files so the 
general approach would still work.

>  The other problem with saying a file 
> is not a text file is determining what is meant by a text file.  Is it 
> strictly a file with only Ascii text like a log file, or does it include 
> formated text like word processor file?  Word processing and spreadsheet 
> files contain many characters that are considered non-printable but 
> display as text with the correct program.

I fully agree: the difficult part is in deciding: what is a text file? 
If that has been clarified enough the algorithm for checking should 
become much more obvious.

Cheers

	robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/