From: g_f <8si.greg@...>
Date: 2009-09-20T00:27:12+09:00
Subject: Re: Determining if a file is binary or text

By convention, source and object files use standardized file-type
extensions, which should help you weed out files to ignore.

As a starting point ask the developers what file-type extensions
they're using. As a second check, run something like the following
commands at the top of the path you'll be checking:

     find . | xargs -n1 basename | egrep '\.\w+$' | awk -F. {'print
$2'} | sort -u

to give you a list of possible extensions, then check those too.

Use "file" and "file -i" to do a best-guess once you've narrowed your
possibilities. Both use "magic" files which define where file should
look inside a target file to determine what type it is. They are
fallible though and you can get false positives. Do a "man magic" from
the command-line on your Linux box for more info.

Also, be careful assuming only binary files have \x00 bytes or high-
order ASCII. Old text files that have migrated from other systems
could have them, as could files where someone ALT+fat-fingered on the
keypad as could a source file coming from a non-english speaking
nation where the developer used variable names in his native language.
You just never know what you'll find in those pesky source files.