From: RichardOnRails <RichardDummyMailbox58407@...>
Date: 2007-11-21T11:05:02+09:00
Subject: Re: False positives in editing data

On Nov 19, 5:47 am, Raul Parolari <raulparol...@gmail.com> wrote:
> RichardOnRails wrote:
> > Below is a 33-line program that analyzes a set of names.
> > The names proper are prefixed with an optional set of period-separated
> > numbers.
> > The analysis checks for the following, and flags violations:
> >   1. A number may not start with zeros.
> >   2. No more that two numbers may be prefixed
> >   3. Names proper must begin with a non-digit
> > ..
> > I'd welcome any ideas on how to correct my mistake(s), especially with
> > suggestion for improved style.
>
> Richard,
>
>   even after your correction, there are things that we can improve in
> your program; the problems I see are:
>
> 1) the array of numbers for the prefix is configured with the maximum
> (MLN) nr of components, rather than with the entries that are present
> within the line. This leads to complicated code (evaluating $n variables
> beyond the valid ones, etc).
> 2) the regular expression sName for the numbers would need to be updated
> if MLN changes (in fact, it currently contains more values than needed).
> 3) the script does not catch if the prefix contains characters other
> than digits and dots. Try eg, prefix '2.!1' ; the prefix will be
> recognized as 2, and consider the entry valid.
>
> So the problem is that the code is very complex, and fragile; the good
> news is that we can have a simpler and more robust solution; let us
> divide the problem in small steps:
>
> 1) we first collect everything until the first letter (not included); we
> will consider this the Prefix.
> 2) we verify that the Prefix has the format that we want: numbers
> separated by dots.
> 3) we then split the Prefix into an array of Numbers.
> 4) we verify that the Array size does not exceed the MLN max size.
> 5) we then check if any of the entries begins with 0.
> *) oh, and we will use the /x modifier to make the regexps legible
>
> Each of the previous steps will be one line of code, so we should have a
> very small script; I will comment the lines with the same number used
> above, so you can map every statement with the 'battle plan' above:
>
> class FormatError < Exception
> # used right now only for unrecoverable errors
> end
>
> input.each do |line|
>
>   # debug info
>   puts "\n" + "="*10 + "DBG", line, "="*10+ "DBG\n"
>
>   # 1) collect everything (not greedy) until first letter
>   if  line =~ /^ (.*?) [a-zA-Z] /x
>
>     prefix = $1
>     puts "prefix= '#{prefix}'"
>
>     # 2) validate format: numbers separated by digits
>     if prefix =~ /^ (?:\d+ [.]?)+ $ /x
>
>       # 3) collect the numbers in an array
>       arr = prefix.split('.')
>
>       # 4) verify max array size
>       puts "  (Error: depth of hierarchy > #{MLN}!)" if
>                                        arr.length > MLN
>
>       # 5) any number begins with 0?
>       puts "  (Error: a nr begins with 0!)  " if
>                               arr.any? { |v| v =~ /^0/ }
>
>       print "  Numbers: ", arr.join(', '), "\n"
>
>     else
>       raise FormatError, "Prefix format NOT nr.nr.nr.."
>     end
>
>   else
>      print "the line has no prefix (option or error?)"
>   end
>
> end # input
>
> This program generated, tested with the same data you used, this output:
>
> ==========DBG
> 05Topic 5
> ==========DBG
> prefix= '05'
>   (Error: a nr begins with 0!)
>   Numbers: 05
>
> ==========DBG
> 1Topic 1
> ==========DBG
> prefix= '1'
>   Numbers: 1
>
> ==========DBG
> 2.002.1Topic 2.2.1
> ==========DBG
> prefix= '2.002.1'
>   (Error: depth of hierarchy > 2!)
>   (Error: a nr begins with 0!)
>   Numbers: 2, 002, 1
>
> ==========DBG
> 2.1Topic 2.1
> ==========DBG
> prefix= '2.1'
>   Numbers: 2, 1
>
> ==========DBG
> 2.2.02Topic 2.2.2
> ==========DBG
> prefix= '2.2.02'
>   (Error: depth of hierarchy > 2!)
>   (Error: a nr begins with 0!)
>   Numbers: 2, 2, 02
>
> As you can see, it seems to work well (it also catches double errors,
> which is nice). Notice that if the max array size had to be changed, we
> just need to modify the MLN value.
>
> I hope this made sense, and that it helped.
> Good luck!
>
> Raul
> --
> Posted viahttp://www.ruby-forum.com/.

Hi Raul,

Thanks for your response.

I like your "battle plan".  It works well for your exposition here.  I
always have to prepare a high-level one when I propose a software
solution to a user.

Harry posted a succinct illustration of applying Ruby to this kind of
validation,  which I appreciated.  I also appreciate your slightly
longer approach  because I need this structure
so that I can make use of data for my purpose beyond validation.

I especially appreciate your showing me how a regex can be written to
handle an arbitrary number of dot-separated numbers (rather than hard-
code distinct sub-expressions).

>   if  line =~ /^ (.*?) [a-zA-Z] /x

I have one question about this regex.  I have a book that I bought in
2002 but never read until today: "Mastering Regular Expressions", 2nd
Ed., from O'Reilly.  I haven't been able to find any reference in
there to a question-mark following ".*".

I thought I could simply remove the question-mark.  That caused the
match to fail and yield the programmed error msg.  I tried omitting
the question-mark and add a closing "$" in the regex.  That made the
parsing fail.  So, your question mark is clearly working,  but HOW?

I'm fully on board with the rest of your code.  Many thanks for it
all!

Regards,
Richard