Re: Design contracts and refactoring (was Re: mathn: ugly warnings)

On 1/13/06, mathew <[e-mail]> wrote:
> Dean Wampler <deanwampler gmail.com> writes:
> > The redundancy part (or DRY in Ruby parlance) I
> > was referring to is the goal of only writing the requirements once and
> > it's better to write them in an executable format so they have
> > enforcement power.
>
> But to go back to the problems I was pointing out:
>
> a) In the real world, the requirements do not constitute the entirety of
> the supported functionality.

The XP philosophy is that if you implement more functionality than is
required, then you are wasting time and resources. Deliver what's
essential today. If you will need the extra stuff eventually (often a
big "if", they often talk of YAGNI: "you ain't gonna need it"), then
implement it then. In otherwords, you should strive to only implement
what's required in the current "scope".

> b) In the real world, actually unit testing the complete scope of the
> requirements requires unreasonable amounts of time and effort.

No doubt it can be hard, but even for math functions, you can
decompose the infinite # of possibilities into finite sets of unique
"classes", determined in large part by the idiosyncrasies of the
finite machine limitations....

> Consider '+'. Requirements are that it add any two things which
> constitute numbers. Trouble is, you can't even test every possible pair
> of floating point numbers, let alone the integers and rationals. Any
> test of N pairs that you do will be statistically insignificant.

... You don't need to test every possibility. The "classes" of
interest are additions that fall well within the n-bit limitations,
and various classes of corner cases like around zero, 2**n + 2**n,
etc., etc. I'm not an expert on this, but certain the implementers of
IEEE floating point standards have a bounded set of tests they use
that are sufficiently comprehensive to give trustworthy results. Not
easy to do; you may recall one of the early pentium chips had some
embarassing floating point errors, but in principle it's doable.

> > So, the acceptance tests used by customers to confirm that
> > requirements are met may be written in a documentation tool from which
> > test code is generated.
>
> Right. Unit tests are not the requirements; they are acceptance tests
> used to confirm that the requirements have been met. There's a subtle
> but very important difference.
>
> Similarly, requirements are not specifications. The specifications are
> (hopefully) derived from the requirements, but they typically describe a
> superset of the functionality strictly required, and almost always
> describe a superset of the functionality tested.

I would say that the specification shouldn't create a superset, but
just itemize all the details that are implied by high-level
requirements statements. So, it's more verbose than the requirements,
but it doesn't add anything, if possible. Yes, you may end up having
to implement some extra stuff in order to fully cover the "stated"
requirements, but you should avoid extrapolation. It's back to my
remark above about YAGNI...

>
> For example:
>
> Requirements: method to add two vectors of Floats.
>
> Specification (as implemented): method adds two vectors of objects which
> support '+'.
>
> Unit tests: take a sample file of vectors of floats, add them in pairs,
> compare against the file of correct output.
>
> All perfectly reasonable. However, the unit tests do not tell you
> whether anything other than floats will work with the method. That's
> perfectly reasonable, because working with something other than floats
> wasn't a requirement. However, it's also perfectly reasonable that the
> first implementation happens to work with integers, rationals, and so
> on--because the power of Ruby is that generic programming doesn't take
> extra effort. In fact, it would take extra effort to prevent the code
> from working with other types.
>
> So, if you only provide unit tests and no documentation, users have to
> guess. They might look at the unit tests, and guess that the method only
> works with floats. End result: code bloat because they go and
> unnecessarily find or implement an integer vector add.

In this case, I would write my own test suite for the new data type
and only after proving compatibility would I use it thusly....

> Or, they might look at the code, and see that it works with Complex.
> Then they go and write a ton of scientific visualization software that
> relies on the fact. Then you want to refactor the method to use CBLAS
> for speed, and it breaks NASA, and there are howls of anguish.

... and I'd try to convince the original implementer to include my
tests in his suite so his subsequent upgrades don't gratuitously break
my work!

> Hopefully I'm getting the problem across better this time around. I am
> *not* discussing unit tests vs documentation, or unit tests vs
> contracts, or unit tests vs anything else, or documentation vs anything
> else. At no point have I said that unit tests are anything other than
> essential. My point is simply that a documented API is *also* essential,
> and at a minimum it should describe the functionality that is
> *supported*, as opposed to that which is tested or that which happens to
> work. Leaving people to guess the supported API from the unit tests (or
> worse, the code) is a flawed approach which has led to problems refactoring.

I don't disagree that supproting documentation is often helpful,
especially for giving new users the "gist" of the module. However, the
beauty of the XP approach is the economy of minimizing duplication and
putting your energy into specifying requirements, specifications, etc.
into an executable form so you prove conformance. Yes, code is harder
to read, but if you can read it, you've got "proven" examples of usage
and also hopefully examples of how to break your app. by using the
module in unsupported ways!

> In a nutshell:
>
> documentation = what is *supported* or *supposed to work*
> unit tests = what is *actually tested*
> code = what *happens to work*

I'm going to trust the tests, if they are good and comprehensive.
Otherwise, the code itself. I'll read the docs for general
understanding, but take them with discretion, knowing that they often
aren't kept up to date and may not reflect the actual behavior! I
won't believe a statement about what's supported unless the tests
prove it.

> What is *tested* != what is *supported*, because you can't test
> everything. I can't feasibly test every possible RSS feed against my RSS
> parser, for example.
>
> What is *supported* != what *happens to work*, not least because Ruby
> code tends to be generic by default. Even ignoring the generic issue,
> though, just because my RSS parser accepts a particular malformed feed
> today, doesn't mean I'm necessarily going support that malformed feed
> after I refactor the parser.

This is anologous to the math problem; there are a finite number of
classes of possible behavior that you can test (valid and invalid
cases...).

> And when I'm using libraries, though I do also care about what's tested,
> my primary concern is what's supported. I care about knowing that
> something I use today won't change in an unexpected way and break my
> application. e.g. Rake, Yaml, ...
>
>
> mathew
> [ And as usual, as soon as I get into a discussion, the mailing list
> software stops sending me mail. ]

Mathew,
This has been a good discussion. We'll probably have to agree to
disagree on some points. I need to get back to work... and give the
rest of this list's readers a break!

Yours,
dean

--
Dean Wampler
http://www.aspectprogramming.com
http://www.newaspects.com
http://www.contract4j.org