From: Ben Giddings <bg-rubytalk@...>
Date: 2003-09-05T03:43:04+09:00
Subject: Re: [OT] Unit Tests and Encapsulation

Aredridel wrote:
> Fake a 500.  Fake a 404.  Fake a timeout -- throw an exception that
> could happen in your code.  Send it bogus data.  Send it good data.  All
> should pass tests.

Sure that's easy.  But what happens if you can't test the "success" case 
reliably because it depends on things outside your control?

In my example, I can:
1) Test to see if I can generate a URL
2) Test to see if the URL is syntactically valid
3) Test to see if I can handle all the various ways in which accessing 
that URL can fail
4) Test to see if I can handle all the various types of data I expect 
the URL to return if it does respond

But what if I want to test to see if the URL I generated points to a 
valid web page?  The only way I know of to test that is to go to that 
URL and see if there's a web page there.  The problem is that you can't 
prove that by counterexample.  Just because I get an error while trying 
to access the page doesn't mean the URL is invalid, it could just be 
another type of error.

If I get a DNS lookup error, it could be that the hostname in the URL is 
wrong, or it could just be a hiccup in the DNS server.  If I get a 404 
error, it could be that a transparent proxy along the way broke, or that 
the server I'm contacting was temporarily misconfigured.

Is there a good way of testing "does the URL I generated point to a 
valid web page"?  For the more general case, is there a good way of 
testing when a failure doesn't necessarily mean that your code is wrong?

Ben