RE: article about testers using Ruby.

Brian:

See comments below...

-----Original Message-----
From: Brian Marick [mailto:[e-mail]]
Sent: Wednesday, April 17, 2002 9:45 PM
To: ruby-talk ML
Subject: RFC: article about testers using Ruby.


I've written a first draft of an article with several ulterior motives. One
is to keep exposing testers to Ruby.

What are the other(s)?

 Would anyone like to comment on my first draft?

Sure.

I'm not assuming any programming background, so I'm simplifying or even
distorting some truths in the interests of understanding.

Hmmm. A strange way to go about teaching... but then again I'm odd: I'm the
kind of guy who likes to align my understanding with the truth, however
horrifying it may be. ;)

 For example, it seemed easier to pretend there's more of a difference
between getters and setters and other kinds of methods than there really is.

Why don't you just say:

---
"Programmatically speaking, 'getters' (e.g., 'getValue()') and 'setters'
(e.g., 'setValue()') are just plain old methods. They are so-called only by
way of social convention to tell the user of a class that he or she has
restricted access to certain private data. One may set private data only by
using a 'setter' method (if provided), and one may acquire the current value
of a certain private piece of data by calling its 'getter' method (if
provided).

"A setter must take a value of the correct type as its only input, and it
has the option of ignoring your input completely if for some good reason it
doesn't like it; a getter must return the current value of the correct type
on demand. If one or both are not provided, it means you have no business
mucking with that private data, by design it's intended to be the class's
internal problem entirely (whether you like it or not). This, in a nutshell,
is what programmer geeks and nutty professors refer to as 'encapsulation' --
controlled access to data intended to be private or internal to the class.

"Getters and setters that you use or write yourself are supposed to be
interfaces to the mysteries inside the black box, like buttons and digital
displays on a telephone -- most people would freak if they saw the tangle of
wires and circuits hidden underneath their sleek, fashionably colored (not
to mention overpriced) designer cordless phones. To call a friend, you need
know only how to dial their number. The hidden stuff inside the phone (most
of which you never see and mercifully have no access to) handles all the
gory details."
---

Testers I talk to (and I do a lot of training in the area of automated
testing) tend to get the point when I explain it that way. The ones that
don't have no business in front of a computer keyboard, as keyboards tend to
short out when exposed to large quantities of dripping drool.

What I'm saying is: Testers really aren't as stupid as they and everybody
else think they are. Well, some are. Then again, so are some developers I've
met in my time. But most are not -- they just never trusted their own
intelligence enough to really apply it on the job. Chalk it up to a bad
childhood and a screwed up education system or something. Bottom Line: Don't
talk down to reach the masses - it may make you feel smarter, but it won't
help the ones that really want to learn make any more sense of this stuff.
Just speak plainly and without pretense -- the folks who will appreciate
your morsels of wisdom will rise to the occasion.

 And I refer to [0, 29, 123] as a "list", since that notion is more likely
to be familiar than "array". Am I going too far?

Gee... I guess I just don't see the big cliff there...

I'm also using win32ole. In the article, I write a test for Word.

I wish Microsoft would do that once in a while... ;)

 This is the first time I've used win32ole and only the second time I've
talked programmatically to Word.

Do you wan't the truth? It kinda shows... :(

But practice makes perfect! (Or is that, 'perfect practice makes
perfect'...? I forget.) ;)

 How's my explanation? Again, I simplified a bit, and I might plain
misunderstand something.

See embedded comments below...

I'd need comments in about a week. I can supply a Word document. I should be
able to provide PDF, but "print to PDF" has suddenly stopped working.
Rebooting might fix it, but I suspect this NT machine knows I'm thinking of
replacing it with a Mac.

Trust me, your NT machine cannot and does not know this. :)

-------------
Shunning the GUI  [I don't like the title. Testers should shun NOTHING. They
should have a zeal about finding new ways to break things that strikes fear
in the hearts of developers (well, weak developers anyway).]

Graphical user interfaces make test automation hard.  [I disagree: they pose
a particular set of challenges that serious automated test engineers need to
know how to overcome, but they do not make automation 'hard' per se. Just
tedious at times.] The problems are well-known. Specialized tools are
required to drive the GUI  [Specialized tools are required to do any kind of
non-trivial testing -- just look at the large number of professional
developer tools, ranging from debuggers, to profilers, to code coverage
analyzers, etc.; does this mean it should not be done?] . Those tools are
often confused by the common programming practice of inventing custom user
interface controls.  [The root cause of this problem vis a vis record/replay
tools is the fact that the only way to hook a GUI object for this purpose is
to register callbacks to various events associated with a given object class
with the Windows event message system, and this requires knowing *how* to
register callbacks to custom windowed objects, which most tools manage for
you. There is no confusion per se on the part of the tool. Most professional
tools, like MI WinRunner (or, for Unix, XRunner), allow you to register
custom classes as you become aware of them and insofar as they behave more
or less like analogous controls, they work fine. Other times more radical
methods are needed, like writing custom object hooks using windows message
cracking macros and other lower level techniques, and registering them with
the tool. Most professional tools provide API's for this. The *only* objects
that record/replay tools choke on (at least on Windows) are non-windowed
(i.e., "graphical") controls. These are nearly impossible to hook in any
reliable way, because windows cannot see the pictures inside them, no matter
how much they may look like standard controls to users. For instance, Access
and FoxPro use graphical controls instead of Windowed controls to minimize
overhead. As a result, they are the bane of every record/replay tool on the
market.]  When they are used in the simplest way, the tools lead to fragile
tests that tend to break en masse when the GUI changes.  [Well, if the
scripts aren't engineered to be reusable, modular, robust, then yes, they
will be fragile. But usually some planning/design and scripting standards
and best practices tend to stiffen those scripts against the elements pretty
well.] Making the tests resistant to change requires elaborate and sometimes
awkward testing frameworks.  [No, just a little sound test engineering.
There is (or should be) nothing awkward about having to think and plan a
little bit before you dive into scripting a test suite.]

Sometimes you must pay this price. Other times, there are alternatives. In
this article, I'll show you one: testing through the program's scripting
interface.

[OK, I'll stop here to explain the enormous gap in your thinking. The
assumption that using an application's scripting interface instead of its
GUI sounds like a nice alternative to the real work of engineering real test
suites with sophisticated record/replay tools against modern-day complex
GUIs, but there is an enormous danger here: You assume that the script
interfaces, particularly COM script interfaces, are flawless and complete.
In reality, they are neither, especially during testing. At all events, you
cannot assume that. And I'll take it a step further: The COM interface
itself must be rigorously tested (and Lord knows Ruby is good for that!),
and cannot be relied upon during testing to provide accurate, reliable,
dependable test data above and beyond their own conformance to whatever
requirements drove their development. Period. I do not trust that a COM
method call to save a file is an adequate substitute for clicking on the
"Save File" button and seeing if something other than "Save File" pops up.
They are apples and oranges and the assumption that these two seemingly
similar things (clicking "Save File" and scripting the "saveFile()" method)
are in fact the same at the code level is naive.]

 My example will be testing Microsoft Word through the interface you'll
often hear described as COM (Component Object Model) or OLE (Object Linking
and Embedding). It's the interface that allows you to embed Excel
spreadsheets in Word documents and vice versa.

[Actually, OLE is a distinct technology built on top of the general-purpose
COM architecture, as is Structured Storage, ADO, and a whole host of
Microsoft technologies.]

 It's also the interface that allows you to write a Visual Basic or Perl
program that uses Word to generate form letters. Since automated tests are
programs, the same interface is dandy for testing.

[Uhhh ... no. That interace itself needs to be tested quite separately from
the GUI behavior, as I explained above, and is not a substitute for GUI
testing.]

This approach can be used with interfaces other than COM. For example, web
services are becoming popular. These applications can be accessed from
anywhere over the internet, and they usually use an interface called SOAP.
[Why do you say this? SOAP is actually relatively new, and there are a bunch
of other far more pervasive protocols out there... read up on it.]  Tests of
such an application would look quite like what you'll see below.

[Testing web services directly via the internet protocol is not a bad
point... it can be done independently of testing the web client GUI to test
the core service functionality... but you still gotta test the GUI before
you deploy it...]

This article will give you four things:

1.      A feel for what a scripting interface looks like.

[The feeling: Warm, tingly, and at times disturbingly like petting a wet
iguana...? ;) ]

2.      An example of how to write tests against one.

[Key distinction: Writing tests AGAINST a scripting interface, vs. Writing
tests USING a scripting interface... two very different propositions.]

3.      Some exposure to the scripting language I think is best for testers,
Ruby. I assume no programming background.

4.      An appreciation of the benefits of shunning the GUI.

 [Scratch this point... Rephrase in the positive: "An appreciation for the
fact that there are often more ways to test an application's functionality,
and more functionality to test, than you might otherwise think." Something
like that. My motto: Shun nothing!!! Plan your attack! Then -- attack! ]

;)

I hope this helped... BTW, I'm co-authoring the "Ruby Developer's Handbook"
by Sams, which focuses on using Ruby as a toolsmith's langauge for general
software engineering, including with particular attention to Ruby as a test
script language. It should be out early next year, as we are in the early
stages of writing it at this time.

-- Bob Calco

--
"Act always so as to increase the number of choices." -- Heinz von Foerster