Which of these is a meaningless buzzword? “Best practices” or “context-driven testing?”” Or does neither phrase mean much?
James Bach, co-author of the 2001 book Lessons Learned in Software Testing: A Context-Driven Approach, consulting software tester and trainer and
principal of Front Royal, Va.-based Satisfice Inc., says the big idea behind the context-driven school of software testing is that there are no best practices. Bach, who identifies himself as a founder of the context-driven school, says practices can only be good in context, but there are no universal right ways to do software testing.
He contrasts that view with what he describes as an overemphasis on so-called best practices and documentation. He says testing has traditionally been viewed as a clerical activity performed according to prepared scripts by people with little knowledge of what they are doing, or approached in a “ridiculously analytical” way by academics.
Bret Pettichord, director of testing practices at Thoughtworks in Chicago and another co-author of Lessons Learned in Software Testing, says the idea of software testing centres arose from his conversations with other founders of the context-driven school. He describes schools as a means of identifying ways of thinking.
Besides the context-driven school, Pettichord identifies three others. The analytic school, popular among academics, sees testing as rigorous and technical. The factory school or routine school — the most popular in industry, according to Pettichord — emphasizes defining processes that can be carried out by relatively unskilled testers, while the quality school sees testing as part of a broader quality-control process. He adds that test-driven development might be seen as a fifth school.
But Boris Beizer, author of five books on software testing including Black Box Testing, Software Testing Techniques and Software System Testing and Quality Assurance, rejects both Bach’s approach and Bach’s characterization of his own ideas. Asked about the context-driven school, Beizer starts by saying he does not know what that is — nor does he know, he adds, what is the analytic school of which Bach considers him an example.
He is familiar with the Bach’s ideas, though. “I would characterize James Bach as a proponent of what I would call the touchy-feely school of software testing,” scoffs Beizer. As for the term context-driven testing, “that sounds like one of James’ neologisms.” References to different schools of testing are “marketing terms,” Beizer argues.
To Bach, on the other hand, the term “best practices” is simply the label high-priced consultants apply to whatever way they prefer to do things.
“When people talk about best practices,” says Michael Bolton, a Toronto-based associate of Bach and principal of software testing consultancy DevelopSense, “we sometimes believe that they’re essentially trying to end the discussion.”
To hear Bach, Pettichord and Bolton tell it, traditional approaches to software testing put too much stock in documentation, test plans and test scripts. Context-driven testing applies experience and critical thinking to determine the best testing approach in a given situation. The right way to test depends on the software to be tested. Take a system that handles foreign-exchange transactions for a bank, Bolton suggests. “The requirements for that kind of software are going to be vastly different from those for an online dating service.”
If you listen to Beizer, on the other hand, established ways of testing software are based on peer-reviewed literature dating back to the 1950s, but nobody is saying there is one right way to do things. The supposed gulf between the different views almost seems to be disappearing when Beizer observes that “nobody can just follow a prescribed set of rules. That’s a very narrow view of software testing.” But he goes on to say that while there are various techniques to be applied in different situations, there is one absolute rule, “and that is coverage …. If you don’t test a piece of code, you can’t find the bugs that are in it. Doing that does not guarantee that you will find the bugs, but not doing it guarantees that you will not find the bugs.” To that end, Beizer says, it’s vital to document that every part of the code has been tested at least once.
Beizer doesn’t believe the context-driven approach does that. “This is the million monkeys at a million keyboards,” he says of the context-driven approach.
But Pettichord says Beizer is oversimplifying the coverage issue. Trying to ensure full coverage is standard procedure when testing critical systems, he says, but “no code metric exists that actually guarantees you have covered all of the code,” and analytic testers devote too much “in-depth analysis” to trying to do so.
The context-driven school doesn’t reject documentation, or test scripts, or any other technique. “Just because we say there are no best practices doesn’t mean we say there are no good practices in context,” Bach says. “We just aren’t fooled by the idea that if you follow this practice, everything will be okay.”
In fact, Bolton says, a carefully scripted, meticulously documented test strategy is sometimes desirable. “A documentation-oriented approach, when that will satisfy the lawyers, is probably the way to go.”
Comment: [email protected]
