Quality of testing always matters. Interview with Michael Bolton. Part I

Michael Bolton is known to be a real Software Testing classic.

One fact tells everything – he has over 20 years of experience in the computer industry testing, developing, managing, and writing about software. Michael delivered a lot of workshops, tutorials, and conference presentations on Rapid Software Testing and other aspects of testing methodology on ALL FIVE CONTINENTS.

He is known to be a regular columnist in Better Software Magazine (formerly Software Testing and Quality Engineering) since 2005. He was an invited participant at the 2003, and 2005-2009 Workshops on Teaching Software Testing in Melbourne, Florida (hosted by Cem Kaner, James Bach, and Scott Barber) and a founding member (with Fiona Charles) of the annual Toronto Workshops on Software Testing. He is also the Program Chair for TASSQ, the Toronto Association of System and Software Quality.

We are happy to announce that today Michael is a honorable guest in our blog and we start to discuss the specifics of quality and testability.

a1qa: Having over 20 years of experience in computer testing industry, developing, managing, and writing about software, what is “quality” for you? Can a product/system be estimated as having 100% level of quality?

My notion of quality comes from Jerry Weinberg: “quality is value to some person(s)” (Quality Software Management, Vol. 1: Systems Thinking; also available as two e-books, “How Software is Built” and “Why Software Gets in Trouble”). James Bach and I have added “who matters” to make explicit some of the things that Jerry presents in the book. When you think of quality as “value to some person who matters”, it should be absolutely clear that evaluating the quality of something starts with a decision about whose values matter.

Quality is always relative to some person. A product can have terrible problems in it, and yet still satisfy some people all of the time and most people most of the time. The idea of quantifying quality at all seems quite silly to me; what would it mean to have a 77% level of quality for a product? What would it mean to have a 100% level of quality? Until you can establish that, and until you can establish a scale that applies to widely differing people with widely differing sets of values, the number isn’t going to tell you anything meaningful. Numbers won’t do the trick there; you need a qualitative description.

What you can do, as a professional working in software testing outsourcing, is to identify important problems in the product that threaten its value such that the product owner (who is ultimately the person who makes decisions about quality on a project) might be unwilling to ship it. But that’s not usually a question of numbers to any serious degree. Instead, it’s a collection of qualitative criteria framed in terms of stories. Both testing and reporting problems are built on the idea of composing, editing, narrating, and justifying stories about our products. Sometimes the stories can be illustrated by numbers, but most of the time stories are about problems that would cause loss or harm or annoyance, loss of value, to somebody who matters. (See “Braiding the Stories (Test Reporting Part 2)”

a1qa: “Testability” is it just a trendy word or the product attribute that simplifies testers` task?

Testability is not just a trendy word. Indeed, it’s one of the quality criteria that I referred to above. In addition to reporting on problems with the product (we call those bugs), the tester must also report on issues — anything that threatens the value of the testing work; anything that makes testing harder or slower. A product that is not very testable will give more time and more opportunity for the bugs to survive below our levels of awareness. There are other dimensions of testability too: figuring out how we know a problem when we see one, and identifying aspects of the tester, or the project, or our notion of quality. There’s more on that here.

In the next post we continue the discussion with Michael and touch upon Rapid testing and test automation.