Blog

Never let a metric go unsupervised. Interview with James Bach

With this blog post a1qa starts interviews series with famous persons of software testing and quality assurance industry. James Bach honored us to be the first one. He is the expert who speaks at prestigious conferences, and an ardent trainer who strives to bring benefit to his students.
18 February 2014
Interviews
The article by a1qa
a1qa

With this blog post a1qa starts interviews series with famous persons of software testing and quality assurance industry. James Bach honored us to be the first one. He is the expert who speaks at prestigious conferences, and an ardent trainer who strives to bring benefit to his students.

James Bach is the person for whom Testing is not only a job … it`s his life. Previously working for Apple and Borland; now he is leading Satisfice Inc., a teaching and QA consulting company. James is the one who is behind of Exploratory Testing concept and the Context Driven School of Testing. His testing blog breaks all records of popularity over the globe. We are delighted to host James in our blog; see James`s blog.

a1qa: James, thank you very much for being our guest. The last post in your blog was about measuring testing quality. It’s not a secret that a developer and project manager usually want to know how good their code is. And test activities often result in test report from responsible QA engineer with a quality analysis and a quality estimates. How do you estimate quality? Is that verbal, numeric attribute or instinct feeling for you?

James Bach (J.B.): All reasonable people assess quality of software the same way: weighing the evidence in their minds. It’s what juries do, it’s what voters do, it’s what political and military leaders do, it’s what everyone does when assessing anything complicated and important. Anyone who tries to specify software quality substantially in terms of numeric attributes is either joking, incompetent, or a fraud. Management is primarily responsible for weighing the evidence about the product. There are no metrics that can take this responsibility away from management. If they attempt to shift their responsibility onto metrics then responsible people must resist that.

My job as a tester (I prefer not to say “QA”) is to collect, arrange, filter, and present the evidence that management needs. I might use metrics as part of that evidence. What metrics I choose is a decision that is completely dependent on the situation. However, I use metrics with skepticism. Numbers are too attractive to people who want decisions to be simple. I never let a metric go unsupervised into the hands of management. And I almost never use any metrics associated with test cases (there are a few, rare exceptions), since they mean nothing. I never reduce quality to any set of numbers.

a1qa: What are your expectations with regards to QA and trends you foresee in testing? Will test automation ever replace manual QA?

J.B.: Skilled testers have always been a tiny minority. Now, at least, there is good international community to support the growth of skilled testers. But the community of serious testers is under constant attack from different quarters. Some Agilists are bored by testing and want all testers to be programmers. Many old factory-style testing consultants are threatened by the new wave of intellectual testers, which is why they have conspired to create the new ISO testing standard that was obsolete long before it was even officially created. Certain large companies famously deny they even have testers.
I think that the days of poorly skilled testers who follow test cases has largely passed. Those people will get less and less work.
As for automation, well testing cannot be automated. There is no such thing as automated testing. I realize that your company publishes blog posts that speak of automation, but that’s not testing– that is fact checking. Unless fact checking is guided and designed and managed by people who know how to test, it will remain an expensive waste of time. Today, most of what is called test automation delivers very little value, relative to the cost.
I am a tester. Of course I’m interested in facts, but I also have the skills to analyze and interpret evidence. I know how to design experiments. Those skills cannot be automated, although I do use tools to help me test.

a1qa: QA to Dev ratio as 1 to 3: do you believe in it? What’s your advice on QA team size? What factors do you consider when setting up a team?

J.B.: Tester/Developer ratios don’t mean anything.
My advice is to hire or train fantastic testers until the critical testing work is getting done quickly enough. I generally start by hiring one tester and go from there. This is the same way developers are hired.
One factor that matters to me in building a team is diversity. I want different kinds of people, with different temperaments and educational backgrounds.
Another factor is service. Testing is a service role. People who test should embrace that. Testers, like lawyers and doctors, are not in charge of their clients’ lives. Instead they serve them. A tester who doesn’t enjoy being of service should go into some other line of work. A tester should be thinking “how can I help?”

a1qa: Who would you recommend to consider QA as their career? How would you recommend to train newcomers and senior engineers? How to learn new if you are really long in this sphere?

J.B.: I would not recommend any career to anyone. It’s like recommending that two strangers get married. They should fall in love, first. I like testing. It’s good for someone who likes to solve puzzles and work on different things each day.
I cannot quickly tell you how to train testers. I’ve worked many years on that, and there is too much to say. I will make one suggestion: do not get ISTQB certification. It’s a waste of time and it merely gives money to a corrupt organization.

a1qa: Looks like there is no an event with Agile, Scrum, or Kanban being discussed. So Agile for you is a silver bullet or pop trend? What is your strategy in selecting approach?

J.B.: Agile is a fairly vague collection of aspirations, habits, and expectations. Skill, motivation, and social relationships are what make projects succeed.
I’m a problem solver. My way of selecting an approach is to understand the problem and my role and my available tools to solve it. Then I do what solves that problem. It’s called the Context-Driven way.

a1qa: Thanks a lot, James, for sharing your insightful views! Obviously, one post is not enough to discuss all topics of the interest. Hope we will have an opportunity to talk more and meet in the future.

Reach James Bach on Twitter.

More Posts

10 March 2020,
by a1qa
6 min read
Dedicated team model in QA: all you should know about it
Check on everything you should know about when to apply, how to run and pay for a dedicated team in QA.
Interviews
QA consulting
Quality assurance
30 September 2019,
by a1qa
4 min read
“Every team member is responsible for software quality”: interview with Head of QA at worldwide media resource
We continue talking about unsurpassed software quality. Consider how to make QA more efficient using shift-left and continuous testing.
Interviews
8 December 2017,
by a1qa
4 min read
a1qa: one-stop shop for first-rate QA services
Dmitry Tishchenko, Head of a1qa Marketing and Pre-Sales Department, answers the questions of The Technology Headlines. 
Interviews
Quality assurance
17 August 2017,
by a1qa
4 min read
From requirements specification to complex business analysis: interview with a1qa head of BA
Check how we at a1qa converge business knowledge with IT skills to deliver maximum value. 
Interviews
QA consulting
1 August 2017,
by a1qa
4 min read
Interview with head of a1qa test automation center of excellence
Dmitry Bogatko on how to manage the in-house Center of Excellence delivering value to the company's projects. 
Interviews
Test automation
19 August 2016,
by a1qa
4 min read
Interview with Adam Knight: Big Data exploratory testing
It is not so much to say that I find exploratory testing necessary. Rather I would say that I found it in my experience to be the most effective approach available to me in testing the business intelligence systems that I have.
Big data testing
Interviews
5 August 2016,
by a1qa
5 min read
Interview with Adam Knight: how much of a Sisyphean task is in software testing?
I’m a great believer in automation. I don’t believe that an agile approach to development is possible without some level of test automation. The use of such approaches does, however, need to be combined with an appreciation of the information that the automation provides you with.
Interviews
Test automation
20 July 2016,
by a1qa
4 min read
Good testing is about asking right questions: Intereview with Thanh Huynh
Software testing is not just to confirm things, it’s a process to explore, exercise the system to discover potential problems. You can achieve that by asking good questions to the system under test, yourself, your customers, your product owner, your manager, your colleagues, etc.
Interviews
20 June 2016,
by a1qa
5 min read
Interview with Lisa Crispin: whole-team approach to quality
To succeed with delivering valuable software over the long term, the whole team must take responsibility for quality, planning and executing testing activities. Our mindset has to shift from finding bugs after coding to preventing bugs from occurring in the first place.
Agile
Interviews

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.