Blog

A novel-long standard. Interview with James Christie

I worked for a big insurance company. They had just re-organized their Audit department. One of the guys who worked there knew me and thought I'd be well suited to audit. I was a developer who had moved on into a mixture of programming and systems analysis. However, I had studied accountancy at university and spent a couple of years working in accountancy and insurance investment, so I had a wider business perspective than most devs. I think that was a major reason for me being approached.
22 January 2015
Interviews
The article by a1qa
a1qa

James is particularly interested in links between testing, auditing, governance and compliance. He spent 14 years working for a large UK insurance company, then nine years with a big IT services supplier working with large clients in the UK and Finland. He has been self-employed for the last eight years.”

1. James, you’ve tried so many IT fields. Can you explain why you switched into auditing?

I worked for a big insurance company. They had just re-organized their Audit department. One of the guys who worked there knew me and thought I’d be well suited to audit. I was a developer who had moved on into a mixture of programming and systems analysis. However, I had studied accountancy at university and spent a couple of years working in accountancy and insurance investment, so I had a wider business perspective than most devs. I think that was a major reason for me being approached.

I turned down the opportunity because I was enjoying my job and I wanted to finish the project I was responsible for. The Audit department kept in touch with me and I gradually realised that it would be a much more interesting role than I’d thought. A couple of years later another opportunity came up at a time when I was doing less interesting work so I jumped at the chance. It was a great decision. I learned a huge amount about how IT fitted into the business.

2. As the person with an audit background, do you think standards improve software testing or block it?

They don’t improve testing. I don’t think there’s any evidence to support that assertion. The most that ISO 29119 defenders have come up with is the claim that quality assurance consultants can now do good testing using the standard. That’s arguable, but even if it is true it is a very weak defence for making something a standard. It’s basically saying that ISO 29119 isn’t necessarily harmful.

I wouldn’t have said that ISO 29119 blocks testing. It’s a distraction from testing because it focuses attention on the documentation, rather than the real testing. An auditor should expect three things; a clear idea about how testing will be performed, and evidence that explains what testing was done, and what the significance was of the results.

ISO 29119, and the previous testing standard IEEE 829, emphasize heavy advance documentation and deal pitifully with the final reports. Auditors should expect an over-arching test strategy saying “this is our approach to testing in this organization”. They should also expect an explanation of how that strategy will be interpreted for the project in question.

Detailed test case specifications shouldn’t impress auditors any more than detailed project plans would convince anyone that the project was successful. ISO 29119 says that “test cases shall be recorded in the test case specification” and “the test case specification shall be approved by the stakeholders”.

That means that if testers are to be compliant with the standard they have to document their planned testing in detail, then get the documents approved by many people who can’t be expected to understand all that detail. Trying to comply with the standard will create a mountain of unnecessary paper. As I said, it’s a distraction from the real work.

3. You started the campaign “STOP 29119”.Tell us a few words about the standard?

I don’t claim that I started the campaign. The people who deserve most credit for that are probably Karen Johnson and Iain McCowatt, who responded so energetically to my talk at CAST 2014 in New York.
ISO 29119 is an ambitious attempt, in ISO’s words “to define an internationally-agreed set of standards for software testing that can be used by any organization when performing any form of software testing.”

The full standard will consist of five documents; glossary, processes, documentation, techniques and finally key-word driven testing. So far the first three documents have been issued, i.e. the glossary, processes and documentation. The fourth document, test techniques, is due to be issued any time now. The fifth, on key-word driven testing should come out in 2015.

The campaign has called on ISO to withdraw the standard. However, I would happily settle for damaging its credibility as a standard for “any organization when performing any form of software testing”. That aim is more than just being ambitious. It stretches credulity.

4. Testing standards are beneficial for testing (hope you agree): they implement some new practices and can school the untutored. Still, what is wrong with the 29119 standard?

The content of ISO 29119 is very old-fashioned. It is based on a world view from the 1970s and 1980s that confused rigour and professionalism with massive documentation. It really is the last place to go to look for new ideas. Newcomers to testing should be encouraged to look elsewhere for ideas about how to perform good testing.

Testing standards can be beneficial in a particular organization. They may even be beneficial in industries that have specific needs, such as medical devices and drugs, and financial services. However, they have to be very carefully written and they must maintain a clear distinction between true standards and overly prescriptive guidance. ISO 29119 fails to make the distinction. It is far too detailed and prescriptive.

The three documents that have been issued so far add up to 89,000 words over 270 pages. That’s as long as many novels. In fact it’s as long as George Orwell’s “Animal Farm” plus Erich Maria Remarque’s “All Quiet on the Western Front” combined. It’s almost exactly the same length as Orwell’s “1984” and Jane Austen’s “Persuasion”.

That is ridiculously long for a standard. The Institute of Internal Auditors’ “International Standards for the Professional Practice of Internal Auditing” runs to only 26 pages and 8,000 words. The IIA’s standards are high level statements of principle, covering all types of auditing. More detailed guidance about how to perform audits in particular fields is published separately. That guidance doesn’t amount to a series of “you shall do x, y & z”. It offers auditors advice on potential problems, and gives useful tips to guide the inexperienced. The difference between standards and guidance is crucial, and ISO blurs that distinction.

The defenders of ISO 29119 argue that tailored compliance is possible; testers don’t have to follow the full standard. There are two problems with that. Tailored compliance requires agreement from all of the stakeholders for all of the tasks that won’t be performed, and documents that won’t be produced. There are hundreds of mandatory tasks and documents, so even tailored compliance imposes a huge bureaucratic overhead. The second problem is that tailored compliance will look irresponsible. The marketing of the standard appeals to fear. Stuart Reid has put it explicitly.

“Imagine something goes noticeably wrong. How easy will you find it to explain that your testing doesn’t comply with international testing standards? So, can you afford not to use them?”

Anyone who is motivated by that to introduce ISO 29119 is likely to believe that full compliance must be safer and more responsible than tailored compliance. The old IIEE 829 test documentation standard also permitted tailored compliance. That wasn’t the way it worked out in practice. Organizations which followed the standard didn’t tailor their compliance and produced far too much wasteful documentation. ISO should have thought more carefully about how they would promote the standard and what the effects might be of their appeal to fear.

5. And in the end, what are the results of your campaign?

It’s hard to say what the results are. No-one seriously expected that ISO would roll over and withdraw the standard. I did think that ISO would make a serious attempt to defend it, and to engage with the arguments of the Stop 29119 campaigners. That hasn’t happened. The result has been that when people search for information about ISO 29119 they can’t fail to find articles by Stop 29119 campaigners. They will find nothing to refute them. I think that damages ISO’s credibility. ISO is now caught in a bind. It can ignore the opposition, and therefore concede the field to its opponents. Or it can try to engage in debate and reveal the lack of credible foundations of the standard.

I think the campaign has been successful in demonstrating that the standard lacks credibility in a very important part of the testing profession and therefore lacks the consensus that a standard should enjoy. I hope that if the campaign keeps going then it will prevent many organizations from forcing the standard onto their testers and thus forcing them to do less effective and efficient testing. Sometimes it feels like Stop 29119 is very negative, but if we can persuade people not to adopt the standard then I think that makes a positive contribution towards more testers doing better testing.

James, thank you for sharing your viewpoint and ideas. We hope to talk to you again and cover a few more interesting issues.

More Posts

10 March 2020,
by a1qa
6 min read
Dedicated team model in QA: all you should know about it
Check on everything you should know about when to apply, how to run and pay for a dedicated team in QA.
Interviews
QA consulting
Quality assurance
30 September 2019,
by a1qa
4 min read
“Every team member is responsible for software quality”: interview with Head of QA at worldwide media resource
We continue talking about unsurpassed software quality. Consider how to make QA more efficient using shift-left and continuous testing.
Interviews
8 December 2017,
by a1qa
4 min read
a1qa: one-stop shop for first-rate QA services
Dmitry Tishchenko, Head of a1qa Marketing and Pre-Sales Department, answers the questions of The Technology Headlines. 
Interviews
Quality assurance
17 August 2017,
by a1qa
4 min read
From requirements specification to complex business analysis: interview with a1qa head of BA
Check how we at a1qa converge business knowledge with IT skills to deliver maximum value. 
Interviews
QA consulting
1 August 2017,
by a1qa
4 min read
Interview with head of a1qa test automation center of excellence
Dmitry Bogatko on how to manage the in-house Center of Excellence delivering value to the company's projects. 
Interviews
Test automation
19 August 2016,
by a1qa
4 min read
Interview with Adam Knight: Big Data exploratory testing
It is not so much to say that I find exploratory testing necessary. Rather I would say that I found it in my experience to be the most effective approach available to me in testing the business intelligence systems that I have.
Big data testing
Interviews
5 August 2016,
by a1qa
5 min read
Interview with Adam Knight: how much of a Sisyphean task is in software testing?
I’m a great believer in automation. I don’t believe that an agile approach to development is possible without some level of test automation. The use of such approaches does, however, need to be combined with an appreciation of the information that the automation provides you with.
Interviews
Test automation
20 July 2016,
by a1qa
4 min read
Good testing is about asking right questions: Intereview with Thanh Huynh
Software testing is not just to confirm things, it’s a process to explore, exercise the system to discover potential problems. You can achieve that by asking good questions to the system under test, yourself, your customers, your product owner, your manager, your colleagues, etc.
Interviews
20 June 2016,
by a1qa
5 min read
Interview with Lisa Crispin: whole-team approach to quality
To succeed with delivering valuable software over the long term, the whole team must take responsibility for quality, planning and executing testing activities. Our mindset has to shift from finding bugs after coding to preventing bugs from occurring in the first place.
Agile
Interviews

Get in touch

Please fill in the required field.
Email address seems invalid.
Please fill in the required field.
We use cookies on our website to improve its functionality and to enhance your user experience. We also use cookies for analytics. If you continue to browse this website, we will assume you agree that we can place cookies on your device. For more details, please read our Privacy and Cookies Policy.