Who can be a good automator? Interview with Shrinivas Kulkarni. Part II
This is the second part of the interview with Shrinivas Kulkarni, international Software expert and popular blogger . If you haven’t read, please check out part 1 of the interview published on Tuesday this week. Shrini suggests, “Test automation to be probably the most misunderstood concept in the field of software Testing…” Today we continue discussing Test Automation using different angles and approaches.
a1qa:. Can you give the concrete example of the Automation Testing positive result for the business processes from your experience or the one you heard about? (How much money saved, time to market accelerated or some other input).
S.K.: As I said before, automation when done “properly” is going to increase skilled testers` productivity and efficiency. Concrete examples of positive results of automation – in my opinion – should be related to these terms mainly. Automation in IT services world is a big money making enterprise. There are automation tool makers, consultants and automation developers that make up the commercial equation of automation. When automation is done using discretionary expenditures – the money spent need to be justified in some commercial terms that executives required to sign-up. This is where the problems start and we slip into realm of manipulation, politics, personal motivations biases, likes-dislikes etc. I have held very skeptic views of claims of automation such as ROI, cycle time reduction, faster time to market; as these terms are too vague when applied to automation and often depend on several parameters outside the control of testing.
For example, if someone claims that they reduced cycle time due testing – here is how I scrutinize the claim. First of all the whole idea of cycle time is often not clearly stated – are they refer to only testing cycle time or overall development or release cycle. If we consider testing cycle alone – how do we mark start and end of test cycle? Test cycle is a provisional construct devised as convenient unit and whose definition changes from project to project. Even within any given definition of testing cycle – what does end of test cycle signify, release of the product? Often, end of one testing cycle throws of bunch of bugs for developers to fix and that needs more testing. Unless you see such a testing cycle where end of the cycle we have zero new bugs or zero outstanding bugs to worry about – end does not technically mean anything about release of product. Release of product at the end of testing cycle is a business decision based on risk appetite of the stakeholders and business/market considerations. Automation, testing or even development team – cannot do anything if executives decide to release a product to meet some specific business goals. My point here is – effectiveness and success of automation needs to be measured at different level – tester’s level, not at a business level. Cycle time or testing cycle time is an example of a parameter dictated at business/stakeholder level.
Same argument goes for “faster time to market”.
My recommendation is to measure success of automation by capturing how it makes skilled human tester productive and efficient. As go higher towards business stakeholder level – picture gets blur and many parameters that are out of control of testing or automation teams get in. This impacts the decision making process about automation and marketing persuasion prevails over critical thinking. Thus, when you hear phrases like ROI, cycle time reduction money saved through automation – take a step back and subject those to critical thinking. Just do not accept them as granted. The devil is in the details.
a1qa:. What are the typical, most common mistakes while implementing Test Automation?
S.K.: One big mistake often committed is to treat automation as magic wand solving any problem related to testing – even reduce or eliminate testing itself. I often advised teams not to accept automation as a solution when there is less than required time for testing. For managers and many executives – automation is often sold as a means or tool when there is insufficient time for testing. This is the wrong way of using automation. I am yet to see a real and practical example of automation balancing the lack of time for testing.
Another big mistake related to above is giving very little or no consideration about testing and how automation fits in it. If you are doing bad testing, throwing some automation at such testing cause worse results. James Bach in his famous article on automation (Test Automation snake oil) lists down a number of reckless assumptions about testing that automation folks often make. For example consider testing as a sequence of actions. If you allow people (consultants and tool vendors) to trivialize testing, you are giving them the freedom to sell anything under the name of automation. You cannot automate testing, as testing is a fundamentally human cognitive process, but such a process can be enhanced by use of tools. A good automation has deep grounding in good testing – find out, for your context what that good testing is and then ask how automation assists the process. It is a pity that many tool vendors market their tools as those doing testing and user’s (testers to test managers) community accepts such claims.
You need to be wary of marketing pitch that comes along automation. Apply critical thinking to marketing claims such as faster automation, easy to use, quick ROI and assess the feasibility of each of these claims in the context of your project, company and industry.
Most tempting marketing pitch by tool vendors is “automation to non-technical users”. Non-technical users (meaning those who do not write code) can create tests but cannot write or maintain code. So, the rhetoric of non-technical automation is an oxymoron. Watch out for this.
Another big problem that we need to deal with automation is about tools that claim “script less” or “code less”. If you reckon automation as software development – how can there be a script less or code less development? I have used several tools that start off as “pure script less”. When we raise questions about support for programming constructs like looping, decision control, exception handling – the tool starts opening up into programming domain. Eventually they end up in halfway where there is some script less part (to market tool to execs in the name of “being easy to use”) and other part is to support programming needs. Such tools, in my opinion are better to be avoided.
In summary: (how to spot and avoid mistakes)
1. Tool cannot do testing – humans do. There is no such thing called automated testing as testing cannot be automated but can be enhanced by the tools
2. Automation for Non-technical users is an oxymoron (like a water-proof towel)
3. Avoid Scriptless automation tool if you can.
4. Beware of Marketing pitches that come with automation – watch out for phrases like cycle time reduction, faster time to market. They are traps.
a1qa:. Who is the test Automator according to you? He is under-developer or over-tester? How to check the aptitude for test automation? Developers may face a problem with motivation, as for testers – their technical level is not always good enough.
S.K.: For anyone to be an automator – software development and programming skills in one formal/full programming language is a-must. It is possible that a tester has such skills and can be an effective automator. One of them is the ability to understand/decipher software code and ability to update/modify code. The general tester`s description (a manual tester) is someone who cannot write or understand code. This has to change. IT service providers have been charging a premium for so called “automation engineer” who is nothing but a tester with some training automation tools (in most of the cases no programming skills). A rejected developer cannot perform well as an automator as the key skill of programming/design is missing and needless to say such a rejected developer would not have invested in becoming a good tester. How much or how deep a tester should know programming is a question based on the context of the project/company/domain.
There is no one-size-fits-all technical bar for an automator. A good automator would have a balanced mix of testing skills and programming skills so that one can blend them (skills) in developing/maintaining automation solution that enhances efficiency and productivity of testing. For all you know, in future there might not be any specific role called “automator” – a tester or a developer would create and maintain automation solution. Accordingly, there would be a need for skill enhancement for developer (of testing) and tester (of programming) or we might just pair a developer and tester for the purpose. Interesting times are ahead.
a1qa: Thank you again, Shrini. I hope for the best in your work, and thanks for contributing to something that is important and cognitive to all curious Testers!
