Archive

Archive for the ‘General’ Category

Can test automation tools replace the human testers?

December 25, 2009 Leave a comment

In Software Testing industry, I think all are thinking on this question once in a while.

I have thought many times and asked some question to my self such as,

1. Can automation tools ever *replace* human testers?
2. Can testing tools catch defects/bugs *automatically*?
3. Can tools ever develop enough (artificial) intelligence to think like a human tester?
4. Talking about scripted regression test tools (like QTP, WinRunner, LoadRunner, JMeter, Rational Robot, blah blah blah), which operate at the GUI (not unit) level. Is their valuation/pricing/license cost fair enough to provide you a healthy ROI (Return On Investment)?
5. Is it a question of Manual Testing Vs. Automated, or something much more complex and intricate?

Before we can come to a conclusion and start answering these questions let me come up with some facts about the contenders here: automation tools and human testers. I am a human being not a robot. So I will prefer starting with the human testers. Here we go.

What is manual testing? I think the term “manual” means, “involving or using human effort, skill, power, energy, etc.” Going by this understanding, any testing that requires the human intervention can be considered as manual testing. Let us see the areas where human testers can fair better than the automation tools:

Manual Testing Pros:
1. A human can think. She has the ability to think on her own and make decisions. Though decision-making is possible in case of computers (automation tools), they are yet to reach the level of (artificial) intelligence to beat a human being when it comes to decision-making.

2. A human tester can explore new applications using her intuition and self-learning abilities. Even if she has no experience of testing anything similar before, if she is skillful she can be able to quickly learn, explore and test the application. An automation tool might not!

3. A human tester can think out side of the pre decided test boundaries. In my knowledge, more defects are unearthed using out of the box exploratory testing approach than running automation scripts. Automation tests can only find you bugs for which they were written. They can’t find you any NEW bug. If you want to find new bugs, then you need to either find those yourself manually (that’s right, by a human tester), or write NEW automation scripts to find them.

4. A human tester needs no less spoon-feeding as compared to a tool. Computers are dumb, and so are automation tools. You change a variable/object name in a line of code of your AUT (Application Under Test) and the tool would struggle to figure out where it has gone! More over tools need step-by-step instructions to perform a specific testing task. They (tools) are poor at understanding natural human language. Imagine asking a human tester to test a “User Help Manual”. What if we are to automate this test? Can a tool test a human readable help manual efficiently? In case, you happen to know of a way to automate this process, kindly let me know. I am in search of such a tool.

5. Human testers rock when it comes to Usability testing where automation tools fail miserably. Tools are poor in judging what is more usable from a human point of view like screen alignments, appearance of windows, smoothness of object alignments, color combinations, ease of usability, an entertaining user experience etc. After all we develop software to be used by human beings, not computers! So who would be a better candidate to judge the usability of such an application? You already know the answer.

6. Human beings are good examples of being adaptable. We could adapt to the changing environment and survived when big creatures like Wooly Mammoth and the Dinosaursperished and became extinct due to their lack of adaptability. Human testers can adapt and learn from past experience. But (automation) tool may fail to learn on it’s own. The entire suite of test automation scripts may soon start failing if a simple thing like an object name has been changed/renamed in recent code refactoring. A tool cannot remember and learn from past experiences to adjust it accordingly. Human testers can.

7. Considering the huge cost of License fees of the so-called GUI automation tools, human testers are still cheaper. More often than not you might run into a project where the ROI (Return On Investment) for such a tool would be in negative as compared to when the tests were done manually by human testers.

I think that is enough of pro-manual and anti-automation rant. Before you (wrongly) classify me as an ardent opposer of test automation, let me clarify that I am in fact a great supporter of test automation, if done rightly! If you have read my earlier post, then you might already know that I love to test in agile environment like TDD and as you know no agile development can be imagined without test automation. What I oppose is the idea of automating tests at the GUI level using commercial tools that eat up a large chunk of your testing budget. I am a clear supporter of test automation if done at the unit level. And of course there are areas of testing (like stress testing, performance testing, penetration testing, link testing, API testing etc) where the need of automating your tests becomes an absolute necessity.

Test Automation Pros:
1. Computers work much faster than a human tester and are less prone to get confused while doing multi-tasking (doing several tasks at the same time, switching between them). Computers never care to spend time on attending a phone call or attending a review meeting/presentation, unlike a human tester. This can result in much more productivity as compared to a human tester, of course under certain contexts.

2. Computers are great at mathematical calculations. Computer’s memory is much more accurate than a human brain, it’s capacity to remember stuffs is much more than a human and it’s capacity to retrieve data from stored memory is much faster and more accurate as compared to human brain. This accuracy of computation can be exploited via test automation, especially in tests that involve high volume of computation work.

3. Computers never skip any hours of testing. People get tired, people get distracted, but computers are great at repetitive tasks humans are not very good at. Computers need no rest, shows no sign of fatigue, never gets bored with repetitive work, can work on weekends, holidays and even night shifts without complaining at all (of course until they encounter a break down/malfunctioning). This power of computers can be used to our advantage via test automation.

4. Computers never complain about salary hikes, number of paid leaves, holidays. Nor do they ask for change of project if it starts to get boring working on the same project for a long time. Tools lack emotions and it can prove to be a good thing when looked from an employee satisfaction point of view. [Remember, an organization spends on a tool. Hence the tool is also like an employee working for the organization]

5. At certain contexts, test automation can be cost effective than manual testing. Paying for the development of a tool and a small group of testers to run it may be cheaper at contexts, as compared to maintaining a whole big group of testers testing everything manually.

6. When it comes to load test, obviously automation is an absolute requirement. Gone are the days when you summoned 100 employees into a big lab, asked each of them to hit the Enter key at the same time and hoped that they hit it exactly at the same point of time (millisecond level)! In testing we need accuracy of actions. And for scenarios like load and performance testing automation tests can offer us such precision.

7. Going for test automation can be a good choice for regression testing nightly builds. If you have a set of tests that you need to test periodically over a long period of time and if you are confident that little or no major change would be made in areas covered by those tests, then you might opt to automate such tests to save you some precious time.

Having said all this let me summarize my opinion. Coming to the core question of “can automation tools ever *replace* human testers”, I honestly don’t think that it is going to happen ever. So I believe – , “Automation testing tools can never replace human testers”.

Advertisements
Categories: General

The Glass

October 29, 2009 Leave a comment
  • To an optimist, the glass is half full.
  • To a pessimist, the glass is half empty.
  • To a good tester, the glass is twice as big as it needs to be.
Categories: General