Can test automation tools replace the human testers?

December 25, 2009 Leave a comment

In Software Testing industry, I think all are thinking on this question once in a while.

I have thought many times and asked some question to my self such as,

1. Can automation tools ever *replace* human testers?
2. Can testing tools catch defects/bugs *automatically*?
3. Can tools ever develop enough (artificial) intelligence to think like a human tester?
4. Talking about scripted regression test tools (like QTP, WinRunner, LoadRunner, JMeter, Rational Robot, blah blah blah), which operate at the GUI (not unit) level. Is their valuation/pricing/license cost fair enough to provide you a healthy ROI (Return On Investment)?
5. Is it a question of Manual Testing Vs. Automated, or something much more complex and intricate?

Before we can come to a conclusion and start answering these questions let me come up with some facts about the contenders here: automation tools and human testers. I am a human being not a robot. So I will prefer starting with the human testers. Here we go.

What is manual testing? I think the term “manual” means, “involving or using human effort, skill, power, energy, etc.” Going by this understanding, any testing that requires the human intervention can be considered as manual testing. Let us see the areas where human testers can fair better than the automation tools:

Manual Testing Pros:
1. A human can think. She has the ability to think on her own and make decisions. Though decision-making is possible in case of computers (automation tools), they are yet to reach the level of (artificial) intelligence to beat a human being when it comes to decision-making.

2. A human tester can explore new applications using her intuition and self-learning abilities. Even if she has no experience of testing anything similar before, if she is skillful she can be able to quickly learn, explore and test the application. An automation tool might not!

3. A human tester can think out side of the pre decided test boundaries. In my knowledge, more defects are unearthed using out of the box exploratory testing approach than running automation scripts. Automation tests can only find you bugs for which they were written. They can’t find you any NEW bug. If you want to find new bugs, then you need to either find those yourself manually (that’s right, by a human tester), or write NEW automation scripts to find them.

4. A human tester needs no less spoon-feeding as compared to a tool. Computers are dumb, and so are automation tools. You change a variable/object name in a line of code of your AUT (Application Under Test) and the tool would struggle to figure out where it has gone! More over tools need step-by-step instructions to perform a specific testing task. They (tools) are poor at understanding natural human language. Imagine asking a human tester to test a “User Help Manual”. What if we are to automate this test? Can a tool test a human readable help manual efficiently? In case, you happen to know of a way to automate this process, kindly let me know. I am in search of such a tool.

5. Human testers rock when it comes to Usability testing where automation tools fail miserably. Tools are poor in judging what is more usable from a human point of view like screen alignments, appearance of windows, smoothness of object alignments, color combinations, ease of usability, an entertaining user experience etc. After all we develop software to be used by human beings, not computers! So who would be a better candidate to judge the usability of such an application? You already know the answer.

6. Human beings are good examples of being adaptable. We could adapt to the changing environment and survived when big creatures like Wooly Mammoth and the Dinosaursperished and became extinct due to their lack of adaptability. Human testers can adapt and learn from past experience. But (automation) tool may fail to learn on it’s own. The entire suite of test automation scripts may soon start failing if a simple thing like an object name has been changed/renamed in recent code refactoring. A tool cannot remember and learn from past experiences to adjust it accordingly. Human testers can.

7. Considering the huge cost of License fees of the so-called GUI automation tools, human testers are still cheaper. More often than not you might run into a project where the ROI (Return On Investment) for such a tool would be in negative as compared to when the tests were done manually by human testers.

I think that is enough of pro-manual and anti-automation rant. Before you (wrongly) classify me as an ardent opposer of test automation, let me clarify that I am in fact a great supporter of test automation, if done rightly! If you have read my earlier post, then you might already know that I love to test in agile environment like TDD and as you know no agile development can be imagined without test automation. What I oppose is the idea of automating tests at the GUI level using commercial tools that eat up a large chunk of your testing budget. I am a clear supporter of test automation if done at the unit level. And of course there are areas of testing (like stress testing, performance testing, penetration testing, link testing, API testing etc) where the need of automating your tests becomes an absolute necessity.

Test Automation Pros:
1. Computers work much faster than a human tester and are less prone to get confused while doing multi-tasking (doing several tasks at the same time, switching between them). Computers never care to spend time on attending a phone call or attending a review meeting/presentation, unlike a human tester. This can result in much more productivity as compared to a human tester, of course under certain contexts.

2. Computers are great at mathematical calculations. Computer’s memory is much more accurate than a human brain, it’s capacity to remember stuffs is much more than a human and it’s capacity to retrieve data from stored memory is much faster and more accurate as compared to human brain. This accuracy of computation can be exploited via test automation, especially in tests that involve high volume of computation work.

3. Computers never skip any hours of testing. People get tired, people get distracted, but computers are great at repetitive tasks humans are not very good at. Computers need no rest, shows no sign of fatigue, never gets bored with repetitive work, can work on weekends, holidays and even night shifts without complaining at all (of course until they encounter a break down/malfunctioning). This power of computers can be used to our advantage via test automation.

4. Computers never complain about salary hikes, number of paid leaves, holidays. Nor do they ask for change of project if it starts to get boring working on the same project for a long time. Tools lack emotions and it can prove to be a good thing when looked from an employee satisfaction point of view. [Remember, an organization spends on a tool. Hence the tool is also like an employee working for the organization]

5. At certain contexts, test automation can be cost effective than manual testing. Paying for the development of a tool and a small group of testers to run it may be cheaper at contexts, as compared to maintaining a whole big group of testers testing everything manually.

6. When it comes to load test, obviously automation is an absolute requirement. Gone are the days when you summoned 100 employees into a big lab, asked each of them to hit the Enter key at the same time and hoped that they hit it exactly at the same point of time (millisecond level)! In testing we need accuracy of actions. And for scenarios like load and performance testing automation tests can offer us such precision.

7. Going for test automation can be a good choice for regression testing nightly builds. If you have a set of tests that you need to test periodically over a long period of time and if you are confident that little or no major change would be made in areas covered by those tests, then you might opt to automate such tests to save you some precious time.

Having said all this let me summarize my opinion. Coming to the core question of “can automation tools ever *replace* human testers”, I honestly don’t think that it is going to happen ever. So I believe – , “Automation testing tools can never replace human testers”.

Categories: General

Agile and Testing: Some Myths Exposed

November 13, 2009 Leave a comment

Agile is a methodology that is seeing increasingly widespread adoption, and it is easy to understand why-especially if you consider the developer and user point of view.

Users: Don’t want to spend ages being quizzed in detail about the exact requirements and processes for the whole system, and then have to review a large specification, which they know could come back to haunt them.

Developers: Don’t want to have to follow a tight specification, without any expression of their own imagination and creative talents, especially if they can see a better way.

Yet for the QA professional an Agile approach can cause discomfort – In the ideal world they would have a ‘finished’ product to verify against a finished specification. To be asked to validate a moving target against a changing backdrop is counter intuitive. It means that the use of technology and automation are much more difficult, and it requires a new approach to testing, in the same way that it does for the users and the developers.

QA teams need to know the real impact of an Agile methodology, there are boundless myths circulating the industry.

Categories: Agile Testing

Codes and Bugs Quotes

October 29, 2009 Leave a comment
  • “All code is guilty, until proven innocent.” – Anonymous
  • “First, solve the problem. Then, write the code.” – Anonymous
  • “A code that cannot be tested is flawed.” – Anonymous
  • “Good programmers write code for humans first and computers next.” – Anonymous
  • “Don’t fix it if it ain’t broke.” – Anonymous
  • “A bug in the hand is worth two in the box.” – Anonymous
  • “The only certainties in life are death, taxes and bugs in code.” – Anonymous
  • “Failure is not an option. It comes bundled with the software.” – Anonymous
  • “Blame doesn’t fix bugs.” – Anonymous

Software Testers Quotes

October 29, 2009 Leave a comment
  • “Software testers do not make software; they only make them better.” – Anonymous
  • “The principle objective of software testing is to give confidence in the software.” – Anonymous
  • “Software testers always go to heaven; they’ve already had their fair share of hell.” – Anonymous
  • “f u cn rd ths, u cn gt a gd jb n sftwr tstng.” – Anonymous

Software Testing Quotes

October 29, 2009 Leave a comment
  • “Software testing proves the existing of bugs not their absence.” – Anonymous
  • “Alpha is simply that you want somebody to share your pain!”  – Anonymous
  • “Just because you’ve counted all the trees doesn’t mean you have seen the forest.” – Anonymous
  • “More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded – indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design , coding and the rest.” – Boris Beizer
  • ” If you don’t like unit testing your product, most likely your customers won’t like to test it either.” – Anonymous


Quality Quotes

October 29, 2009 Leave a comment

A collection of Software Testing Quotes. Some are inspirational, some are outrageous and some are stark. Be stirred!

  • “Quality is never an accident; it is always the result of intelligent effort.” – John Ruskin
  • “Quality is free, but only to those who are willing to pay heavily for it.” – T. DeMarco and T. Lister
  • “Quality is the ally of schedule and cost, not their adversary. If we have to sacrifice quality to meet schedule, it’s because we are doing the job wrong from the very beginning.” – James A. Ward
  • “The bitterness of poor quality remains long after the sweetness of meeting the schedule has been forgotten.” – Anonymous
  • “Software never was perfect and won’t get perfect. But is that a licence to create garbage? The missing ingredient is our reluctance to quantify quality.” – Boris Beizer
  • “A true professional does not waste the time and money of other people by handing over software that is not reasonably free of obvious bugs; that has not undergone minimal unit testing; that does not meet the specifications and requirements; that is gold-plated  with unnecessary features; or that looks like junk.” – Daniel Read
  • “It’s more about good enough than it is about right or wrong.” – James Bach

The Glass

October 29, 2009 Leave a comment
  • To an optimist, the glass is half full.
  • To a pessimist, the glass is half empty.
  • To a good tester, the glass is twice as big as it needs to be.
Categories: General