Koderly will soon be launching Qucate, a test management platform for functional software. Qucate has a strong focus on manual testing, which recently sparked an internal discussion when I was asked the purpose of Qucate (we love challenging ourselves!), given that automated testing appears to be the leader in the world of software testing.
For this reason, we thought it would be useful to explain the differences between manual vs automated testing and explore use cases of each.
What's the difference between manual and automated testing?
Before diving head-first into the use cases, we need to understand the differences between automated vs manual testing.
With manual testing, a human, whether it’s a developer, a testing engineer, or a system user, will manually execute test cases to validate the functionality and behaviour of a piece of software. The tester will need to consider which scenarios and edge cases that they want to test and then trigger the appropriate inputs or actions to test each scenario.
By contrast, with automated testing – as the name implies – this process is automated. By utilising tools and predefined testing scripts, tests are executed automatically by setting inputs and triggering actions or events whilst verifying the outputs.
Like manual testing, a human, usually a developer or a testing engineer will need to consider which scenarios and edge cases are going to be tested and write the corresponding testing scripts.
| Manual Testing | Automated Testing |
|---|---|
|
Manual Testing
Testers manually scour through various features of a system, website, or application to identify bugs, errors, anomalies, and the like.
|
Automated Testing
Testers use test frameworks and test scripts that automate user actions required for system, website, or application testing.
|
It’s easy to decipher the advantages of automated testing with it being less prone to human error and benefitting from its reliability and repeatability. You’ll only need to consider the testing scenarios and write the test scripts once, and never have to worry about it again! At face value, you’d be correct; but, unfortunately, in the real world, it becomes much more complicated than that.
While automated testing does a lot of the heavy lifting, a human is still required to determine which test scenarios require testing, to write the automated test scripts and interpret the test results to make sure they match the goals. This leaves automation open to human error too! You may also have a complex case that needs an in-depth test, something more than what automation can offer.
"With great power there must also come -- great responsibility"
A bug in your software isn’t ideal, but a bug in your testing scripts can be worse! Development teams can place too much reliance on their automated test suites and the green ticks on the test run results reports. This can lead to a false sense of comfort and an incorrect conclusion that everything is working as it should be.
In my own professional experience, I’ve witnessed a poorly written test assertion cause a large ecommerce company to suffer millions in lost revenue. If a real-world user had tested the changes prior to deploying to production, it would have been obvious that significant discounts we’re being offered to all customers regardless of eligibility. Ouch..
As software evolves and changes are made, the test cases and scenarios will need to change too, or they are likely to become irrelevant and lay by the wayside.
When to use manual vs automated testing
Using automated testing
Automation is great, and can make our lives easier in the right context. Computers, unlike humans, are really good at completing repetitive and mundane tasks, and can do these tasks quickly and accurately, and that’s why some testing processes have been replaced with automation!
Testing, boiled down to its simplest form, is the process of checking that a specific set of inputs and actions returns the expected result or behaviour. Automation is very well suited to these kinds of tasks, but isn’t as good at predicting human behaviour or dealing with edge cases.
Automated testing really comes into its own when the scale of testing is large, and where the development cycles are short. This allows for frequent and repeated execution of test scripts, providing early identification of issues whilst supporting the ability to ship incremental changes and new features quickly. The key advantages of automated testing are:
- Cost efficiency
- Ability to perform testing at a large scale
- Faster turnaround time, and
- Highly accurate (but inflexible)
Below, I’ve provided some examples of some common scenarios where automated testing is particularly advantageous:
When repetitive tests have to be run frequently:
Automation tests are the best option for running repetitive tests. Lets take an online shop for example, which trades in multiple currencies and selects the currency based on the customers location.
Whilst it is possible to manually test this kind of scenario, it becomes unfeasible owing to the amount of effort required from a manual tester to change their location, observe that the correct currency is being presented and that the prices are correctly being converted using the correct conversion rate.
Automation is a great use case here, being able to run through all the scenarios efficiently with high precision in barely any time at all.
When resources are scarce or limited:
Automation tests are also a great way of freeing up development and testing resource on projects where the number of contributors is scarce, and they don’t need human interaction. Automation will allow the testers to focus on tasks which require their attention rather than being forced to repeat testing on features which are subject to frequent change.
Automated testing is an excellent tool to run large volumes of tests in a short period of time without compromising on the results.
As you can see, automated testing is a great tool to have in our tool belt, helping us to become more efficient whilst improving quality. Let’s now take a look at when to use manual testing, and how it might be leveraged within our development and quality assurance (QA) teams.
Using manual testing
Manual testing relies on people to perform actions and to verify behaviours. It’s not uncommon for even the best testers to make a mistake. We’re only human after all, and mistakes are inevitable.
Quite frankly, mistakes will be more common with manual testing, however, whereas automation is great at completing repetitive tasks, we, as humans, have a significant advantage over computers, in that we have cognitive and behavioural abilities which are particularly useful for the following types of testing:
- User Interface (UI) testing
- User Experience (UX) and Usability testing
- User Acceptance testing
- Exploratory and Experimental testing
- Localisation testing
Below, I’ve provided some examples of some common scenarios where manual testing is advantageous:
Short-term projects:
Since automation requires more investment and planning, it’s not particularly suitable for a short-term project that focuses on minor features or minimal code. It would require a lot of effort from the developer and tester to create and maintain the automated tests and could be too prohibitive for a narrow scope of work.
As an example, if I developed a small form which posts some data to a server, in total it took a couple of hours to develop. The form isn’t likely to be subject to any future changes, and if it is, they’ll be minor in scope and likely only affect the presentation of the form. Writing an automated test suite for this could be extremely prohibitive, as I can assure you, it’ll take a lot longer than the couple of hours developing the form to set up and configure the automation framework and define and write my test cases. In this scenario, your time will be better spent performing a manual test run to verify the behaviour of the form.
When end-user usability is being tested:
Automation tests are terrible at predicting how a real-world user will interact with and utilise your software. A computer cannot judge whether the user interface (UI) provides a good real-world experience. Manual testing is a great way to ensure that your software provides a stellar user-experience.
When flexibility is required:
Manual testing processes allow testers to have much more flexibility in their testing operations. Automated testing is inherently useful for testing repeated cases, which often means that the same process flows are being leveraged on every time. However, exploration and experimental testing is important for discovery, investigation, learning and creativity, by leveraging the testers freedom to explore the software and suggest change and enhancements.
Ultimately, any testing that you do against software created by you or your organisation is better than none at all, if you’d like to do some further reading of the benefits of software testing, be sure to read our blog on What is Software Testing, and How Does it Work?
I’m of the opinion that, to achieve the best results with software testing, it is important to strike the right balance between both manual and automated testing and it’s my hope that this blog has helped to convince you that both should be considered within your software development lifecycle.
At Koderly, we use Qucate to manage and track progress of our manual tests. Qucate could help your organisation too, by allowing you to create fully audited test plans for compliance, visibility, and transparency, whilst improving quality and test coverage. Head to our Qucate page for more information, or feel free to contact us.
