What is a bug report and why do we write them?
The software development environment has shifted significantly in the recent years with the introduction of Agile and Devops. This new focus on higher speed and the shift to smaller, more frequent releases, has a great influence on testing. In the last year the average percentage of test automation has increased from 28 percent to 45 percent, according to the World Quality Report 2015-2016 . But what is test automation actually? Is it the Holy Grail?
What is test automation actually? Test automation or ‘automatic testing’ is an automated test process where in predefined and set values are controlled by a ‘robot’ in a particular information system or subsystem. This obviously isn’t a real ‘robot’, but a piece of software that allows another piece of software to check the defined values. To create a visual, the term ‘Robot’ best fits the description.
The testrobot has many advantages as long as the instructions and data sets given to the robot are correct. A robot is better, faster and more capable to find mistakes, it can compare millions (conversion data) of lines without ever needing a cup of coffee, it can run unlimited regression test on labor-intensive processes, it can quickly compare installations with each other and we can still keep going on from there. Test automation offers many advantages.
Instructions and dataset
This robot also comes with it’s disadvantages. First, all instructions that should be given to the robot should be concretely identified and ‘explained’ to that robot. Mapping of what is being tested and how it is being tested is a profession on its own. Now imagine all the input, process and output variables and conditions have to be written as well. And as these instructions and conditions are conceived and organized, the robot still needs to be ‘trained’ to do the right comparisons. This is people’s work. The field of artificial intelligence is growing rapidly, but it will certainly not take those activities out of the hands of humans in the near future. Test automation is expensive, as a result of which a good cost/benefit analysis with clear trade-offs is needed, to make a clear distinction between which parts will be performed by the robot and manually by humans.
‘Test Automation is therefore perfect for comparing variables, and most preferably in large quantities, but for the time being will not exceed the intuition and the implicit observation of a human being.’
Secondly, the robot can make high-speed comparisons, but during manual testing the scope beyond test execution is more broad. The human perspective is much more aware and intuitive and therefore ‘automatically’ looks beyond the test case. Think of connections between input, illogical acts, spelling mistakes and so forth. The robot has no intuition, it does absolutely nothing more or less than to compare variables and therefore has no implicit observation. Precisely because of the implicit observation a human can detect coincidences. Whereby an error is discovered only at a specific combination of variables or while having knowledge of daily operational problems, which still can change in the course of time. Test Automation is therefore perfect for comparing variables, and most preferably in large quantities, but for the time being will not exceed the intuition and the implicit observation of a human being. As an example: exploratory testing, which is very valuable to quickly find faults, but a robot won’t be able to do this in the near future.
Thirdly, it is often overlooked that some integrated processes aren’t automatable. These include financial transactions to be processed by batch through another institution before permanently booking these in your financial systems. In addition, maintenance is of the instruction and data set of the robot is more work than initially thought. Because the context (both business processes and IT) of organizations are constant in motion, the robot must be trained again with new instructions and custom data. In addition, the software which is automatically tested is also changing. All new functionalities, new interfaces and the like. Will first be evaluated and tested by humans on functionality, but as well in the organizations specific setting. Before the new instructions can be given tot he robot. Test Automation is ideal for stable conditions, in which the frequency of changes is limited.
Is the Testrobot going to accept the software?
Fourth and final point, In our field of user/acceptance testing it’s difficult to carry these out by a robot. The definition of acceptance tests is to establish that the organization is ready for the use of the software. Thereby it is not only determined if it works, but also whether the user can do their work with it. Will the robot determine if the organization is ready to use it?
In addition, the review and analysis of the test result is people’s work. At the moment that 10 users review 10 test cases I get a 100 test results. What if 4 users say that it is good, and 6 says it is an error. At that point a dialogue should start on the interpretation of the data and the larger context in which the good- or error-finding is reached. You can not have a robot do this.
In our vision all test results must be managed from one central location; the test results of the human, but as well the test results of the robot. Because also the test results of the robot must be evaluated and analyzed on truth and context. From there on issues are created and shared with stakeholders. These aren’t only the software vendors, but also the organization itself. Test automation is a nice addition in the entire test spectrum, but all test results in the end need to be reviewed by humans.
Test automation has many advantages in the context of efficiency and effectiveness during the implementation, but it has its share of disadvantages. Focussing on Agile-like software development and deployments test automation is essential to manage the quality and speed of increments. However: Testing is a profession and testing is thinking. Therefore it is crucial to test manually, with all it’s intuitions, implicit observations and determination if an organization is actually ready to use the software within a healthy balance with automated testing. Follow us on LinkedIn.
If you’re a newcomer in the world of testing, you easily become overwhelmed by all the testing terminology getting thrown around. In this article, we’ll try to explain some of the jargon you might encounter in your daily testing life, just to make it all a little bit more understandable.
So, we’ll start with test cases. Most importantly, test cases provide a series of instructions that validates some part of the software is doing it job as expected. Say, you require a contact form on your website that sends an e-mail when a visitor leaves a message, you’ll probably create a test case verifying the e-mail actually arrives when the form is filled and submitted. A test case can help you determine whether or not the software meets your requirements.
In some situations, a single test case can verify a simple requirement. However, it is more common practice to think of multiple situations that tests your feature more thoroughly. When your contact form offers an optional text field, try both filling and clearing the field in two separate test cases. Even better: try to go beyond the “happy path” and think of things the software shouldn’t do. If your contact form accepts an e-mail address, why not check if a non-valid address is accepted too?
Test cases do not merely consist out of instructions – or test steps, as we’d like to call them. For example: simply performing a few steps without explicitly telling the excepted outcome could cause misunderstandings. Should our contact form rejected an invalid e-mail address? Perhaps – or perhaps, we really don’t care. A proper test case usually comes with a predefined expected result, telling you what to expect when completing a test case. Similarly, it is common good practice to define the initial expected state when writing a test case. You don’t want to cover every piece of preparation in your test steps, like accessing a test environment, setting up a browser, enabling mail notifications, etc. These pre-conditions are usually defined as a separate property of the test case. When a condition isn’t met, there’s no point in testing the test case.
So, that’s it then? Well, there’s more. Like writing a blog post, creating test cases require certain writing skills, like imagining things from the end-user’s perspective. If you want to read more, be sure to read 7 tips for great test cases.
When you know about test cases, the concept of test suites should be really simple. With a growing amount of test cases, the need for categorizing them increases as well. You don’t want to throw all your books on a pile, you’ll organize them on shelves, for better accessibility. In a way, test suites are very similar to shelves, as they group test cases together as well.
In the process of planning and running tests, keeping track of hundreds of test cases really becomes cumbersome. Thinking in terms of “We need to run a performance test tomorrow” or “Next week we should schedule a test for our website” forces you to know the purpose and domain of each test case. A test suite allows you categorize test cases in such a way that they match your planning and analysis needs. Do you run functional and performance tests? Create two suites and label them accordingly. Do you test different applications and want to monitor them? Create a suite for every application. Are you still able to manage without test suites? Don’t use them! Remember, they are only there to help you organize test cases when needed.
Finally, we end up with test runs. They are little crossroads in your testing project, where several aspects come together. Simply put, test runs determine which test cases are tested by which user(s) at what particular time. Quite a mouthful!
Image you have created your test cases for your contact form and its about to become available to your organization for testing. Are you going to test it for yourself, or do you pick someone else? When are you going to start testing – and, when do you want to gather the results? Which test cases are you going to select – all of them, or just the ones marked “happy flow”? These questions should be answered when planning a test run.
When you have created a test run, you have reached an important milestone: people are able to run a test, using your test cases, grouped by your test suites. In one of our next posts, we’ll continue with some useful tips running tests and gathering test results.