Software testing is too much work for medium sized organizations

René Ceelen, 
19 April 2017

What is a bug report and why do we write them?

Medium sized organizations are hardly capable of controlling their IT systems, the testing of their software is too complex. Even the relative easy software that is needed to anonymize production data, is too much work. This is not because of unwillingness, say René Ceelen, but inability: It’s too expensive.

Package software is popular

Software packages are hugely popular. At a lot of medium sized organizations like healthcare, housing associations they form the core of the IT system, and all the ins and outs of the organization completely depend on the ups and downs of the software package. However medium sized organizations underestimate the complexity of the information system that has been built with the software package. They inadequately realize that they hardly are able to control the quality of that information system, because testing these have become too specialized and time consuming.

Linking packages?

The complexity of such information systems usually starts when multiple software packages are mutually linked. Subsequently this whole will be linked to the outside world – hospitals with general practitioners and pharmacies, and soon also with a national switchboard (our situation in the Netherlands). In this kind of configuration, the information streams wiggle themselves through a spider web of systems and links, as confusing as an Eastern Kasbah, where tourist easily get lost, filled with houses, slums and alleys.

Because of the dependence to their IT-solutions, it isn’t an unnecessary luxury for organizations to demonstrate (again) that everything is still functional with every new release. Under the specific situation of where the organization utilizes the information system, including the links with other software, not just for one type of software packet that the software developer tests in his private laboratory situation. Recognized standards for information security like the Code of Practice for information security management or the derivative thereof for information security management in the healthcare industry (Dutch NEN7510) constitute clear requirements to this type of work.


To start off with an easy demand, that is mentioned in the standard for information security management in the healthcare industry (Dutch NEN7511-1): if test data is a derivative from production data, then this data should be made anonymous, a demand coming straight from the Data Protection Act (DPA) amongst other things. Completely correct, especially for an institution for mental health care. The same goes for simple looking financial administration of for example housing associations. Again with an organization like the aforementioned, there is confidential data traffic like information about social issues, late payments and income data. Too often, however the relatively simple software which is necessary for anonymizing of production data, is already too much work. The core requirements of good testing are availability of acceptance criteria that are developed in test scenarios with specific test kits, approval procedures for changes and a test environment that is separate from the production environment. Numerous medium-sized organizations have only partially completed the requirements. This is not a matter of unwillingness but mostly inability because all of the work is hardly affordable.

This way risk and accidents arise, for example: sending correspondence to deceased patients or renters, losing medical data, inability to send out invoices for months et cetera. The risks are further amplified when complicated arrangements get introduced under high time pressure, whereby further testing is put under even more pressure.. As were when the complex Diagnosis Treatment Combinations (DTC’s) were introduced into the healthcare industry with a high tempo, and it’s conceivable that at numerous organizations calculation errors are made that stay(ed) unnoticed. This has happened before, even with banks that have much more money to spend on IT compared to medium-sized organizations. According to a report in the Dutch newspaper the Volkskrant, the ING bank discovered that in 2005 they miscalculated many commissions, for years, and they had to repay 70 million euros to their customers.


One can, with a simple checklist verify if their organization has supplied the basic requirements for good testing. If that is not the case, apparently they trust the software supplier to test her own software adequately before it’s released. With this expectation they will not only be disappointed regularly, but it’s also not realistic. Every organization should carry its own responsibility to control critical business software before taking it into production this is a fortiori the case for the many mutual links between different software packages. Many software suppliers also indicate this by mentioning it in their terms and conditions, that the customer automatically accepts a release when they put the software into production.

For the medium-sized organizations, however, it’s difficult to bear that responsibility, not only because of the cost but sometimes also because of externally imposed deadlines, think of adjustments within legislation, new policy by the government or European guidelines, et cetera.

If one realizes that the customers of a similar software developer do the same test effort on the same moment, then there is also a solution to the problem above. By merging these efforts and placing it under professional direction, the test approach and software quality are improved for an acceptable price. This approach is in fact a professionalization that requires close cooperation between software users and suppliers.

Affordable test-solution

Affordable and good quality testing is available for the medium-sized organizations.

  • Harmonization of acceptance criteria are possible if midsize organizations work together with peers in the branch.
  • A basic test scenario can save on work.This requires only additions for the individual organizations, an extension for specific arrangements or custom software. In essence the basic test scenario shows the uniformed acceptance criteria.
  • The means to improve productivity of the tests only have to be developed once and maintained. The software supplier can play an important role here
  • Professionalization of the findings administration may help the supplier to improve the quality of his development process.

Snakes in the grass.

The legal division of responsibilities. The test organization can in this approach not give a 100 percent coverage for the quality of the tested software. Because the test organization is working for multiple organizations, each with its own production environment, multiple test environments have to be maintained. The management of licenses can be costly if the software developer doesn’t want to support this approach and create obstacles in the form of high licensing fees.

Basic requirements for good testing

The following basic requirements are needed for decent testing. Medium sized organizations want to suffice to these terms, but usually they can’t.

Measurable acceptance criteria have been formed.

Without a measurable acceptance an information system can’t be accepted with a foundation. But in reality during the selection and purchase of software the extensive programs of demands is hardly refashioned to the measurable acceptance criteria of the ISO-9126 quality attributes: functionality, reliability, usability, efficiency, maintainability and portability. For example: errors in the blood group of a patient are unacceptable, errors in a postal code are mostly just annoying.

– The test environment and the production environment are separated but duplicates.

In reality when transferring software to the production environment settings often have to be modified. There is a real probability of creating unwanted errors, while the chance of quickly discovering them is minute, especially when an erroneous test database gets connected with in there a recent copy of the production database.

– The acceptance criteria are concretized in test scenario’s.

Only with this approach a systematic and planned consideration of satisfaction of the information system can be made. Testing by simply finding some usable situations from the production database, is clearly inferior and often insufficient of depth. However, the preparation of test scenarios is labor-intensive and costly, and therefore done too little.

– There are tools available to increase productivity of testing.

Testing is too much manual labor while systems are getting more complex and need more test actions. Automated testing of new releases saves a lot of work, performance testing can hardly be done without automation and even the possibility to change system dates can save a lot of work. However, most medium sized organization only have a limited toolkit of test aids.

 – The findings get documented and are used by the suppliers management.

Testing usually is carrying water to the sea because a new release contains errors and bugs that were solved at a prior release, after much testing. Then serious conversations should be made with the software supplier about the development procedures. And if you tested you’ll have all facts on the table.

Drs. René Ceelen ( is the founder of TESTMONITOR, the specialized tool for software testing and implementations.

Drs. Jaap van der Wel ( is director of Comfort-IA and a member of the standards commission for Information security in the Healthcare industry.

If you’re a newcomer in the world of testing, you easily become overwhelmed by all the testing terminology getting thrown around. In this article, we’ll try to explain some of the jargon you might encounter in your daily testing life, just to make it all a little bit more understandable.

Test Cases

So, we’ll start with test cases. Most importantly, test cases provide a series of instructions that validates some part of the software is doing it job as expected. Say, you require a contact form on your website that sends an e-mail when a visitor leaves a message, you’ll probably create a test case verifying the e-mail actually arrives when the form is filled and submitted. A test case can help you determine whether or not the software meets your requirements.

In some situations, a single test case can verify a simple requirement. However, it is more common practice to think of multiple situations that tests your feature more thoroughly. When your contact form offers an optional text field, try both filling and clearing the field in two separate test cases. Even better: try to go beyond the “happy path” and think of things the software shouldn’t do. If your contact form accepts an e-mail address, why not check if a non-valid address is accepted too?

Test cases do not merely consist out of instructions – or test steps, as we’d like to call them. For example: simply performing a few steps without explicitly telling the excepted outcome could cause misunderstandings. Should our contact form rejected an invalid e-mail address? Perhaps – or perhaps, we really don’t care. A proper test case usually comes with a predefined expected result, telling you what to expect when completing a test case. Similarly, it is common good practice to define the initial expected state when writing a test case. You don’t want to cover every piece of preparation in your test steps, like accessing a test environment, setting up a browser, enabling mail notifications, etc. These pre-conditions are usually defined as a separate property of the test case. When a condition isn’t met, there’s no point in testing the test case.

So, that’s it then? Well, there’s more. Like writing a blog post, creating test cases require certain writing skills, like imagining things from the end-user’s perspective. If you want to read more, be sure to read 7 tips for great test cases.

Test Suites

When you know about test cases, the concept of test suites should be really simple. With a growing amount of test cases, the need for categorizing them increases as well. You don’t want to throw all your books on a pile, you’ll organize them on shelves, for better accessibility. In a way, test suites are very similar to shelves, as they group test cases together as well.

In the process of planning and running tests, keeping track of hundreds of test cases really becomes cumbersome. Thinking in terms of “We need to run a performance test tomorrow” or “Next week we should schedule a test for our website” forces you to know the purpose and domain of each test case. A test suite allows you categorize test cases in such a way that they match your planning and analysis needs. Do you run functional and performance tests? Create two suites and label them accordingly. Do you test different applications and want to monitor them? Create a suite for every application. Are you still able to manage without test suites? Don’t use them! Remember, they are only there to help you organize test cases when needed.

Test Runs

Finally, we end up with test runs. They are little crossroads in your testing project, where several aspects come together. Simply put, test runs determine which test cases are tested by which user(s) at what particular time. Quite a mouthful!

Image you have created your test cases for your contact form and its about to become available to your organization for testing. Are you going to test it for yourself, or do you pick someone else? When are you going to start testing – and, when do you want to gather the results? Which test cases are you going to select – all of them, or just the ones marked “happy flow”? These questions should be answered when planning a test run.

When you have created a test run, you have reached an important milestone: people are able to run a test, using your test cases, grouped by your test suites. In one of our next posts, we’ll continue with some useful tips running tests and gathering test results.

Testmonitor Brochure

Find out more about TestMonitor!