In our earlier post of this series, we made discussion on “How to design effective test cases? Benefits of writing test cases & the essential design elements of the test case”. Now we shall discuss about “When test cases should be written? Information objectives & Tests intended to expose defects”.
When test cases should be written
As requirements are developed, create your test cases once each requirement is at the approval stage. This has many benefits. First, it forces the test team to polish the requirement and brings to light insufficient or inaccurate detail for the requirement. Additionally, it ensures the requirement is fully testable. Last, it allows the developers to review the test cases prior to coding. This illuminates your test plan and developers will spend time testing each scenario before it enters QA, saving valuable re-work time.
Information/Core Objectives
So what are we trying to learn or achieve when we run test cases? Here are some examples:
Find defects: This is the core objective of testing. A test is run in order to generate failures that reveal defects. Generally, we look for defects in all interesting parts of the product.
Increase bug count: The difference between this and “finding defects” is that total number of bugs is more important than reporting. We may focus hardly, on only a few high risk features, if this is the way to find the most bugs in the defined time.
Obstruct untimely/early product releases: The tester stops early consignment by finding bugs so serious that no one would ship the product until they are fixed. For every build decision meeting, the tester’s goal is to have new bugs on his/her credit.
Help managers make ship / no-ship decisions: Managers are generally concerned with risk in the field. They want to know about coverage/exposure (say, how much of the product has been addressed and how much is left), and how important the known problems are. Problems that appear important on paper but will not lead to customer disappointment are possibly not related to the ship decision.
Reduce technical support expenses: Working in combination with a technical support or help desk group, the test team recognizes the problems that lead to calls for support. These are frequently linked to the product under test.
Evaluate compliance to specification (requirement): Any claim made in the specification is checked. Program characteristics not addressed in the specification are not (as part of this objective) checked.
Fulfillment to system/rules: If a rule states a definite type of coverage (such as, at least one test for every state made about the product), the testing people creates the suitable tests. If the rule states a style for the specifications or other documentation, the testers most likely to check the methods or processes. In general, the tester is focusing on anything covered by rule and nothing that is not covered by rule.
Discover secure scenarios for use of the product (find ways to get it to work, in spite of the bugs): Occasionally, all that we are looking for is one way to do a task that will constantly–one set of instructions that someone else can follow that will reliably deliver the benefit they are supposed to lead to. In this case, the tester is not looking for bugs. He is trying out, practically refining and documenting, a way to do a task.
Review and Assess Quality: This is a difficult goal because quality is multi-dimensional. The nature of quality depends on the nature of the product. For example, a web application that is rock solid but not interesting possibly fed-up its users. To assess or evaluate quality is to measure and report back on the level of quality, we possibly need a clear definition of the most important quality criteria for this product, and then you need a theory that relates test results to the definition.
Verify correctness and precision of the product: It is impossible to do this by testing. We can show that the product is not correct or we can reveal that we did not find any errors in a given time using a given testing strategy. On the other hand, we cannot test in detail, and the product may fail under conditions that we did not test. The best we can do (if we have a solid, realistic model) is assessment that is the test-based evaluation of the probability of errors.
Assure quality: Even though the general title, quality assurance, we cannot assure quality by testing. We cannot assure quality by assembling metrics. We cannot assure quality by setting standards. Quality assurance involves building a high quality product and for that, we need skilled people throughout development who have time and motivation and have decent balance of direction and creative freedom. This is out of scope for a test organization. It is within scope for the peers (project manager and related executives). The testing company can surely help in this process by performing a broad range of technical investigations, but those investigations are not quality assurance.
Tests (Test Cases) Proposed to Expose Defects
Let us summarize our focus to the testing group that has two main goals:
- Find bugs that the rest of the development group will consider relevant (worth reporting), &
- Get these bugs to be fixed.
Still within these goals, tests can be excellent in many different ways. For an example, we could say that one test (test case) is better than another if it is:
More powerful: We define power in the normal statistical or mathematical sense as more likely to expose a bug if it the bug is there. For an example, Test A can be more powerful than Test B for one type of bug and less powerful than Test B for a different type of bug.
More possible to produce important results: A problem is important when a stakeholder with authority would complain when the problem is not fixed.
More convincing and reliable: A reliable test (test case) is more likely to be taken as a realistic set of operations by the programmer or another stakeholder with influence/authority. “Corner case” is an example of a saying used by programmers to say that a test case or bug is non-reliable: “Nobody would do that.” A test case is convincing when some (or all) stakeholders agree that it is realistic.
More useful for troubleshooting: For an example, test cases written in more detail will often break down the system under test without providing a great deal of information about the related test conditions needed to reproduce the problem. They are usually not useful for troubleshooting. Tests that are harder to repeat are less useful for troubleshooting.
More informative: A test case provides value to the extent that we learn from it. In most cases, we learn more from the test that the program passes than the one the program fails, but the useful test case will teach us something whether the program passes it or fails.
For an example, if we have already run a test case in several builds, and the program reliably passed it each time, we will expect the program to pass this test again. Another “pass” result from the reused test does not contribute anything to our intellectual model of the program.
Appropriately complex: If the program has many bugs, a complex test case might fail so quickly that we do not get to run much of it. Testers that rely mainly on complex tests complain of blocking bugs. A blocking bug causes many tests cases to fail, preventing the tester from learning the other things about the program that these tests are supposed to expose. Therefore, early in testing, simple tests are desirable. As the application gets more stable, or as more established features are included into the application, greater complexity becomes more desirable.
At times, we test to understand the product, to learn how it works or where its risks might be. Later, we might design test cases to expose faults, but mainly early in testing we are interested in learning what it is and how to test it. In our forthcoming post, we’ll discuss “Test specification and Test case templates”.
