- New JobEsterlineProgram Manager
The need to test products of all kinds, software and physical products, often runs into challenges connected with:
Software is particularly challenging because the properties worth exercising and assessing in tests are hard to see and measure, and sometimes even hard to define. While there are no silver bullets, there are some approaches to testing that improve the odds of finding the most important software operational and performance weaknesses with reasonable investments of time and resources. One such test approach, often called “usage-based,” has statistical roots and some connections that Lean Six Sigma Black Belts could effectively support and perhaps evolve.
One way to sort out test strategies is by the way each one looks at and plans to exercise the unit, component or system of interest. Such sorts might be:
A “states and transitions” view of any system, where the future behavior only depends on the current state (i,e, it is memory-less) may be modeled as a stochastic network or, more commonly, a Markov Chain (for the mathematician Andrey Markov).
In software, a “state machine” may often have this property – as the way a state machine will respond to a next event depends only on its current state (not how it got there or what happened prior).
Here is a very simple example to walk through the steps involved in building and using a software system Markov Chain. In doing so, some key ways that the model can help inform test planning and assessment can be explored.
1. Understand the scope of users, requirements and system behavior detail to be modeled. Usage-based implies a sense of the scope and nature of actor-triggered events that may be encountered. Sometimes this depends on the class of users (e.g., experts, new users, special-purpose users).
Once clear about what is meant by users, then, which range of requirements and specifications will be in the scope of the model? While it may be tempting to set out to model all aspects of a broad system – common sense and other reality often dictates some divide-and-conquer thinking that carves out parts of the system for test in any one endeavor
Given the user and requirements scope, the related question to answer is: What level of detail? Is it at the highest level, looking at end-user interface events and actions, or at a finer granularity, looking at components and messages triggering actions in side the system? Figure 1 illustrates the range of granularity that a usage-based approach might span. At the high level, a user with keyboard and mouse are often the interface of interest. Transitions and new states mean things like web-page or screen changes
A lower level may depict the interfaces of objects and components, where transitions involve the messages sent and methods invoked. There is a branch of usage-based test at this component level that makes use of things like Unified Modeling Language (UML) state diagrams to build usage-based views. As a simple introduction, it is best to follow the higher level view. Once understood, many of the same notions apply in the finer granularity views.
2. Identify the states and transitions that describe behavior of interest. This is not always as easy as it sounds – but it can actually be an added value part of this approach. The process of thinking through “what are the states and transitions?” may help designers, developers and test people uncover things they were not mutually clear about and to reach a better shared understanding of the system they are working on together. Doing this before design and coding are done is most beneficial, as that shared understanding can prevent defects from being inserted in the first place – the best impact of testing one could hope for.
Figure 1A is a very simple view of a high level states and transitions map. Each state is in a bubble and each transition shown with a directed arc indicating the change from the one state to the other. Note that a state may loop back on itself (e.g., on the home page and clicking home page).
3. Estimate transition probabilities or long-range likelihood. Figure 2 illustrates a simple case with “probability” (the formal Markov terminology) or “long-range frequency” (the equivalent-enough practical meaning) labeled on each arc to show how often that transition would be expected to happen. Transitions can be thought of as one of two ways:
In this case, the article will proceed with an event-driven view.
4. Use the Model to Inform Test Case Sampling Plans. Once the states and transitions model is built it can be used in a few ways.
An interesting property of Markov Chains is the review of “stationary distribution.” In simple terms, this means: “What is the long-range frequency with which we’d expect to see each of the states happening?”
Figure 4A shows a simple Markov Chain depicted as a matrix. Using that form, the statuary distribution can be directly computed (using some matrix algebra). Figure 4B shows the stationary distribution for the simple states and transitions diagram. The interpretation of that is the system will spend about 59 percent of its time in State 2. That might be a first indication of how to allocate some test time and resource. (Obviously, this is more useful in a more complex case than this introductory example.)
While the stationary distribution gives some general guidance, more concrete test planning help can come from the extraction of test case sequences from the usage model. In simple terms that involve:
|Test No.||Usage Sequence|
The table to the right illustrates the first few test cases for this simple example. Note that the first case follows path with highest probabilities, then the next case follows untested paths with the next highest probabilities, etc.
5. Assess test results.
Like a lot of things in the Lean Six Sigma world, the science of usage-based testing originated and evolves outside of Six Sigma – but the practitioner’s toolkit, and potentially that science, can be advanced by bringing the practice onto the Six Sigma radar. Black Belts and project team members can bring their Six Sigma skills to bear in a number of ways that could strengthen usage-based testing, while deriving some benefits as they tailor it to their software test process: