Writing High-Quality Test Scripts: Taming the Chaos

Writing High-Quality Test Scripts: Taming the Chaos

The Complexity of Testing Requires Robust Solutions

The automotive world is evolving at a breathtaking pace. The technology and software that companies produce to drive this evolution are becoming more advanced at a faster rate than ever before. To make both the vehicles and the road systems they run on work together to produce a functionally safe ecosystem, these component elements and systems must be thoroughly tested.

Several ingredients go into the recipe for trustworthy and effective testing. There are stationary elements in the environment, such as the roads themselves as well as roadside objects like structures, signs, pavement markings, and light poles. Then there are dynamic elements that are constantly changing, like weather (and thus road conditions), and driver alertness and focus. There are also elements that may or may not move and interact within the environment at a given moment in time—animals, people, and other vehicles, all of which can be quite unpredictable. And lastly, there are more predictable elements, such as the laws of physics and how they exert influence on all the aforementioned unpredictable objects in mathematically predictable ways.

All these elements and more combine to create a snapshot of a scenario in which a vehicle can be operating at a given place and at a given time. And then, like the frames in an animated movie, these snapshots play out in a timeline stream to provide us with a dynamic and constantly changing view of the immediate world within which the tested vehicle is immersed and with which it is interacting.

However, the complexity does not end there.

In addition to all of the aforementioned ingredients that go into a given test scenario, there are also those that go into the test processes themselves. These are used to test and grade the performance of that vehicle in that environment. Test development processes are used to create a test for the vehicle itself. But likewise, the test itself must be tested to ensure that it is accurate and therefore trustworthy. We test the vehicle, we test the operating environment, we test the test, and we test the testing.

So, how is all that testing kept organized and focused? By following vetted processes in their proper order. Standards define how to set up a testing team and the processes through which they will work. Standards also define how to set up a testing scenario and properly define all the complex elements that feed into it. How is order brought to this chaos once all those elements and processes start interacting with each other in real time? The answer: writing test scripts.

What Is a “Test Script”?

In a highly technical environment, the word script can mean different things to different people. For this discussion, a test script is a planned process used for running programs that gauge the performance of different components. In this sense, the function of a test script is not that far removed from the function of a script written for a theatrical production. To better illustrate the purpose and power of a test script, let’s compare the two.

In the instances of both a theatrical script and a test script, the script provides context and specific instructions for a constantly changing environment. Certain elements must be or do certain things at certain times on the “stage,” driven either by a sequence of events, timing, or both.

In the case of a theatrical performance, the script tells the actors what to say in a specific sequence, and how to act and react to other actors and elements; it instructs them when to move and what to do as they move. Likewise, a test script defines a specific sequence of events. It tells the elements in the test scenario how to behave, how to move (if they are supposed to), and how to react to the other elements.

A theatrical script provides instruction within the context of a performance that is defined by the scope and duration of a sequence of scenes acted out in a prescribed order. Each scene is a unique instance of the same leading actors in different situations interacting with the rest of the cast as the script calls for them to come and go. The script is intended to be read and utilized by human actors, and the sequence of events is defined in the script. The quality of the performance is measured by the reaction of the audience. Therefore, the theatrical script is predictive, and the quality of the performance is subjective.

In comparison, a test script provides instruction within the context of a test case that is defined by the scope and duration of a sequence of test scenarios acted out in a prescribed order. Each test scenario is a unique instance of the same test vehicle in different situations interacting with other computer-based elements as the test script calls for them to come and go. The script is intended to be read and utilized by computer programs, and the sequence of events is defined in the test script. The quality of the test is measured by evaluating the test results. Therefore, the test script is predictive, the responses of the test elements are reactive, and the quality of the test is specific, measurable, and purely objective.

In other words, a test case is analogous to a scene in a play, but the outputs are very different.

Test Scripts and Test Cases

A test script contains the instructions for the behavior of elements within the test environment and the test vehicle itself as they operate within the scope of a test case. Test cases consist of a series of test requirements specific to that which is being tested, based upon a foundation of preconditions that are defined by the testing team early in the process.

Examining the structural flow which is used to build a test case illustrates how a high-quality test script is created:

1. The requirements of the test case are specific inputs and outputs that developers define by calibrating variable values. Based on the requirements and the inputs and outputs that they define, a test script is made.
2. Because the testing itself is all computer-based, code is programmed to carry out the instructions in the test script.
3. The program is run; i.e., the test script feeds relevant data and instructions into the program, the program executes the script, and the program outputs the results of the test.
4. The test results are evaluated by the appropriate personnel. Where necessary, the script is modified to reflect new elements that need to be tested or attributes that need to be modified to test other considerations and outcomes. Cause and effect are documented in detail so that every change to the script can be traced to both a need and an outcome.
5. The process is repeated as required until functional safety is achieved across the system.

Why Make Test Scripts?

Test scripts drive testing that helps validate the functional safety of automotive technology. Of course, there are many ways of evaluating automotive technology, including simulations with costly prototype test hardware, test track operations, and real-world tests of unvalidated or under-validated components. But most of these other methods carry inherent risks and expense, especially those tests carried out with real hardware out in the real world.

The most efficient test methods reduce variation and eliminate waste, while effectively managing risks. Script-driven computer-based testing is very efficient, revealing the true nature of what is being tested in a consistently trustworthy manner that does not directly expose the real world to risk. These types of tests help engineers to safely and efficiently determine whether the technology is functionally safe.

The test scripts are structured so that issues can be properly documented and then traced to specific instructions in the script. This closes the loop on cause-and-effect, enabling the engineers to observe and measure the correlation between a specific issue and the portions of the test script that may have had an impact on that issue. This degree of detail is imperative both for ensuring accuracy, as well as for documenting more complex issues that may have several inputs and outputs, and a mix of predicted and unexpected outcomes.

Designing and Developing Test Scripts

Developers can draw upon a variety of design techniques, but the choice usually depends on the testing scenario. Test requirements for specific subjects help determine which techniques are used, because each subject is different and will have different requirements. An efficient framework is built around the test case, reflecting qualities such as functionality, compatibility, fault tolerances, and other considerations.

Involving the Right People to Achieve Success

The number of people working on test cases can vary greatly. However, having the right people is far more important than having a default/arbitrary quantity of people. Further, that number will—and should—vary depending on the project. People may be drawn from a variety of departments and, in some instances, even from different companies.

Brainstorming starts with who and what the testing will impact, along with identifying all the pieces involved. Sometimes, testing may not directly involve multiple teams, but can still indirectly affect them in a significant way. Efficient and effective communication can be one of the most significant contributors to the success of a testing project.

Project managers must ask themselves who could really add to the team and help guarantee success. Evaluating and anticipating changes that may need to be made—in software, for example—provides valuable perspective as to which teams should be included. Necessary attention is also called to scheduling and timing issues that may need to be addressed while it is still early enough to resolve them. Often the addition of one key contributor with a unique set of skills and experience can make all the difference.

This type of people-oriented planning is foundational and must be completed properly by key stakeholders before the project moves forward. Proper care is paramount. Having the right balance of scope and depth of knowledge within a team is what will ensure that a good test script is produced in the end.

Establishing the Boundaries, Preconditions, and Assumed Outcomes

After assembling the right team, the next step in the process is to define what needs to be tested. Key stakeholders must sign off on these needs so that the scope and purpose of the project is clearly defined and agreed to by all. A minimalized, customizable test plan is built around this list of testing needs. It is not generic; the test plan is uniquely tailored to the specific subject. This degree of customization, and sign-off by key stakeholders, helps optimize resources to business needs, and also prevents scope creep as the project progresses.

Preconditions can play a significant role in how the test script is developed. It is inevitable that developers will have to make some assumptions around the testing outcomes. Preconditions usually focus on what state the software will be operating in, which in turn has an impact on whether the test will fail or not. The selection of an incorrect operating state can result in an erroneously failed test. Conversely, determining the proper operating state usually boils down to a simple case of cause and effect.

The following is a simple example: If you take an ice cream cone outside, with the precondition that the weather is hot and sunny, you can safely anticipate that the ice cream will melt as expected. A level of heat that generates a temperature above the freezing point of the ice cream is a precondition for melting the ice cream. Similarly, defining straightforward but foundational preconditions such as this are the basis upon which a trustworthy and effective script is created. Attention to detail can really pay off. And sometimes, the most important building blocks are those which are the most basic.

Additionally, certain tests may carry an expected outcome of failure because they were designed to fail. This enables developers to verify that certain applications are operating as intended or expected. For example, if the engineers understand that the oil pressure in a component might fail in a specific testing scenario, and then it does, they at least have an idea as to how to ensure that it will not fail in another testing scenario. The same can also happen for situations where the outcome is positive.

The point is that these outcomes should match the expectations. This kind of alignment between expectations and outcomes reflects that the engineers and developers have an exceptional understanding of how given elements will function under certain conditions.

This is Part 1 in a series on creating high-quality test scripts. In the next installment, we will examine: inputs and outputs; the programming languages used; the impact of peer reviews; and the impact of test scripts on functional safety, now and in the future. 

Interested in learning more about writing test scripts for your organization? Contact our team today!