by Leo Cordaro
Sr. Systems Engineer at MAS

When giving the challenge of designing an automated test system, it is important to evaluate all the tools and possibilities available before developing you application. You may find that what you’re looking for has already been developed, or there are tools available to help meet your objectives. When designing an automated test system and developing a test system architecture, there are several design considerations and challenges that need to be taken into account besides the technicalities of the software. For example, you must contend with the evolution of your product as it increases in complexity, shorten development cycles, decrease budgets, and test system longevity and maintenance.

A test executive application is a framework to help organize and execute a series of tests for a product or device. In other words, the test executive gives you the framework to manage many test steps. This is a modular software architecture, a test executive decouples the software that is specific to testing the product from the rest of the test system. The test executive brings all the common tasks to the table, such as device drivers, logging, reporting, configuration settings, etc. The test steps are the only unique item for your product, sequencing the tests and logging data are the common tasks.

Many industries can benefit from using a test executive, the framework provides a starting point for many different product testing applications. The framework is not limited to high volume manufacturing environments, it can be used in research and development as well. For example, in an R&D environment, a test executive allows the user to quickly create a sequence of test steps to run and collect data without worrying about other software nuances (such as logging routines, database connectivity, etc.). On the other hand, using a test executive in a manufacturing environment gives you the flexibility to test products sequentially, or if increased production is necessary, multiple devices in parallel.

We certainly don’t recommend finding a new job, so this leaves you with two options, build or buy? Typically we have found that some developers will attempt to create their own in-house test executive before fully realizing the magnitude of the task. While this may be a viable option, it is important to consider all of the alternatives before embarking down this path. Developing a test executive from scratch is not trivial, and requires many hours of proper design and planning. For example, here are a few items that you must take into account to properly design a test executive:

• Test Sequence Editor: You will need to have the ability to configure each test step, the sequence editor provides the capability to configure each test step and allow you to sequentially build a list of all test steps needed. For instance, configuration requirements for a voltage test step may include the pin numbers to measure voltage across, expected voltage range, and what to do if the step fails (do you continue to the next test step, or abort the test?). Also, you may need to make multiple voltage measurements, so rather than duplicating the test step and creating another physical copy of the step, the sequence editor will allow you to reuse the same voltage test step, just change the configuration items.
• Are there different tests needed for your product line? For example, if multiple models exist for your product, you may need to run different tests. The Sequence Editor will give you the ability to create the sequence of test steps based on the product model.
• Test Sequence Execution Engine: This gives you the ability to load a test sequence file and run through the test steps. The execution engine should handle all errors generated by the test steps, log all results and display results to the operator. In some cases, the executive will need to run multiple devices at the same time, so your engine must be designed to handle serial or parallel execution.

There are many more design considerations not listed that you need to take into account when developing an automated test system. The initial development of a test executive is only the beginning task. Once complete, the software must be maintained in order to add functionality as needed and fix bugs or defects that are found. Many times, this maintenance effort can become a major undertaking and consume months of development time as well. For instance, when new operating systems are released, simply porting the same application to a new OS can be a monumental task.

You must walk a fine line between developing a robust application, considering possible future changes in your product, and not over-engineering and missing your project deadline.

Once business owners and developers realize that an in-house solution takes a significant amount of development effort and investment, they often seek outside help. At MAS, we have designed test executives that have met most company's needs by starting with a basic framework and customizing it. MAS has many years of experience in working with automated test systems and testing products from a variety of manufactures, that our test executive framework encompasses many details that you may not have foreseen. We have recognized that test executives are not a “one-size fits all” solution, therefore, our executive is a starting point for many of the common automated test applications. Also, in addition to our test executive, we always consider using commercially available off-the-shelf technology first, such as National Instruments’ TestStand. Even though this product works right out of the box, there are numerous ways to customize it so that this can become another starting point for your application. A modular system architecture has proven to be very successful for a number of reasons:

• Increased test system flexibility deployable to a variety of applications, business segments, and product generations
• Higher-performance that significantly increase test system throughput and deliver tight correlation and integration of instruments
• Increased test system longevity based on widely adopted industry standards that implement technology upgrades to improve performance and meet future test requirements