Software-Defined Test

Fundamentals
Understanding the Architecture of Modular,
High-Performance Test Systems
Contents
Executive Summary ___________________________________________________________ 4
Architecture Layer No. 5: System Management/Test Executive _____________________________ 5
Architecture Layer No. 4: Application Development Software ______________________________ 6
Architecture Layer No. 3: Measurement and Control Services ______________________________ 7
Architecture Layer No. 2: Computing and Measurement Bus_______________________________ 8
Architecture Layer No. 1: Measurement and Device I/O __________________________________ 9

Chapter 1: Developing a Modular Test Software Framework _________________________ 10

National Instruments Modular Test Software Framework ________________________________ 10

Chapter 2: Choosing the Right Software ADE for Your Automated Test System ___________ 13
Factors to Consider When Selecting an ADE ____________________________________________ 13
NI LabVIEW ______________________________________________________________________ 15
NI LabWindows™/CVI ______________________________________________________________ 18
Microsoft Visual Studio .NET (C++, Visual Basic .NET, C#, and ASP.NET) ______________________ 20

Chapter 3: Choosing the Right Data Bus __________________________________________ 22

Understanding Bus Performance _____________________________________________________ 22
Instrument Control Bus Comparison (GPIB, USB, PCI, PCI Express, and Ethernet/LAN/LXI) _______ 24
Conclusion: Instrument Bus Performance ______________________________________________ 26

Chapter 4: Modular Instruments Basics __________________________________________ 28

Anatomy of a Modular Instrument ___________________________________________________ 29
Effects of Front End _______________________________________________________________ 31
Bandwidth ____________________________________________________________________ 310
Accuracy_______________________________________________________________________ 31
Effects of ADC/DAC ________________________________________________________________ 33
Sampling Rate __________________________________________________________________ 33
Resolution _____________________________________________________________________ 33
Instrument Types _________________________________________________________________ 36
Digital Multimeters (DMMs) _______________________________________________________ 36
Arbitrary Waveform Generators (Arbs) and Digitizers ___________________________________ 36
Dynamic Signal Analyzers (DSAs) ____________________________________________________ 36
RF Analyzers and Generators_______________________________________________________ 37
Conclusion _______________________________________________________________________ 37

Chapter 5: PXI Modular Instrumentation Platform _________________________________ 38

2
 Hardware Architecture _____________________________________________________________ 38
PXI Chassis _____________________________________________________________________ 39
PXI Controllers __________________________________________________________________ 39
Software Architecture _____________________________________________________________ 43
PXI – Industry Standard for Modular Instrumentation ____________________________________ 43
Why Customers Choose PXI _________________________________________________________ 44
Extension of the PXI Platform: PXI Express _____________________________________________ 45

Chapter 6: Case Studies _______________________________________________________ 46

Microsoft Uses NI LabVIEW and PXI Modular Instruments to Develop Production Test System for
Xbox 360 Controllers ______________________________________________________________ 46
U.S. Navy – Developing Digital Test Equipment for Navy Aircraft Communications Using NI LabVIEW
and the PXI Platform ______________________________________________________________ 49
Sanmina-SCI Exceeds Throughput Goals with PXI Tester and Multithreaded Software __________ 51

©2010 National Instruments. All rights reserved. CVI, FlexDMM, HS488, LabVIEW, Measurement Studio,
MXI, National Instruments, NI, ni.com, and NI TestStand are trademarks of National Instruments. The
mark LabWindows is used under a license from Microsoft Corporation. Windows is a registered
trademark of Microsoft Corporation in the United States and other countries. Other product and
company names listed are trademarks or trade names of their respective companies.

3
Executive Summary
When designing automated test systems and choosing a test system architecture, you need to take into
account several design considerations and challenges including increased device complexity, shorter
development cycles, decreased budgets, and test system longevity. For your test system, determine the
factors that are most important and choose an architecture that best meets your needs.

A modular, software-defined test system architecture, like the one shown in Figure 1, is based on a
layered approach with the following advantages:

Increased test system flexibility deployable to a variety of applications, business segments, and
product generations
Higher-performance architectures that significantly increase test system throughput and deliver
tight correlation and integration of instruments from multiple suppliers including precision DC,
high-speed analog and digital, and RF signal generation and analysis
Lower test system investments by reducing initial capital investment and maintenance cost
while increasing equipment use across multiple test requirements
Increased test system longevity based on widely adopted industry standards that implement
technology upgrades to improve performance and meet future test requirements

Figure 1. Five-Layer Architecture for Developing Test Systems

4
Architecture Layer No. 5: System Management/Test Executive
An automated test system requires the implementation of several tasks and measurement functions –
some specific to the device under test (DUT) and others repeated for every device tested. To minimize
maintenance costs and ensure test system longevity, it is important to implement a test strategy that
separates the DUT-level tasks from the system-level tasks so you can quickly reuse, maintain, and
change test programs (or modules) created throughout the development cycle to meet specific test
requirements.

In any test system, operations are often different and common for each device tested such as system-
level tasks.

Operations Different for Each Device Operations Common for Each Device
• Instrument configuration • Operator interfaces
• Measurements • User management
• Data acquisition • DUT tracking
• Results analysis • Test flow control
• Calibration • Results storage
• Test modules • Test reports

Operations that are common for each device should be handled by the test executive. A test executive
that handles the common operations can save time for your developers because they do not have to
write the same code for multiple devices. They can spend their time writing the code to handle the
operations that are different for each device. Using a test executive also ensures consistency among
common operations and circumvents having multiple developers write the same type of code for several
devices.

You can choose from several test executives. Some companies write their own test executive, and
others opt to use commercially available software such as NI TestStand. You should select the test
executive that is best for your test system, whether you are creating a custom test executive or using
one that is commercially available.

NI TestStand includes the Sequence Editor development environment for automated test system
development, as shown in Figure 2.

5
Figure 2. The NI TestStand Sequence Editor helps you shorten your automated test system development
time.

With the NI TestStand Sequence Editor, you can create test sequences, which automate the execution of
code modules written in any programming language. Each code module executes a test on the device
under test and returns measurement information to NI TestStand. You can automatically log test result
information in a report or database. In addition, systems written in NI TestStand can integrate with
source code control, requirements management, and data management systems.

NI TestStand was designed to address four key areas: (1) simplify complex sequence development, (2)
accelerate complex sequence development, (3) increase code and test system reusability and
maintenance, and (4) improve test system execution performance. These focus areas have led to the
adoption of NI TestStand in consumer electronics for validation and manufacturing test, military and
aerospace applications, the medical industry, and IC characterization.

Architecture Layer No. 4: Application Development Software

The application development environment (ADE) plays a critical role in test system architectures. Using
these tools, you can communicate with a variety of instruments, integrate measurements, display
information, connect with other applications, and more. Ideally, the ADEs used to develop test and
measurement applications provide ease of use, compiled performance, integration of a diverse set of
I/O, and programming flexibility to meet the requirements for a range of applications.

You spend most of your development time working with an ADE, so it is critical to choose one that is
easy to use, supports multiple platforms, and integrates easily with measurement and control services

6
such as drivers. Other features that you should consider when choosing an ADE for developing your test
system are its presentation and reporting features, likelihood of product obsolescence, and the kind of
training and support available worldwide.

Factors to consider when selecting an ADE include the following:

Ease of use
Measurement and analysis capabilities
Integration with measurement and control drivers
Training and support
Multicore support
Operating system independence
Presentation and reporting features
Protection against obsolescence
Upgrades

Examples of ADEs include NI LabVIEW, NI LabWindows/CVI, and Microsoft Visual Studio.

Architecture Layer No. 3: Measurement and Control Services

Choosing test and measurement hardware with scalable software interfaces is another important layer
in defining modular test architectures. Measurement and control services software provides modular
software interfaces for configuring and programming your tests. Examples of these types of software
interfaces include configuration managers such as NI Measurement & Automation Explorer (MAX),
Virtual Instrument Software Architecture (VISA), Plug and Play instrument drivers, and Interchangeable
Virtual Instrument (IVI) drivers. Other examples include instrument drivers provided by your instrument
vendor such as NI-DAQmx.

Configuration Manager
A configuration manager, such as MAX, presents a unified system view of measurement hardware
supported in the measurement and control services software. With MAX, you can define channel names
to organize signals or specify scaling functions to convert digitized signals to measurement quantities.
The key benefit of the configuration manager is the integration with the ADEs, which gives you the
ability to easily integrate multiple measurements into a single application without tedious programming.
With these configuration tools, you can avoid spending time configuring these measurement functions
programmatically.

Instrument Connectivity
Integrating traditional instruments into the test software framework requires technologies such as Plug
and Play instrument drivers and IVI to facilitate the communication with these instruments and their
interchangeability. A Plug and Play instrument driver is a set of functions, or LabVIEW VIs, that control a
programmable instrument. Instrument drivers help you get started using your instrument from your
computer and save you development time and cost because you do not need to learn the programming
protocol for each instrument. With open-source, well-documented instrument drivers, you can

7
customize your operation for better performance.

IVI implements a driver framework that facilitates instrument interchangeability by using a general API
for each kind of instrument and separately implementing the driver to communicate with particular
instruments. Separating the API from the particular driver implementation of each instrument gives you
the ability to design a system using a particular IVI-compliant oscilloscope; after the system is deployed,
you can change the brand and model of the instrument without having to rewrite the test application.

Programming Tools
Drivers can go beyond providing an easy-to-use API by adding tools to facilitate development and save
you time. I/O assistants are interactive tools for rapidly creating a measurement or stimulus application.
The DAQ Assistant, part of the NI-DAQmx driver, is an example of an I/O assistant. The DAQ Assistant
presents a panel to the user for configuring common data acquisition parameters without programming.
The combination of easy-to-use assistants and powerful programming environments is necessary to
provide rapid development and the capabilities to meet a variety of application requirements.

Architecture Layer No. 2: Computing and Measurement Bus

At the center of every modern automated test system is a computer in the form of a desktop PC, server
workstation, laptop, or embedded computer used with PXI and VXI. An important aspect of the
computing platform is the ability to connect (and communicate) with multiple instruments in a test
system. Several instrumentation buses including GPIB, USB, LAN, PCI, and PCI Express are available for
stand-alone and modular instruments. These buses have different strengths that make some more
suitable for certain applications than others. For example, GPIB has the widest adoption for instrument
control and availability of instrumentation; USB provides wide availability, easy connectivity, and high
throughput; LAN is well-suited for distributed systems; and PCI Express delivers the highest
performance.

The widespread use of the PC has generated the proliferation of high-performance internal buses
including PCI and PCI Express, which offer the lowest latency and highest data throughput or bandwidth.
The PCI bus provides up to 132 MB/s of bus bandwidth, and PCI Express, an evolution of PCI, can scale
up to 4 GB/s to meet growing bandwidth needs and can provide complete software compatibility with
PCI. Figure 3 illustrates the latency and bandwidth performance of the most popular instrument control
buses. Oftentimes, you need a test system that incorporates multiple buses to maximize performance,
longevity, and reusability.

8
Figure 3. In an instrument control bus comparison, PCI and PCI Express provide the best bandwidth and
latency or overall throughput performance.

Architecture Layer No. 1: Measurement and Device I/O

The final architecture layer, measurement and device I/O, includes the instruments and modules used in
your test system. Depending on the computing and measurement bus you chose for layer No. 2, this
layer can include VXI and/or PXI modules; GPIB, LAN, and/or USB benchtop instruments; and PCI or PCI
Express instruments, as shown in Figure 4. Most automated test systems have several I/O devices
connected by a mix of instrumentation buses. After defining the DUT or DUTs and the types of tests that
you need to perform, you can choose the appropriate device I/O for your test system.

With a modular approach, you can define the test system measurement functionality and build systems
that scale to meet future demands. Using a modular, software-defined approach, you can make custom
measurements, perform measurements for emerging standards, or modify the system if requirements
change to add instruments, channels, or new measurements. The combination of flexible, user-defined
software and scalable hardware components is the core of modular instrumentation.

Figure 4. Example of a Hybrid Test System Based on PXI, LAN, and USB Instruments

9
Chapter 1: Developing a Modular Test
Software Framework
Designing efficient test systems requires a modular software architecture and development tools
optimized for test. To develop test systems faster and more cost-effectively, it is critical that you
evaluate your test software architecture to maximize code reuse. Examining your test software
architecture consists of evaluating the software development tools you are using and studying your
approach to test code development. Understanding the importance of modular test software
architectures and how to develop your tests as modules rather than building stand-alone applications
significantly improves your test software reuse.

Incorporating modular test software architectures begins with choosing a software development
environment that is designed for easily connecting to your instruments and quickly performing any type
of measurement and analysis required for your tests. Such test software development tools include NI
LabVIEW, LabWindows/CVI, and Measurement Studio for Visual Studio .NET. Using the proper test
development environment, you can more easily share your test programs with others in your group and
across test departments in your organization.

Test modules created by design engineers for prototype and validation tests are an integral asset to
production test departments. Production test engineers can easily integrate tests developed during the
prototype and validation phases into final automated test systems using industry-standard test
management software such as NI TestStand.

NI TestStand provides built-in test management capabilities such as test module adapters for calling
tests written in common test languages such as LabVIEW, LabWindows/CVI, C/C++, and Visual Studio
.NET regardless of the function prototypes defined for each test. Maximizing code reuse between the
product design and manufacturing teams reduces test development effort, so production schedules are
met and demands for improved quality are upheld. The flexible NI TestStand module adapters ensure
maximum code reuse across the product development cycle with minimal training and code.

National Instruments Modular Test Software Framework

The management level of the modular test software framework is responsible for directing the
execution of the whole test system. The open software architecture of NI TestStand, a popular choice
for test management software, greatly reduces the effort required to implement a scalable test
software framework. It provides a completely modular and open architecture that you can use “as is”
off-the-shelf or as individual components for designing your completely customized test systems based
on NI TestStand. Figure 1 shows the NI TestStand architecture.

10
 Figure 1. NI TestStand Test Management Software Architecture

The center of the NI TestStand architecture is the NI TestStand Engine, a powerful multithreaded test
server with complete and extensively documented APIs. By communicating with the NI TestStand
Engine, the module adapters provide an open language interface to automate tests written in any
language. The process models offer superior modularity between the test code and the system-level
functions that must be performed. The Sequence Editor provides an easy-to-use and powerful
development environment for test sequences. Lastly, the operator interfaces are included in source
code in multiple programming languages for rapid customization to match your exact needs.

NI TestStand Engine
All parts of the NI TestStand architecture are directed by the NI TestStand Engine, a set of libraries that
export an ActiveX/COM API. With the API, you can perform any operation on the NI TestStand Engine
programmatically by taking advantage of the more than 1,400 exported functions. The NI TestStand
Engine implements multithreading so you can increase throughput by testing multiple units
simultaneously. You also no longer have to include limit testing in your test code by implementing this
functionality natively. By not including limit testing, test code becomes more flexible and reusable.
Another feature of the NI TestStand Engine is that it implements flow control functionality much like any
programming language. Finally, it increases the security of your test system by implementing multilevel
user access and management.

Module Adapters
To call code written in different languages, the NI TestStand Engine uses the different module adapters
available with NI TestStand. The module adapters provide an open language interface between the NI
TestStand Engine and your test code written in LabVIEW, LabWindows/CVI, .NET, C/C++ DLLs,
ActiveX/COM, and HT Basic. By calling code in different languages, you are able to take advantage of any
legacy code as well as newer technologies. You can send and receive information from your code
modules using an arbitrary number of parameters or by leveraging the NI TestStand API. Other module
adapter features include stepping into code modules for debugging and leveraging code templates to
improve programmer productivity.

11
Process Model
Testing a DUT requires more than just executing a set of tests. Usually, the test system must perform a
series of operations such as identifying the DUT, logging results, and generating a test report. The set of
such operations and their flow of execution are called a process model. Process models provide superior
modularity between the test code and the system-level functions that must be performed by using
these functions with multiple test sequences. NI TestStand is shipped with three process models that
you can use as is or fully customize. The Sequential process model tests one unit at a time and the Batch
and Parallel process models, featuring NI TestStand multithreading functionality, test more than one
unit at the same time.

Sequence Editor
The Sequence Editor offers all the functionality and tools you need to develop the most sophisticated
automated test systems. In the Sequence Editor, you can create, debug, and modify test sequence files.
These files contain test steps that can include code modules developed in any test programming
language. Furthermore, the Sequence Editor features a utility to build deployment packages to ease the
distribution of test sequences and operator interfaces. The Sequence Editor also provides user
management services, which prevents some users from accessing restricted functionality based on the
privileges defined by your NI TestStand administrator. Figure 2 shows the Sequence Editor displaying a
test sequence written in LabWindows/CVI.

Figure 2. NI TestStand Sequence Editor

Operator Interfaces
Lastly, an operator interface is a customizable user interface for NI TestStand that you can use to
execute and debug test sequence files created in the Sequence Editor. The operator interface is typically
used on a manufacturing floor or if you need to deliver a custom look and feel to your test or validation
system. NI TestStand operator interfaces use NI TestStand user interface controls, which facilitate
development by fully implementing common features such as sequence file display and execution
tracing. NI TestStand includes ready-to-run operator interfaces written in LabVIEW, LabWindows/CVI,
C#, Visual Basic, and Visual Basic .NET.

12