A Guide to Software Test tool

A Guide to Software Test tool

Overview

Whether you are buying or making software, test tools help you get the most from your investment. A useful test tool gives you far more than efficiency; it will help you avoid trouble, provide vital information, and can enable your organization to take new opportunities with greater agility and strength.

Testing a system well is harder than building it. In the face of new risks introduced by the ever-increasing complexity of software and hardware, test tools may be your only practical way to be confident that you've got just what you paid for. From gathering the initial requirements to supporting the system after it has been deployed, test tools are available to support all phases of a project. This article looks at the ways that test tools can be used to design, code, and (of course) test a system.

Tool Vendors and the Test Tool Market

A handful of large vendors dominate the enterprise test tool market, offering integrated tool suites around a core product, and supporting the full project life cycle on a range of platforms. However, many users swear by tools from smaller vendors who offer a single tool, concentrate on one aspect of a project, or specialize in a particular industry, region, or technology

The best-recognised, and perhaps most numerous tools, are test automation tools. These tools allow a machine to take charge of running a predetermined test, allowing those known tests to be executed quickly, accurately, and repeated as desired. Although clearly powerful, test automation tools have historically had a mixed reputation amongst practitioners—but have become more favoured as their technology, and their usage, has become more sophisticated.

However, the market supports a wide range of tools that may give you greater bang for your buck, some integrated with test automation, and some offering an entirely different way to address the quality of your system. Buyers who ignore these tools may be missing the opportunity to make the most of their investment.

Test Automation Tools—a General Description

Test automation tools fall into three broad categories; unit test tools, capture-replay tools, and load test tools. Each is described in the sections below. However, all test automation tools are built on the same basis. All use detailed scripts to describe tests and a set of expected results—and while the construction of these scripts may be hidden from the tester, they nevertheless introduce software engineering complexities. Trying to simulate manual tests—in particular working with point-and-click interfaces—introduces further complications.

Most tools incorporate functionality that helps to deal with this � from providing a fully-fledged development environment to recognizing objects in a GUI. Some tools allow skilled testers to leverage the tool's abilities with sophisticated scripting techniques. However, if used simply to mirror manual testing, test automation tools may help to confirm existing value, but are unlikely to highlight new risks.

Automated testing can produce unwieldy amounts of raw information; many tools provide functionality to process this into a more palatable form.

Tools Used while Designing a System

Design tools allow fundamental information about the plans for a system to be captured and shared, and are typically set up and used most intensely in the early stages of a project. Early consideration of testing can pay great dividends later in the project, and many design tools are built with testing in mind.

Requirements capture and analysis tools allow requirements to be shared across the whole team and controlled throughout the life of a project by creating a central repository and archive. They often allow requirements to be linked to documentation, code, and tests. Requirements management tools may be stand-alone, or part of a larger document management system. Complex tools may incorporate sophisticated natural-language parsing and validation, while simpler tools more closely resemble list managers.

Visual modeling tools allow system architects and designers to make models of the system and to visualize those models in a variety of ways. Formal diagrams may be used to create models that can be converted into code structures, tests, and data schemas.

Tools Used while Coding a System

Test tools used during coding are generally closely linked to the structure of the code. They are often called "white-box" or "clear-box" tools—a reference to their need to have visibility of the workings of the system. Many tools can be integrated with the build environment, running automatically in the background. These tools do their work on individual pieces of code, rather than integrated and configured systems, and tend to be specific to a particular language or technology.

Static test tools examine written code—but do not run it. They are an effective way of spotting a wide range of faults, which may be difficult to diagnose or reproduce reliably at later stages. Tests are based on coding standards and expected faults rather than any direct linkage to the functionality or requirements of the system, and will need to be adjusted to fit working practices. Some static test tools also provide functionality to help developers visualize code structure, measure complexity, and show the way that programs and data are interlinked.

Unit test automation tools allow a component of the system to be tested in isolation, before it is made part of an overall system. These tools typically allow a suite of unit tests to be run on demand, and are an important part of test-driven development and related agile techniques. Aside from managing automated test execution and summarizing test results, they may also monitor the component's use of resources, or even the specific lines of code exercised by a given test. Some are able to assist isolated testing by simulating other resources that the code might need.

Tools Used while Testing a System

After coding errors have been picked up, testing takes a broader, business-focused view of the system. Technical insight into the guts of the system may not be as important as an understanding of business risk and requirements, and tools used during testing are typically "black-box" tools; they do not need much information about the internal structures of the system.

Capture-replay automation tools are test automation tools that create initial scripts by recording the actions of a single person—typically while that person uses a graphical interface. These scripts are modified and replayed at will—deviations from expected actions or their recorded consequences are analyzed and highlighted where necessary.

Load test automation tools (also known as volume test or stress test tools) are test automation tools that replay the actions of many users simultaneously—although they may require dedicated hardware to perform properly, and may entirely avoid the user interface in favor of a deeper interaction. They often provide functionality to monitor behavior of the system as a whole, rather than any individual test.

Monitoring tools (also known as "Flight Recorders") also measure overall properties of the system and its environment during testing, and may allow the measurements they make to be compared historically. They can provide system tuning information and early warnings of unexpected systematic problems, and so are often also used in unit testing and live operation. Fault injection tools simulate problems in the system's environment. They allow tests that would otherwise be particularly difficult to re-create, and can provide a fast and effective route to discovering weaknesses that would be hard to show in an unequipped test environment—particularly those faults that may be exploited by malicious use.

You'd Use These Tools If You Were Managing Testing

Testing involves lots of lists; chiefly, lists of tests and lists of problems. Tools to manage those lists can improve communication and documentation, and help the whole project go more smoothly. Generic tools can not only be used to manage lists of tests and problems, but can be configured to support lists of requirements, enhancements, work assignment, customer contacts and so on. Testing-specific tools have functionality to directly support the requirements of testing and QA teams.

Test management tools allow a list of tests to be grouped, sorted, prioritized and assigned—helping the team to manage the work of testing. More focused tools will report on test progress and may also capture and summarize test results and histories, allowing comparisons and trend analysis.

Test generation tools generate tests within parameters to fit a model of the system under test. While they are most effectively used when integrated with a test automation tool and test management tool, they can be useful analytical tools for manual testing.

Most test teams use some sort of a defect tracker to hold a centralized list of logged problems. The tools allow problems to be classified and prioritized, and to be assigned to individuals as they progress from detection to resolution.

Data manipulation tools work with bulk data sets and databases. They allow a team to analyze output data, and make it simpler to construct accurate and comprehensive test data.

Environment management tools are typically used with the machines in the test lab. They allow the configuration of the lab to be monitored and may help the testers to actively reconfigure their test environment.

Tool Use and Reuse

Many tools incorporate functionality that is not described above—but might be expected of any software tool. Experienced users will know that most tools require user authorization of some sort, change control and archiving are often available, and an application programming interface (API) is always handy. Some tools will suit your environment perfectly—while others were never designed to work on your kit. Some may suit your budget, your existing skill set, or your requirements for a mature, flexible, cutting-edge tool.

Of course, these are generalizations, rather than hard-and-fast categories. Any one tool may incorporate functionality from a range of tool types, and it is not uncommon to see a single flexible tool used for a variety of tasks throughout a project. Exploratory and model-based approaches, in particular, often make use of individual features from an otherwise disparate set of tools. However, tools selected in haste are often found redeployed solely as a doorstop or bookend.

Test Tools Are a Strategic Purchase

Test tools need time, money, and commitment. To work well, a tool needs to become part of your information infrastructure and processes. Picking the wrong tool is not only money badly spent, but may create an obstacle that your teams will find hard to overcome. Picking the right test tool will do much more than increase your efficiency. The right tool will open up a range of otherwise unobtainable opportunities that can differentiate you operationally from your commercial competition, or allow you to compete on a more level playing field.