By James Pasley

The non-functional requirements for a project such as performance, scalability, reliability, high availability, and fail-over have a significant impact on its architecture. However, the creation of tests for these requirements is often left to the end of a project. There are often differences in the terminology used to describe such tests.
The use of a Service-Oriented Architecture (SOA) provides results in an open architecture that lends itself to various kinds of performance testing. This document categorizes these tests, describes their purpose, and summarizes how results should be interpreted.

Tests should be set up as early as possible in the development of the system. The results obtained from testing may require tuning of the configuration, modifications to the architecture of the system to eliminate bottlenecks, or the addition of hardware resources.

Testing Categories

Each category of testing serves a specific purpose. The tests need to be planned with clearly stated goals and criteria for success. Because the results of some tests provide input for others, the tests are listed here in the order in which you might attempt them. However, testing should be started as early in the project lifecycle as possible. The various kinds of tests should be performed in parallel. Finishing one testing task before moving on to the next is a classic testing mistake. See Classic Testing Mistakes [1] for other such classics.
A Note on Terminology: A number of the terms used to describe testing have been overloaded and no industry-wide agreement has been reached. This is particularly true of the terms “performance testing” and "load testing". The categorization presented here has been taken from Types of Tests [2]. Do not become distracted by discussions on the correct terms to use – instead concentrate on the purpose of each task.

Targeted Infrastructure Tests

Targeted infrastructure tests are isolated tests of each layer and/or component in the system.
Approach
The use of SOA means that a Web service interface will probably exist for each of these components. This provides an excellent means of isolating the various components from each other for the purpose of testing. Each of the following kinds of tests (performance, stress, load, and so on) can be performed on the isolated components before testing the system as a whole.
Components need to be isolated from each other for these tests. The test tools can be targeted directly at lower-level services, using the Web service interfaces they expose. Higher level components such as BPEL scripts can be isolated from the services they depend on by creating stub versions of those services that only implement sufficient logic to enable the tests to run.
Purpose
The purpose of these tests is to identify any individual components that would act as bottlenecks and limit the overall ability of the system to deliver at a given performance level.
Interpreting the Results
The results for these individual tests allow predictions for the overall performance of the system to be made. This can be done at a very early stage in the project, even before system integration has taken place. This allows remedial action to be taken at the earliest possible time.
The throughput of a single component may limit the overall throughput of the system. If a single request arriving at the top-level service will result in several requests to an individual component, this should be taken into account when considering the overall throughput requirements.
Response times for the overall system can be predicted based on the response times for the individual components.

Performance Tests

Performance tests identify the end-to-end timing of the system under low load.
Approach
Performance tests need to execute in a production-like environment and model the production system as closely as possible.
Purpose
The purpose of these tests is to set expectations of the ‘best possible’ response time for the system under a given configuration.
Interpreting the Results
Performance testing should validate the prediction of the overall system performance made as the result of targeted performance tests.

Stress Tests

Stress tests determine the load under which a system fails, and how it fails.
Approach
Test the system by slowing increasing the load from the low load used in the performance tests to the point where failures start to occur.
Purpose
The purpose of these tests is to identify the maximum possible load for the system under a given configuration. This can then be compared with the requirements.
It is important to know whether this load will result in catastrophic system failure, or if everything just “goes really slow”.
Interpreting the Results
Identifying a point at which failures start to occur does not mean that the system has failed the tests; after all, the purpose of these tests is to identify that point. However, this should only occur at a point that is greater than the required load for the system. If this is not the case, then remedial action should be taken, such as increasing the specification of the hardware used.
Stress testing also provides the opportunity to throttle the system so that the applications developed continue to operate efficiently and within limits that ensure stability. Throttling can be achieved by setting limits for the rate at which messages are read from input queues, the number of concurrent incoming HTTP requests that will be accepted, and the thread pools used within the server.

Load Tests

Load tests are end-to-end performance tests under anticipated production load.
Approach
Test the system under the anticipated load and measure the response times. A precise definition of the anticipated load is a prerequisite for this kind of testing.
Purpose
A load test may be used to determine the likelihood that the system will meet performance expectations under load. It may also be used to identify the minimum hardware configuration necessary.
Load testing also provides the opportunity to do “background testing”. For systems that include a user interface. This is a form of user-acceptance testing, which is performed while the system is subjected to the expected load.
Interpreting the Results
Verify that response times remain acceptable. How do they vary? For example, the average response time could be acceptable, but if response times vary significantly, this may be unacceptable.
The ability to perform load tests as part of the targeted infrastructure tests as well as on the system as a whole is an important factor in identifying why response times might be unacceptable.

Volume Tests

Volume tests measure the throughput of the system as the nature of the load it is subjected to changes.
Approach
Identify things that directly impact the capacity of the system. For example, increasing message size may have a more significant impact on capacity than increasing the number of messages.
Consider the various aspects of the system where the volume of data could increase and design specific tests for these. For example, allow database table sizes to grow, or message queues to increase in size. If completing a transaction causes resources to be freed up, test the system where transactions are slow to complete or left to timeout.
Purpose
These tests can be used to ensure the system meets performance requirements during the lifetime of the deployment. These tests should also be used to identify operational procedures such as maintenance of a database.
Where operational procedures can be used to alleviate the impact, these tests can identify the appropriate settings for raising alarms. This can provide an opportunity to ensure that SLAs are not violated.
Interpreting the Results
The results of volume testing may be used in conjunction with the results of stress testing to identify the appropriate throttling of the system.
Performance of the system can be expected to fluctuate as the load on the system varies. These fluctuations might be within acceptable limits, or could be controlled with operational procedures. If not, steps to minimize the variations should be taken.

Fail-over Tests

Fail-over tests verify the redundancy mechanisms while the system is under load.
Approach
Where a system involves a cluster of more than one server, test the system under the expected load and then remove one of the servers.
Fail-back testing also needs to be performed to verify that a server removed from the cluster can be successfully added back in.
Purpose
These tests ensure that the fail-over mechanisms work and that the remaining components can handle the load exerted on them once one of the components is removed.
Interpreting the Results
Fail-over is used to ensure that the system remains available even in the event of a failure of an individual component. For a system to be truly redundant it must have at least two of everything. Mean time between failures should also be considered resulting in the proactive replacement of components before failures occur. Often economic factors prevent this, and redundancy is only provided where the probability of failure is considered to be a risk. If this is the case, a clear statement of where redundancy is not available should be provided as well as procedures to be followed in the event of a failure of non-redundancy components.
Fail-over ensures that the system continues to be available in the event of a component failure. This concept is well understood. It is just as important to define what happens to the requests being processed at the time the failure occurs. Will the failure be reported to the client, who will be required to resubmit the request, or will the client be isolated from any errors. It is important that these requirements are built into the design of the system. For example, isolating clients from such errors may require the use of a store-and-forward transport.

Soak Tests

Soak tests verify that the system can run at high levels of load for prolonged periods of time.
Approach
Set up a load test that is greater than the expected load, but less than the failure point identified by the stress tests. Run the system under this load for an extended period of time. The length of time required for such tests is an important factor in planning. The required soak time needs to be agreed early on and built into schedules.
Operational procedures identified during volume testing may need to be carried out while the soak test is running.
Purpose
The purpose of these tests is to prove that the system can run for prolonged periods of time. This ensures that resources are freed up for reuse once processing of individual requests completes.
Interpreting the Results
The first goal is to ensure that the system stays running for the defined period of time. In addition, the performance of the system should not degrade. Resources such as memory could also be monitored to ensure that usage remains constant.

Network Sensitivity Tests

Network sensitivity tests focus on the Wide Area Network (WAN) limitations and network activity. These are particularly important in a SOA where systems are distributed across the Internet.
Approach
Network sensitivity tests are variations on load tests and performance that are conducted over a larger or more geographically dispersed network, and perhaps in a more "real-life" scenario than the development lab.
Purpose
The purpose of these tests is to ensure that the system, which was probably developed and tested on a single LAN, behaved in the same manner on a WAN.
Interpreting the Results
The nature of the WAN may cause response times to vary more than on a LAN. The throughput of the system may also be less. Intermittent connection errors may be more likely. While the performance characteristics will be different, testing should ensure that performance remains within acceptable limits.

Conclusion

Careful planning, analysis, and design of the system is required to deliver on non-functional requirements. The performance of the resulting system must then be tuned and verified through a series of tests.
Testing needs to start early and tests developed in parallel with the system itself. Addressing the results of testing may require re-factoring of the system, so contingency needs to be built into project plans to allow for this possibility.
The Cape Clear ESB provides a high performance and scalable solution with the features necessary to provide reliability, high availability, and fail-over. Careful analysis, design, and testing will ensure that systems built using the ESB, deliver these non-functional requirements for SOA projects.

References

[1] Classic Testing Mistakes, Copyright ©1997 by Brian Marick,

[2] Types of Tests, Copyright © 2004 RPM Solutions Pty Ltd,