Testing web performance

Testing Concepts:------------

The purpose of a testing method is to provide a structured approach and discipline to testing. It includes test planning, managing and controlling the testing process, and testing techniques. The full lifecycle testing approach being introduced in this concepts manual outlines the generic testing processes and techniques needed to test applications. This methodology is targeted for the development project teams (business and technical staff).
When testing responsibility falls outside of the scope of the development team, reference is made to external testing processes for completeness purposes only. As preparations are made to pass the system to a group outside of the development team for testing, it is expected that the project development team will consult the external group or their documentation for guidance.
· Testing Concepts· Test Process Model
· Testing Standards
Testing Concepts, describes the "why" and "what" of the testing discipline. It presents the concepts relating to test planning, test preparation, test execution and test reporting. It covers, at a generic level, every facet of testing; types, techniques, levels, and integration approaches.
Test Process Model, describes the "how," "who" and "when" of testing. It explains the processes, procedures and tools needed to support a structured approach to testing.
Testing Standards, describes the criteria to be met in order for specific levels of testing (such as Operability testing) to be considered acceptable by the organization. In most organizations the standards are owned and managed by the Quality Assurance group.
Testing Courses, available separately, teach the concepts of testing a software application using the Test Process Model and the full lifecycle testing approach and reinforce them with a case study that applies these processes, tools, and techniques.
Since this is a generic methodology, we will only be describing tools and techniques that are generic to testing. The descriptions will be at a level that will give the reader the concept of what the tool or technique is and its relevance to the testing process. Inspections are one example of such a technique. The reader may pursue additional education at a greater level of detail outside of this methodology documentation.
It is suggested that anyone who is a novice to testing should read the concepts document completely from front-to-back and/or attend the course. Experienced test or development personnel can use this document as reference. It is expected that the Testing Process Model content will be tailored to meet the unique needs of your application, project life cycle, and testing requirements. Testing Standards, where available as part of organizational standards, will be used alongside the Test Process Model to insure the application development is conducted according to the needs of the organization.
Table of Contents
1. Introduction to Testing
This technique paper will take you through the concepts necessary to successfully test a software product under development. It outlines a full lifecycle testing methodology.
A critical element in the successful development of any software application is effective testing. Testing moves the evolution of a product from a state of hypothetical usefulness to proven usefulness. It includes the testing of requirements, design, systems' code, documentation, and operational procedures; and is an inseparable part of the development of a product which the purchaser and end-user view as beneficial and meeting their needs. Testing is one of the ways in which a product achieves high quality. FLT helps improve the quality of the product as well as the test process itself.
2. Testing Fundamentals
Testing is conducted to ensure that you develop a product that will prove to be useful to the end user. The primary objectives of testing assure that:
· the system meets the users' needs ... has 'the right system been built'
· the user requirements are built as specified ... has 'the system been built right'
Other secondary objectives of testing are to:
· Instill confidence in the system, through user involvement
· Ensure the system will work from both a functional and performance viewpoint
· Ensure that the interfaces between systems work
· Establish exactly what the system does (and does not do) so that the user does not receive any "surprises" at implementation time
· Identify problem areas where the system deliverables do not meet the agreed to specifications
· Improve the development processes that cause errors.
Achieving these objectives will ensure that:
· The associated risks of failing to successfully deliver an acceptable product are minimized,
· A high quality product (as the application purchaser and user view it) is delivered, and
· The ability to deliver high quality applications is improved.
The purpose of a testing method is to provide a framework and a set of disciplines and approaches for testing of a software application, so that the process is consistent and repeatable.
2.1 Definition
Testing is the systematic search for defects in all project deliverables. It is the process of examining an output of a process under consideration, comparing the results against a set of pre-determined expectations, and dealing with the variances.
Testing will take on many forms. As the form and content of the output change, the approaches and techniques used to test them must be adapted.
Throughout the discussion of testing concepts, we will use several terms that are fundamental:
· Validation: the act of ensuring compliance against an original requirement. An example is the comparison of the actual system response of an on-line transaction to what was originally expected, requested, and finally approved in the External Design
· Verification: the act of checking the current work product to insure it performs as specified by its predecessor. The comparison of a module's code against its technical design specifications document is one example.
· Process: a series of actions performed to achieve a desired result that transforms a set of inputs (usually information) into useful outputs
· Expectations: a set of requirements or specifications to which an output result of a process must conform in order to be acceptable. One such example is the performance specification that an on-line application must return a response in less than two seconds.
· Variances: deviations of the output of a process from the expected outcome. These variances are often referred to as defects.
Testing is then a process of verifying and/or validating an output against a set of expectations and observing the variances.
2.2 Ensuring Testability
Another important but difficult term to clarify is what is meant by a testable condition. To illustrate the definition, let’s discuss the above term, expectations, further.
In order to be able to assess if an output meets or exceeds a given expectation, an expectation itself must be stated in testable terms. This means that when the characteristics of an output under test are compared against the expected characteristics, they can be matched in a clear and unambiguous way. You would not request that the answer to a calculation of (1 + 4) is to be an "appropriate" amount. To be testable, you would specify the answer as a single value like 5 or as a range between 0 and 10 if the exact answer were not known. If the result of the test were anything other than 5 or a number within the specified range, the test would unequivocally fail and you would record the variance. The example, of course, is trivial but it serves to illustrate the point.
While you may strive to have requirements stated in testable terms, it may not always be possible. The required level of detail when you document and test the requirements may not yet be available. The process of requirements specification and external design should evolve the functional request from a collection of imprecise statements of user needs to testable user specifications. Even though the root of the word specification is specific, achieving specific requirements is an extremely challenging exercise. At every point in the specification process, you can check the "testability" of the requirement or specification by ensuring it is S.M.A.R.T. This acronym stands for:
S Specific
M Measurable
A Attainable or Achievable
R Realistic
T Timely
These specifications will form the basis for the criteria upon which the system is tested and ultimately accepted by the system purchaser and end-user.
2.3 Testing Principles
Here are some powerful basic principles of testing. Although they are expressed simply and most of them appear to be intuitive, they are often overlooked or compromised.
· An author must not be the final tester of his/her own work product.
· While exhaustive testing is desirable, it is not always practical.
· Expected results should be documented for each test.
· Both valid and invalid conditions should be tested.
· Both expected and unexpected results should be validated.
· Test Cases should be reused, i.e., no throwaway test cases unless the work product itself is throwaway.
· Testing is a skilled discipline (on par with such skills as technical coding and analysis).
· Testing is a "no-fault" process of detecting variances or defects.
2.4 "No-Fault" Testing
It is important to realize that the introduction of variances is a normal and expected consequence of the human activity of developing any product. How many times have you heard ... "To err is human ... "? This implies that testing should be a no-fault process of variance removal.
Testing is a cooperative exercise between the tester and developer to detect and repair defects that the development process has injected into the product. It should be apparent to anyone who has tested that this process of removing defects is complex and demanding, accomplished only by a skilled team of personnel. They must use good techniques and tools in a planned and systematic way.
At the end of a development project, the team hopes that testing has been so successful that they don't have to invoke the last half of that famous saying ".... to forgive is divine!"
2.5 Entrance and Exit Criteria
The concept of establishing pre-requisites (entrance criteria) and post conditions (exit criteria) for an activity to be undertaken is extremely useful for managing any process, testing being no exception.
Entrance criteria are those factors that must be present, at a minimum, to be able to start an activity. In Integration Testing for example, before a module can be integrated into a program, it must be compiled cleanly and have successfully completed unit testing.
If the entrance criteria of the next phase have been met, the next phase may be started even though the current phase is still under way. This is how overlapping schedules are allowed.
Exit criteria are those factors that must be present to declare an activity completed. To proclaim System Testing completed, two criteria might be that all test cases must have been executed with a defined level of success (if other than 100%) and that there must be no more than a mutually agreed upon number of outstanding problems left unresolved. Exit criteria must be specifically expressed using terms such as, "X will be accepted if Y and a Z are completed."
You may view the User Acceptance Test as the Exit Criteria for the development project.
2.6 Testing Approaches
Testing approaches fall into two broad classes. They are static and dynamic testing.
Both are effective if applied properly and can be used throughout the application lifecycle. Testing tools and techniques will share characteristics from both these classes. The development and test teams are responsible for selection and use of the tools and techniques best suited for their project.
2.6.1 Static
Static testing is a detailed examination of a work product's characteristics to an expected set of attributes, experiences, and standards. Since the product under scrutiny is static and not exercised (as in a module being executed), its behavior to changing inputs and environments cannot be assessed.
Discovering variances or defects early in a project will result in less expensive and less painful changes. In the development cycle, the only early testing usually available during pre-construction phases is static testing.
Some representative examples of static testing are:
· Plan reviews
· Requirements walkthroughs or sign-off reviews
· Design or code inspections
· Test plan inspections
· Test case reviews
Here is an example of how a static test would be applied in a walkthrough of a single statement in a functional specification: "If withdrawal request is greater than $400 reject the request, otherwise allow subject to ATM cash dispensing rules and account balance."
Static test questions:
· Are all ATM dispensing rules the same (different types of machines)?
· Do the ATM dispensing rules allow for withdrawals up to $400?
· Is the account balance always available? If not, what then?
· What multiples of amounts are allowed?
· What types of accounts are allowed?
· If the host is not available do we use the card balance?
· If the host is not available but a deposit preceded the withdrawal request, should this be taken into account?
· If the request is denied, what is the message to the customer?
The point is that seemingly complete and precise requirements can, with practice, generate many questions. If these questions are answered as soon as the specification is written, testing costs will be much less than if the questions are not asked until after coding has commenced.
For a more thorough explanation of Inspections and Walkthroughs see the Static Testing Guidelines Technique Paper.
2.6.2 Dynamic Testing
Dynamic testing is a process of verification or validation by exercising (or operating) a work product under scrutiny and observing its behavior to changing inputs or environments. Where a module is statically tested by looking at its code documentation, it is executed dynamically to test the behavior of its logic and its response to inputs. Dynamic testing used to be the mainstay of system testing and was traditionally known as "testing the application."
Some representative examples of dynamic testing are:
· Application prototyping
· Executing test cases in an working system
· Simulating usage scenarios with real end-users to test usability
· Parallel testing in a production environment
2.7 Testing Techniques
Three common testing techniques are Black Box Testing, White Box Testing, and Error Guessing.
Testing techniques that examine the internal workings and details of a work product are said to use a white box approach. The technique that looks at how a unit under test behaves by only examining its inputs and outputs is said to be a black box approach. These are explained more thoroughly in Black Box/ White Box Testing in the Appendix.
The Error Guessing technique is where experience and intuition are applied to look for unexpected but prevalent errors.
2.7.1 Black Box Testing
In the Black Box approach, the testers have an "outside" view of the system. They are concerned with "what is done" NOT "how it is done."
Black Box testing is requirements and/or specifications-oriented and is used at all test levels.
The system or work product is defined and viewed functionally. To test the system, all possible input combinations are entered and the outputs are examined. Both valid and invalid input are used to test the system.
Examples of black box testing are:
· Enter an Automated Teller Machine withdrawal transaction and observe expected cash dispensed
· Exercising a program function with external data and observing the results
2.7.2 White Box Testing
In the White Box approach, the testers have an inside view of the system. They are concerned with "how it is done" NOT "what is done".
White Box testing is logic oriented. The testers are concerned with the execution of all possible paths of control flow through the program.
The "White Box" approach is essentially a Unit test method (which is sometimes used in the Integration test or in the Operability test) and is almost always performed by technical staff.
Examples of white box testing are:
· Testing of branches and decisions in code
· Tracking of a logic path in a programmer
2.7.3 Error Guessing
Based on past experience, test data can be created to anticipate those errors that will most often occur. Using experience and knowledge of the application, invalid data representing common mistakes a user might be expected to make can be entered to verify that the system will handle these types of errors.
An example of error guessing is:
· Hitting the 'CTRL' key instead of the 'ENTER' key for a PC application and verifying that the application responds properly
2.8 Full Lifecycle Testing

1 Application Development Phases
The software development process is a systematic method of evolving from a vague statement of a business problem to a detailed functional and concrete representation of a solution to that problem.
The development process itself is sub-divided into a discrete number of phases that align to a life cycle representing major stages or phases of system evolution. At the termination of each phase, a checkpoint is taken to insure that the current phase is completed properly and that the project team is prepared to start the next phase. The process we use to define and build a software product is the application development lifecycle. The testing process will align to these same phases. The development and test activities will be coordinated so that they complement each other.
As each interim work product is created by the developer, it is in turn tested to insure that it meets the requirements and specifications for that phase so that the resulting work products will be constructed on a sound foundation. This strategy of testing-as-you-go will minimize the risk of delivering a solution that is error prone or does not meet the needs of the end-user. We refer to testing at every phase in the development process as full lifecycle testing. It will test each and every interim work product delivered as part of the development process.
Testing begins with requirements and continues throughout the life of the application. During application development, all aspects of the system are exercised to ensure that the system is thoroughly tested before implementation. Static and dynamic, black box and white box, and error guessing testing approaches and techniques are used.
Even after an application has been in production, testing continues to play an important role in the systems maintenance process. While an application is being maintained, all changes to the application must be tested to ensure that it continues to provide the business function it was originally designed for, without impacting the rest of the application or the other systems with which it interfaces. When the hardware or software environment changes, the application must be retested in all areas to ensure that it continues to function as expected. This is often referred to as regression testing.
2.9 What Is Tested
Those on a system development or maintenance team usually test the following:
· The function of the application system manual and automated procedures in their operating environment to ensure it operates as specified
· Exception (error) conditions that could occur to ensure that they will be handled appropriately
· The interim deliverables of the development phases to demonstrate that they are satisfactory and in compliance with the original requirements.
2.10 The Testing Team
Developing a system requires the involvement of a team of individuals that bring varied resources, skills, experiences, and viewpoints to the process. Each plays a necessary role, each may play one or more roles, and the roles may change over the complete life cycle of development.
Added to the roles of sponsor, user, and developer defined in the application development process, is the tester. Their role is to validate and verify the interim and final work products of the system using both user and operational product and technical expertise where necessary
In many cases, the sponsor and the user are a single representative. In other cases, a special group of user representatives act as surrogate end-users. While they may represent the end-users, they themselves will usually never directly operate or use the application.
An organization where both the development team and user and/or sponsor personnel participate in the testing process is strongly recommended. It encourages user "buy-in" and acceptance through early involvement and participation throughout the development life cycle.
Some benefits are:
· Involvement of developers and users in walkthroughs and inspections of the deliverables during the earlier phases of the development life cycle serves as a training ground for the developers to gain business knowledge
· Testing will be more thorough and complete because users with greater product knowledge will be involved to test the business functions
· The User Acceptance Testing will provide a formal validation of a more completely tested system rather than serve as a process where basic product defects are being uncovered for the first time
· User personnel will become familiar with the forms and procedures to be used in the new system prior to implementation
· Users who gain experience in using the system during the test period will be able to train other user personnel and sell the new system
It is important to note the following.
The extent of user participation will depend on the specific nature of your project or application.
· The user should only get involved in "hands-on" dynamic testing at the point when the system is stable enough that meaningful testing can be carried out without repeated interruptions.
· Each user representative must be responsible for representing his or her specific organization and for providing the product knowledge and experience to effectively exercise the test conditions necessary to sufficiently test the application.
· The test team is responsible for making certain that the system produced by the development team adequately performs according to the system specifications. It is the responsibility of the test group to accept or reject the system based on the criteria that were documented and approved in the test plan.
Traditionally, users are viewed as being from the business community. They should also be considered as being from the technical (computer) operations area. Systems should incorporate the requirements of all the groups that will use and/or operate the services provided by the system being developed.
Finally, while the technical staff may build the technical components of the application, they will also assume the role of tester when they verify the compliance to design specifications of their own work or the work of others. This is an example of changing roles that was mentioned earlier.

3. The Testing Process
The basic testing process consists of four basic steps. They are:
· Plan for the tests
· Prepare for tests
· Execute the tests
· Report the results
Testing is an iterative process. These steps are followed at the overall project level and repeated for each level of testing required in the development lifecycle. This process is shown in the Test Process Overview diagram below.
2The development of the application will be undertaken in stages. Each stage represents a known level of physical integration and quality. These stages of integration are known as Testing Levels and will be discussed later.
The Levels of Testing used in the application development lifecycle are:
· Unit testing
· Integration testing
· System testing
· Systems integration testing
· User acceptance testing
· Operability testing
The first layer of plans and tests address the overall project at a high level and is called the Master Test Plan. Successive Levels of Testing will each be handled as additional refinements to the Master Test Plan to meet the specific needs of that Level of Testing. The FLT methodology refers to these as Detailed Test Plans. They are explained in more detail in the following sections.
For a more detailed treatment of the process, refer to the FLT models in the Appendix. The overview of the process is depicted as a level 0 data flow model. Each of the major processes on the level 0 data flow diagram has its own data flow diagram shown separately on the following pages.
3.1 Test Planning
Planning itself is a process. Performing each step of the planning process will insure that the plan is built systematically and completely. Documenting these steps will ensure that the plan is itself testable by others who have to approve it. Most of the topics under planning should be familiar to all but the novice developer or tester. The Test Process Model will lend some guidance in this area.
A few topics deserve special attention since they are specific to testing or helpful in understanding testing in general.
3.1.1 The Administrative Plan
This portion of the plan deals with test team organization, test schedules, and resource requirements. These plans should be fully integrated into the overall project schedules.
We will more fully cover the specifics of test team planning later in the section on Managing and Controlling the Test Effort.
3.1.2 Risk Assessment
Why is Risk Assessment a part of planning? The reason for risk analysis is to gain an understanding of the potential sources and causes of failure and their associated costs. Measuring the risk prior to testing can help the process in two ways:
· High-risk applications can be identified and more extensive testing can be performed.
· Risk analysis can help draw attention to the critical components and/or focus areas for testing that are most important from the system or user standpoint.
In general, the greater the potential cost of an error in a system, the greater the effort and the resources assigned to prevent these errors from occurring.
Risk is the product of the probability of occurrence of an error and the cost and consequence of failure.
Three key areas of risk that have significant influence on a project are:
· Project size
· Experience with technology
· Project structure
The potential cost/risk factor, independent of any particular error, must be taken into consideration when developing an overall testing strategy and a test plan for a particular system.
One approach to assessing risk is:
· List what can go wrong during the operation of the application once implemented, and the likelihood of occurrence.
· Determine the cost of failure. For example, loss of business, loss of image, loss of confidence in the system, security risk, and financial risk, for each of these problems to the organization if it occurred.
· Determine what level of confidence is required and what the critical success factors are from an application development and testing point of view.
A strategy and subsequent plans to manage these risks will form the basis of the testing strategy.
· How much effort and expense should be put into testing
· When the potential loss exceeds the cost of testing, the testing is cost justified
3.1.3 Test Focus
The overall objective of testing is to reduce the risks inherent in computer systems. The methodology must address and minimize those risks. Areas are identified to assist in drawing attention to the specific risks or issues where testing effort is focused and which must be handled by the test strategy.
We have so far discussed the importance of quality of systems being developed and implemented. It is virtually impossible and economically not feasible to perform exhaustive testing. We have to avoid both under testing and over testing, while optimizing the testing resources, and reducing cycle time. All of these lead to the question of management of conflicts and trade-offs. When projects do not have unlimited resources, we have to make the best use of what we have. This comes down to one question: What are the factors or risks that are most important to:
· The user or the client
· The developer from a system perspective
· The service provider (usually Computer Services)
The answer lies in considering the Focus areas and selecting the ones that will address or minimize risks.
Test Focus can be defined as those attributes of a system that must be tested in order to assure that the business and technical requirements can be met. Some of the commonly used test focus areas are:
· Auditability
· Continuity of Processing
· Correctness
· Maintainability
· Operability
· Performance
· Portability
· Reliability
· Security
· Usability

You may consider and include other focus areas as necessary.
For example:
· Technology may be a Test Focus in a cooperative processing environment
· Compatibility of software may be a test focus item for vendor packages
Each of the items on the list of common test focus areas is a potential factor that may impact the proper functioning of the system. For example, the risk of the system not being usable may result in certain functions not being used. The challenge to effective testing is to identify and prioritize those risks that are important to minimize and focus testing in those areas.
It is therefore critical that a joint decision be taken by the users and developers as to what is important and what is not. It should not be decided from a single perspective.
It is important to note that, to a major extent, correctness is most likely a high priority test focus for most applications. Further considerations are:
· What degree of correctness is required?
· What are the risks if the functions are not tested exhaustively?
· What price are the users willing to pay?
This means that the:
· Process of determining test focus areas has to be selective, and
· Selected focus areas have to be ordered in terms of priority.
The concept of risk and risk evaluation makes the decision of how much testing to do, or what types of testing are needed to be performed, an economic decision. The economics of testing and risk management determines whether defects are acceptable in the system, and if so, what is the tolerable level of residual defects. This means that the decisions of what gets tested more or less thoroughly are shifted from the judgment of the developers and users to ones based more objectively on economics.
3.1.4 Test Objectives
From a testing standpoint, the test objectives are a statement of the goals that must be achieved to assure that the system objectives are met and that the system will be demonstrated to be acceptable to the user.
They should answer the question:
"What must be achieved
· in order to assure that the system is at the level of quality that the acceptor of the system will expect, and
· that he/she will also be confident that the developer has achieved these expectations?"
They should draw the attention or focus on the major testing tasks that must be accomplished to successfully complete full testing of the system. The purpose of developing test objectives is to define evaluation criteria to know when testing is completed. Test objectives also assist in gauging the effectiveness of the testing process.
Test objectives are based on:
· Business functions
· Technical requirements
· Risks and
and are driven by test focus areas. Test objectives are defined with varying levels of detail at various levels of testing; there will be test objectives at the global cross-functional level as well as the major functional level.
For example, if correctness of bank statement processing is to be a focus area, then one testing objective will concentrate on how correctness will be thoroughly proved in bank statement processing by the functional testing results.
3.1.5 Test Strategy
Test strategy is a high level system-wide expression of major activities that collectively achieve the overall desired result as expressed by the testing objectives.
For example, if performance (transaction response time) is a focus area, then a strategy might be to perform Stress/Volume Testing at the Integration Test Level before the entire system is subjected to System Testing. Similarly, if auditability is a focus area, then a strategy might be to perform rigorous audit tests at all levels of testing.
As part of forming the strategy, the risks, constraints, and exposures present must be identified. These considerations must be accommodated in the strategy. This is where the planner can introduce significant added value to the plan.
As an example of one such strategy that handles constraints, if you have to develop the application to run on two technical platforms but only one is available in the development environment. The strategy might be to exhaustively test on one platform and test the other platform in the production operating environment as a parallel run. This strategy recognizes the constraint on the availability of test platforms and manages it with acceptable risk.
Another practical area of constraint may be where the organization's Testing Standards play a significant part in reducing the freedom you can adopt in your strategy.
All strategy statements are expressed in high level terms of physical components and activities, resources (people and machines), types and levels of testing, schedules and activities ... the stuff of which action plans are made at lower levels. The strategic plan will be specific to the system being developed and will be capable of being further refined into tactical approaches and operating plans in the Detailed Test Plans for each level of testing.
Strategic plans will drive future needs in resources or schedules in testing (or development) and will often force trade-offs. Fortunately, since this is usually well in advance of the anticipated need, the project will have enough time to respond.
Detailed Test Plans should repeat the exercise of reviewing and setting objectives and strategy specific to each level of testing. These objectives and associated strategies (often called approaches at the lower levels) should dovetail with their predecessor higher level Master Test Plan (MTP) set of objectives and strategies.

3.1.6 The Build Strategy
A set of testing plans will include the high level list of items for which testing is needed to be performed. These are the lists of Business Functions and Structural Functions that will be covered by testing activities. You may consider that these are testing blueprints and use them in the way that architects use blueprints as plans to guide them in their design and construction processes. These "architect's plans" are not like resource and schedule plans - they serve very different purposes. They are more like a functional road map for building test cases. Both these types of plans will exist on a development project, and both will share dependencies.
The Build Strategy is the approach to how the physical system components will be built, assembled, and tested. It is driven by overall project requirements such as user priorities, development schedules, earlier delivery of specific functions, resource constraints, or any factor where there is a need to consider building the system in small pieces. Each build can be assembled and tested as a stand-alone unit although there may be dependencies on previous builds.
Since it is desirable to test a component or assembly of components as soon as possible after they are built, there must be close co-ordination between the development team (builders) and the test team (testers). There must be a close collaboration on what components are built first, what testing facilities must be present, how the components will be assembled, and how they will be tested and released to the next level of integration.

3.1.7 Problem Management & Change Control
Problem Management is the process by which problems (variance reports) are controlled and tracked within a development project. This is a key tool to the testers. It is best to use the same Problem Management System used by the overall project. If one does not exist, it should be set up early enough to be in place for the discovery of the first variances. It is recommended that an automated system be used to capture, store, and report on the status of variances. The amount of detail becomes enormous even on small projects.
Change Control or Change Management is similar to Problem Management in that it captures, stores, and reports on the status of valid and recognized change requirements. Testing Teams will both request change and also be impacted by change. Again it is recommended that an automated system be used if the amount of change will be large. Once again, the project's Change Management System should be the one used by all.
Plans should include provision for, and use of, both Problem Management and Change Management Systems. If these systems do not exist, plans should include provisions to create and maintain them.
3.2 Test Case Design
Test Case Design must focus on the testing techniques, tools, the build and integration strategies, and the basic characteristics of the system to be tested.
The first approaches to consider are Black Box and White Box. Black Box techniques help to validate that the business requirements are met while White Box techniques facilitate testing the technical requirements. The former tests for requirements' coverage whereas the latter provides logic coverage.
The tools that provide the maximum pay back are keystroke capture and play back and test coverage tools. The underlying principles for both these tools are very simple. Keystroke capture tools eliminate the tedious and repetitive effort involved in keying in data. Test coverage tools assist in ensuring that the myriad paths that the logic branches and decisions can take are tested. Both are very useful but need a lot of advance planning and organization of the test cases. As well as anticipating all expected results, testing tool procedures must provide for handling the unexpected situations. Therefore, they need a comprehensive test case design plan.
Care must be exercised in using these automated tools so as to get the most effective usage. If the use of these tools is not carefully planned, a testing team can become frustrated and discouraged with the tool itself. The testers will abandon these tools or fail to maintain the test cases already in their test packages.
The test design must also consider whether the tests involve on-line or batch systems, and whether they are input or output driven. Test cases would be built to accommodate these system characteristics.
Another consideration to take into account will be the grouping of valid and invalid test conditions. The integration approach (top down, bottom up, or a combination of both) can sometimes influence the test case design. For a more complete discussion of integration approaches please see Integration Approaches in the Appendix B58.
It is recommended that the development of regression test packages be considered part of the test design. Test cases should always be built for reuse. Test case design strategy entails the following steps:
· Examine the development / testing approach. This was explained in the Build Strategy section..
· Consider the type of processing - whether on-line, batch, conversion program, etc.
· Determine the techniques to be used - white box / black box, or error guessing.
· Develop the test conditions.
· Develop the test cases.
· Create the test script.
· Define the expected results.
· Develop the procedures and data (prerequisites, steps, expectations, and post-test steps).
In developing a list of conditions that should be tested, it is useful to have a list of standard conditions to test for some common application types. A comprehensive list of considerations for ensuring a complete list of conditions to test are included the Test Case Design in the Appendix D. They can be used as an aid to develop test cases.
4. Levels of Testing
Testing proceeds through various physical levels as described in the application development lifecycle. Each completed level represents a milestone on the project plan and each stage represents a known level of physical integration and quality. These stages of integration are known as Testing Levels. The Levels of Testing used in the application development lifecycle are:
· Unit testing
· Integration testing
· System testing
· Systems integration testing
· User acceptance testing
· Operability testing
In FLT Methodology, requirements testing and design testing are used as additional levels of testing.
4.1 Testing Model
The descriptions of testing levels will follow. For each of these levels, these attributes will be covered:
· Objectives
· When to perform the tests
· Inputs / outputs
· Who performs the tests
· Methods
· Tools
· Education / training pre-requisites
The tables on the following pages describe the testing levels for each level of testing:
4.2 Requirements Testing
Requirements testing involves the verification and validation of requirements through static and dynamic tests. The validation testing of requirements will be covered under User Acceptance Testing. This section only covers the verification testing of requirements.

Objectives
· To verify that the stated requirements meet the business needs of the end-user before the external design is started
· To evaluate the requirements for testability
When
· After requirements have been stated
Input
· Detailed Requirements
Output
· Verified Requirements
Who
· Users & Developers
Methods
· JAD
· Static Testing Techniques
· Checklists
· Mapping
· Document Reviews
Tools
· CASE
Education
· Application training
4.3 Design Testing
Design testing involves the verification and validation of the system design through static and dynamic tests. The validation testing of external design is done during User Acceptance Testing and the validation testing of internal design is covered during, Unit, Integration and System Testing. This section only covers the verification testing of external and internal design.

Objectives
· To verify that the system design meets the agreed to business and technical requirements before the system construction begins
· To identify missed requirements
When
· After External Design is completed
· After Internal Design is completed
Input
· External Application Design
· Internal Application Design
Output
· Verified External Design
· Verified Internal Design
Who
· Users & Developers
Methods
· JAD
· Static Testing Techniques
· Checklists
· Prototyping
· Mapping
· Document Reviews
Tools
· CASE
· Prototyping tools
Education
· Application training
· Technical training

4.4 Unit Testing
Unit level test is the initial testing of new and changed code in a module. It verifies the program specifications to the internal logic of the program or module and validates the logic.

Objectives
· To test the function of a program or unit of code such as a program or module
· To test internal logic
· To verify internal design
· To test path & conditions coverage
· To test exception conditions & error handling

· After modules are coded
Input
· Internal Application Design
· Master Test Plan
· Unit Test Plan
Output
· Unit Test Report
Who
· Developer
Methods
· White Box testing techniques
· Test Coverage techniques
Tools
· Debug
· Re-structure
· Code Analyzers
· Path/statement coverage tools
Education
· Testing Methodology
· Effective use of tools
Programs and modules should be desk checked or walked-through before they are integrated and tested. This review must be done first by the programmer and then in a more formal way through a structured walkthrough or code inspection.
The steps to prepare for unit testing are:
· Determine the development integration and testing approach, top-down, bottom-up or a combination of both.
· Determine the testing techniques to be used (white box), and the particular sub-techniques that apply best to the level of testing such as statement coverage, decision coverage, path coverage, equivalence partitioning, boundary value analysis and so on. These techniques will be described later in this document.
· Develop the test sets of conditions (one or more levels depending on detail).
Some of the common things to check for are that:
· All variables are explicitly declared and initialized
· The initialization is correct after every processing level and that work areas are cleared or reset properly
· Array subscripts are integers and within the bounds of the array
· Reference variables are correctly allocated
· Unexpected error conditions are handled correctly
· File attributes are correct
· End-of-file conditions are handled correctly
During program development, design the test cases at the same time as the code is designed. Advantages are that test conditions / cases are:
· Designed more objectively
· Not influenced by coding style
· Not overlooked
4.5 Integration Testing
Integration level tests verify proper execution of application components and do not require that the application under test interface with other applications. Communication between modules within the sub-system is tested in a controlled and isolated environment within the project.

Objectives
· To technically verify proper interfacing between modules, and within sub-systems
When
· After modules are unit tested
Input
· Internal & External Application Design
· Master Test Plan
· Integration Test Plan
Output
· Integration Test report
Who
· Developers
Methods
· White and Black Box techniques
· Problem / Configuration Management
Tools
· Debug
· Re-structure
· Code Analyzers
Education
· Testing Methodology
· Effective use of tools
4.6 System Testing
System level tests verify proper execution of the entire application components including interfaces to other applications. Both functional and structural types of tests are performed to verify that the system is functionally and operationally sound.

Objectives
· To verify that the system components perform control functions
· To perform inter-system test
· To demonstrate that the system performs both functionally and operationally as specified
· To perform appropriate types of tests relating to Transaction Flow, Installation, Reliability, Regression etc.
When
· After Integration Testing
Input
· Detailed Requirements & External Application Design
· Master Test Plan
· System Test Plan
Output
· System Test Report
Who
· Development Team and Users
Methods
· Problem / Configuration Management
Tools
· Recommended set of tools
Education
· Testing Methodology
· Effective use of tools

4.7 Systems Integration Testing
Systems Integration testing is a test level which verifies the integration of all applications, including interfaces internal and external to the organization, with their hardware, software and infrastructure components in a production-like environment.

Objectives
· To test the co-existence of products and applications that are required to perform together in the production-like operational environment (hardware, software, network)
· To ensure that the system functions together with all the components of its environment as a total system
· To ensure that the system releases can be deployed in the current environment
When
· After system testing
· Often performed outside of project life-cycle
Input
· Test Strategy
· Master Test Plan
· Systems Integration Test Plan
Output
· Systems Integration Test report
Who
· System Testers
Methods
· White and Black Box techniques
· Problem / Configuration Management
Tools
· Recommended set of tools
Education
· Testing Methodology
· Effective use of tools

4.8 User Acceptance Testing
User acceptance tests (UAT) verify that the system meets user requirements as specified. The UAT simulates the user environment and emphasizes security, documentation and regression tests and will demonstrate that the system performs as expected to the sponsor and end-user so that they may accept the system.

Objectives
· To verify that the system meets the user requirements
When
· After System Testing
Input
· Business Needs & Detailed Requirements
· Master Test Plan
· User Acceptance Test Plan
Output
· User Acceptance Test report
Who
· Users / End Users
Methods
· Black Box techniques
· Problem / Configuration Management
Tools
· Compare, keystroke capture & playback, regression testing
Education
· Testing Methodology
· Effective use of tools
· Product knowledge
· Business Release Strategy

4.9 Operability Testing
Operability tests verify that the application can operate in the production environment. Operability tests are performed after, or concurrent with, User Acceptance Tests.

Objectives
· To ensure the product can operate in the production environment
· To ensure the product meets the acceptable level of service as per Service Level Agreement
· To ensure the product operates as stated in the Operations Standards
· To ensure the system can be recovered / re-started as per standards
· To ensure that JCL is as per standard
When
· Concurrent with or after User Acceptance Testing is completed
Input
· User Acceptance Test Plan
· User Sign-off from UAT (if available)
· Operability Test Plan
· Operations Standards (as appropriate)
Output
· Operability test report
Who
· Operations staff
Methods
· Problem / change management
Tools
· Performance monitoring tools
Education

5. Types of Tests
Testing types are logical tests, which may be conducted in isolation or as combined exercises. They will be performed during the physical levels of testing as previously described. Success of the testing process depends on:
· Selecting appropriate types of testing necessary to meet the test objectives
· Determining the stages or levels of testing when these types of testing would be most effectively used (A test type can appear in more than one test level)
· Developing test conditions to meet the test evaluation criteria
· Creating test scripts / test data required to test the conditions above
· Managing fixes and re-testing
Types of testing are broadly classified as:
· Functional Testing
· Structural Testing
5.1 Functional Testing
The purpose of functional testing is to ensure that the user functional requirements and specifications are met. Test conditions are generated to evaluate the correctness of the application. The following are some of the categories. They are organized alphabetically:
· Audit and Controls testing
· Conversion testing
· Documentation & Procedures testing
· Error Handling testing
· Functions / Requirements testing
· Interface / Inter-system testing
· Installation testing
· Parallel testing
· Regression testing
· Transaction Flow (Path) testing
· Usability testing
They are each discussed in the following sections.
5.2 Audit and Controls Testing
Description
Audit and Controls testing verifies the adequacy and effectiveness of controls and ensures the capability to prove the completeness of data processing results. Their validity would have been verified during design.
Audit & Controls testing would normally be carried out as part of System Testing once the primary application functions have been stabilized.
Objectives
The objectives for Audit and Controls testing are to ensure or verify that:
· Audit trail data is accurate and complete
· The transactions are authorized
· The audit trail information is produced and maintained as needed
Examples
· Using hash totals to ensure that the accumulation of detailed records reconciles to the total records
· Inspecting manual control availability and operation to ensure audit effectiveness
· Reviewing the audit trail from the full parallel run to ensure correctness
5.2.1 Conversion Testing
Conversion testing verifies the compatibility of the converted program, data, and procedures with those from existing systems that are being converted or replaced. Most programs that are developed for conversion purposes are not totally new. They are often enhancements or replacements for old, deficient, or manual systems. The conversion may involve files, databases, screens, report formats, etc.
Portions of conversion testing could start prior to Unit Testing and, in fact, some of the conversion programs may even be used as drivers for unit testing or to create unit test data.
Objectives
The objectives for Conversion testing are to ensure or verify that:
· New programs are compatible with old programs
· The conversion procedures for documentation, operation, user interfaces, etc., work
· Converted data files and format are compatible with the new system
· The new programs are compatible with the new databases
· The new functions meet requirements
· Unchanged functions continue to perform as before
· Structural functions perform as specified
· Back out/recovery and/or parallel operation procedures work
Examples
· Conversion from one operating system to another
· Conversion from host to distributed systems
· Conversion from IMS databases to DB2 tables
5.2.2 User Documentation and Procedures Testing
Description
User documentation and procedures testing ensures that the interface between the system and the people works and is useable. Documentation testing is often done as part of procedure testing to verify that the instruction guides are helpful and accurate. Both areas of testing are normally carried out late in the cycle as part of System Testing or in the UAT. It is normally a mistake to invest a lot of effort to develop and test the user documentation and procedures until the externals of the system have stabilized. Ideally, the persons who will use the documentation and procedures are the ones who should conduct these tests.
Objectives
The objectives are to ensure or verify that:
· User / operational procedures are documented, complete and correct, and are easy to use
· People responsibilities are properly assigned, understood and coordinated
· User / operations staff are adequately trained
Examples
· Providing Operations staff with manuals and having them use them to run the test application to observe their actions in a real operations simulation
· Providing data entry personnel with the kind of information they normally receive from customers and verifying that the information is entered correctly as per manuals / procedures
· Simulating real end-user scenarios (like telemarketing) using the system with the documentation and procedures developed
5.2.3 Error-Handling Testing
Description
Error-handling is the system function for detecting and responding to exception conditions (such as erroneous input). The completeness of the error handling capability of an application system is often key to the usability of the system. It ensures that incorrect transactions will be properly processed and that the system will terminate in a controlled and predictable way in case of a disastrous failure.
Note: Error-handling tests should be included in all levels of testing.
Objectives
The objectives are to ensure or verify that:
· All reasonably expected errors can be detected by the system
· The system can adequately handle the error conditions and ensure continuity of processing
· Proper controls are in place during the correction process
Error handling logic must deal with all possible conditions related to the arrival of erroneous input such as:
· Initial detection and reporting of the error
· Storage of the erroneous input pending resolution
· Periodic flagging of the outstanding error until it is resolved
· Processing of the correction to the error
Examples
· Seeding some common transactions with known errors into the system
· Simulating an environmental failure to see how the application recovers from a major error
5.2.4 Function Testing
Description
Function Testing verifies, at each stage of development, that each business function operates as stated in the Requirements and as specified in the External and Internal Design documents. Function testing is usually completed in System Testing so that by the time the system is handed over to the user for UAT, the test group has already verified to the best of their ability that the system meets requirements.
Objectives
The objectives are to ensure or verify that:
· The system meets the user requirements
· The system performs its functions consistently and accurately
· The application processes information in accordance with the organization's standards, policies, and procedures
Examples
· Manually mapping each external design element back to a requirement to ensure they are all included in the design (static test)
· Simulating a variety of usage scenarios with test cases in an operational test system
5.2.5 Installation Testing
Description
Any application that will be installed and run in an environment remote from the development location requires installation testing. This is especially true of network systems that may be run in many locations. This is also the case with packages where changes were developed at the vendor's site. Installation testing is necessary if the installation is complex, critical, should be completed in a short window, or of high volume such as in microcomputer installations. This type of testing should always be performed by those who will perform the installation process.
Installation testing is done after exiting from System Test, or in parallel with the User Acceptance Test.
Objectives
The objectives are to ensure or verify that:
· All required components are in the installation package
· The installation procedure is user-friendly and easy to use
· The installation documentation is complete and accurate
· The machine readable data is in the correct and usable format
Examples
· Verifying the contents of installation package with its checklist of enclosures
· Actually having a person outside of the development or test team install the application using just the installation documentation as guidance
5.2.6 Interface / Inter-system Testing
Description
Application systems often interface with other application systems. Most often, there are multiple applications involved in a single project implementation. Interface or inter-system testing ensures that the interconnections between applications function correctly. The interface testing is even more complex if the applications operate on different platforms, in different locations or use different languages.
Interface testing is typically carried out during System Testing when all the components are available and working. It is also acceptable for a certain amount of Interface testing to be performed during UAT to verify that System-tested interfaces function properly in a production-like environment. Interface testing in UAT should always be the first practical tests in order to resolve incompatibility issues before the users commence their tests.
Objectives
The objectives are to ensure or verify that:
· Proper parameters and data are correctly passed between the applications
· The applications agree on the format and sequence of data being passed
· Proper timing and coordination of functions exists between the application systems and that the processing schedules reflect these
· Interface documentation for the various systems is complete and accurate
· Missing data files are properly handled
· It is not possible for the same file to be processed twice or to be processed out of sequence
· The implications are clearly identified if the interfacing applications are delayed, not available or have been cancelled.
Examples
· Taking a set of test transactions from one application to be passed to another application and passing them into the interfacing application
· Simulating the loss of an upstream interface file in a full System Test
5.2.7 Parallel Testing
Description
Parallel testing compares the results of processing the same data in both the old and new systems.
Parallel testing is useful when a new application replaces an existing system, when the same transaction input is used in both, and when the output from both is reconcilable. It is also useful when switching from a manual system to an automated system.
Parallel testing is performed by using the same data to run both the new and old systems. The outputs are compared, and when all the variances are explained and acceptable, the old system is discontinued.
Objectives
The objectives are to ensure or verify that:
· The new system gives results consistent with the old system (in those situations where the old system was satisfactory)
· Expected differences in results occur (in those circumstances where the old system was unsatisfactory)
Examples
· First running the old system, restoring the environment to as it was at the beginning of the first run, and then running the new version in a production environment
· Executing two manual procedures side-by-side and observing the parallel results
5.2.8 Regression Testing
Description
Regression testing verifies that no unwanted changes were introduced to one part of the system as a result of making changes to another part of the system.
To perform a regression test, the application must be run through the same test scenario at least twice. The first test is run when your application or a specific part of your application is responding correctly. Your application's responses to the first test serve as a base against which you can compare later responses. The second test is run after you make changes to the application. The application responses are compared for both executions of the same test cases.
The results of comparison can be used to document and analyze the differences. By analyzing the differences between two executions of the same test cases, you can determine if your application's responses have changed unexpectedly.
Regression testing should always be used by the developer during Unit testing and, in conjunction with a Change Management discipline, will help prevent code changes being lost or being overwritten by subsequent changes. A last regression test should be done as the final act of System Testing, once all the function is stabilized and further changes are not expected.
Objectives
The objectives are to ensure or verify that:
· The test unit or system continues to function correctly after changes have been made
· Later changes do not affect previous system functions

Examples
· Final test a group of changes by executing a regression test run as the last System Test
· Reusing accumulated test cases to re-unit-test a module each time an incremental change is coded
5.2.9 Transaction Flow Testing
Description
Transaction Flow Testing can be defined as the testing of the path of a transaction from the time it enters the system until it is completely processed and exits a suite of applications. For example, if it is an on-line branch transaction for issue of a draft on another bank, path testing starts when the branch teller enters the transaction at the terminal; through account debiting, draft issuance, funds transfer to the paying bank, draft settlement, account reconciliation and reporting of the transaction on the customer's statement. This implies that transaction flow is not necessarily limited to testing one application especially if the end-to-end process is handled by more than one system application. Wherever any one component in a flow of processes changes, it is the responsibility of the person or group making that change to ensure that all other processes continue to function properly.
Transaction Flow testing may begin once System Testing has progressed to the point that the application is demonstrably stable and capable of processing transactions from start to finish.
Objectives
The objectives are to ensure or verify that:
· The transaction is correctly processed from the time of its entry into the system until the time it is expected to exit
· All the output from the various systems or sub-systems, which are input to the other systems or sub-systems, are processed correctly and passed through
· Interfaces can handle unexpected situations without causing any uncontrolled ABENDs or any break in services
· The business system functions "seamlessly" across a set of application systems
Examples
· Entering an on-line transaction and tracing its progress completely through the applications
· Creating unexpected problem situations in interface systems and verifying that the system is equipped to handle them
5.2.10 Usability Testing
Description
The purpose of usability testing is to ensure that the final product is usable in a practical, day-to-day fashion. Whereas functional testing looks for accuracy of the product, this type of test looks for simplicity and user-friendliness of the product.
Usability testing would normally be performed as part of functional testing during System and User Acceptance Testing.

Objectives
The objectives of this type of test are to ensure that:
· The system is easy to operate from an end-user and a service provider (Computer Services) standpoint
· Screens and output are clear, concise and easy to use and help screens or clerical instructions are readable, accurately describe the process and are expressed in simple jargon-free language
· Input processes, whether via terminal or paper, follow natural and intuitive steps.
Examples
· Testing a data input screen to ensure that data is requested in the order that the client would normally use it.
· Asking someone unconnected with the project to complete a balancing process on a printed report by following the instructions.
5.3 Structural Testing
The purpose of structural testing is to ensure that the technical and "housekeeping" functions of the system work. It is designed to verify that the system is structurally sound and can perform the intended tasks.
Its objective is also to ensure that the technology has been used properly and that when the component parts are integrated they perform as a cohesive unit. The tests are not intended to verify the functional correctness of the system, but rather that the system is technically sound.
The categories covered in the next sub-sections are:
· Backup and Recovery testing
· Contingency testing
· Job Stream testing
· Operational testing
· Performance testing
· Security testing
· Stress / Volume testing
5.3.1 Backup and Recovery Testing
Description
Recovery is the ability of an application to be restarted after failure. The process usually involves backing up to a point in the processing cycle where the integrity of the system is assured and then re-processing the transactions past the original point of failure. The nature of the application, the volume of transactions, the internal design of the application to handle a restart process, the skill level of the people involved in the recovery procedures, documentation and tools provided, all impact the recovery process.
Backup and recovery testing should be performed as part of the System Testing and verified during Operability Testing whenever continuity of processing is a critical requirement for the application. Risk of failure and potential loss due to the inability of an application to recover will dictate the extent of testing required.
Objectives
The objectives are to ensure or verify that:
· The system can continue to operate after a failure
· All necessary data to restore / re-start the system is saved
· Backed up data is accessible and can be restored
· Backup and recovery procedures are well documented and available
· People responsible for conducting the recovery are adequately trained
Examples
· Simulating a full production environment with production data volumes
· Simulating a system failure and verifying that procedures are adequate to handle the recovery process
5.3.2 Contingency Testing
Description
Operational situations may occur which result in major outages or "disasters". Some applications are so crucial that special precautions need to be taken to minimize the effects of these situations and speed the recovery process. This is called Contingency.
Usually each application is rated by its users in terms of its importance to the company and contingency plans are drawn up accordingly. In some cases, where an application is of no major significance, a contingency plan may be to simply wait for the disaster to go away. In other cases more extreme measures have to be taken. For example, a back-up processor in a different site may be an essential element required by the contingency plan. Contingency testing will have to verify that an application and its databases, networks, and operating processes can all be migrated smoothly to the other site.
Contingency testing is a specialized exercise normally conducted by operations staff. There is no mandated phase in which this type of test is to be performed although, in the case of highly important applications, this will occur after System Testing and probably concurrent with the UAT and Operability Testing.
Objectives
· The system can be restored according to prescribed requirements
· All data is recoverable and restartable under the prescribed contingency conditions
· All processes, instructions and procedures function correctly in the contingency conditions
· When normal conditions return, the system and all its processes and data can be restored
Examples:
· Simulate a collapse of the application with no controlled back-ups and test its recovery and the impact in terms of data and service loss
· Test the operational instructions to ensure that they are not site specific.
5.3.3 Job Stream Testing
Description
Job Stream testing is usually done as a part of operational testing (the test type not the test level, although this is still performed during Operability Testing). Job Stream testing starts early and continues throughout all levels of testing. Conformance to standards is checked in User Acceptance and Operability testing.
Objectives
The objectives are to ensure or verify that:
· The JCL is defect free, compliant to standards, and will execute correctly
· Each program can handle expected parameter input
· The programs are generating the required return codes
· Jobs are sequenced and released properly
· File formats are consistent across programs to ensure that the programs can "talk to" each other
· The system is compatible with the operational environment
Examples
· Having the JCL inspected by an Operations staff member or checked by a tool like JCL Check
· Running a full operating test with a final version of the operations JCL
· Attempting to run the application with an invalid set of parameters
5.3.4 Operational Testing
Description
All products delivered into production must obviously perform according to user requirements. However, a product's performance is not limited solely to its functional characteristics. Its operational characteristics are just as important since users expect and demand a guaranteed level of service from Computer Services. Therefore, even though Operability Testing is the final point where a system's operational behavior is tested, it is still the responsibility of the developers to consider and test operational factors during the construction phase.
Operational Testing should be performed as part of Integration and System Testing and verified during Operability Testing.
Objectives
The objectives are to ensure that:
· All modules and programs are available and operational
· All on-line commands and procedures function properly
· All JCL conforms to standards and operates the application successfully
· All scheduling information is correct
· All operational documentation correctly represents the application, is clear, concise and complete
· All batch functions can be completed within an acceptable window of operation
Examples
· Simulate a full batch cycle and operate the process exactly as prescribed in the run documentation
· Subject all JCL to a standard check
5.3.5 Performance Testing
Description
Performance Testing is designed to test whether the system meets the desired level of performance in a production environment. Performance considerations may relate to response times, turn around times (through-put), technical design issues and so on. Performance testing can be conducted using a production system, a simulated environment, or a prototype.
Attention to performance issues (e.g. response time or availability) begins during the Design Phase. At that time the performance criteria should be established. Performance models may be constructed at that time if warranted by the nature of the project.
Actual performance measurement should begin as soon as working programs (not necessarily defect-free programs) are ready.
Objectives
The objectives are to ensure or verify that:
· The system performs as requested (e.g., transaction response, availability, etc.).
Examples
· Using performance monitoring tools to verify system performance specifications
· Logging transaction response times in a System Test
· Using performance monitoring software to ensure identification of dead code in programs, efficient processing of transactions etc. - normal and peak periods should be identified and testing carried out to cover both
5.3.6 Security Testing
Description
Security of an application system is required to ensure the protection of confidential information in a system and in other affected systems is protected against loss, corruption, or misuse; either by deliberate or accidental actions. The amount of testing needed depends on the risk assessment of the consequences of a breach in security. Tests should focus on, and be limited to those security features developed as part of the system, but may include security functions previously implemented but necessary to fully test the system.
Security testing can begin at any time during System Testing, continue in UAT and are completed in Operability Testing.
Objectives
The objectives are to ensure or verify that:
· The security features cannot be bypassed, altered, or broken
· Security risks are properly identified and accepted, and contingency plans tested
· The security provided by the system functions correctly
Examples
· Attempting a 'sign-on' to the system without authorization if one is required
· Verifying passwords are not visible on terminals or on printed output
· Attempting to give yourself authorization to perform restricted tasks
· Attempting to enter unauthorized on-line transactions to ensure that the system can not only identify and prevent such unauthorized access but also that they are reported if required
5.3.7 Stress / Volume Testing
Description
Stress testing is defined as the processing of a large number of transactions through the system in a defined period of time. It is done to measure the performance characteristics of the system under peak load conditions.
Stress factors may apply to different aspects of the system such as input transactions, report lines, internal tables, communications, computer processing capacity, throughput, disk space, I/O and so on.
Stress testing should not begin until the system functions are fully tested and stable. The need for Stress Testing must be identified in the Design Phase and should commence as soon as operationally stable system units are available.
It is not necessary to have all functions fully tested in order to start Stress Testing. The reason why this is started early is so that any design defects can be rectified before the system exits the construction phase.
Objectives
The objectives are to ensure or verify that:
· The production system can process large volumes of transactions within the expected time frame
· The system architecture and construction is capable of processing large volumes of data
· The hardware / software are adequate to process the volumes
· The system has adequate resources to handle expected turn around time
· The report processing support functions (e.g., print services) can handle the volume of data output by the system
Examples
· Testing system overflow conditions by entering more volume than can be handled by tables, transaction queues, internal storage facilities and so on
· Testing communication lines during peak processing simulations
· Using test data generators and multiple terminals, stressing the on-line systems for an extended period of time and stressing the batch system with more than one batch of transactions
5.4 Relationship Between Levels and Types of Testing
The diagram below illustrates the relationship between testing levels and testing types. The testing levels are the columns and the types are the rows in the table. The table shows the level(s) where each type of test might be performed. These are only suggestions. Obviously each project has different characteristics, which have to be considered when planning the testing process.
As an example, attention is drawn to Interface/Inter-System testing. This is clearly a Systems Test function although, given environmental constraints within the Development facility, some Interface testing may have to be performed as an initial process within User Acceptance Testing:

Levels
Unit
Integration
System
Systems Integration
UAT
Operability
Types

6. Test Data Set Up
Test data set up is a very tedious and time-consuming process. After the test case design has been selected, we have to consider all the available sources of data and build new or modify existing test data. Frequently, the testers tend to look at the functional specifications and set up data to test the specifications. This tends to overlook how the software will be used. It is therefore important that the production data are analyzed to understand the types, frequency and characteristics of data so that they can be simulated during testing.
Choose from the following sources while setting up test data:
· Production data
· Data from previous testing
· Data from centralized test beds (if available)
· Data used in other systems (for interface testing)
In many situations, data from the selected source(s) have to be supplemented to cover additional conditions and cases. When setting up effective test data, the goal is to provide adequate requirements and conditions coverage. Try to include some of each of the following types:
· Frequently occurring types and characteristics of data that have high risk. For example, deposit or withdrawal transactions at a banking terminal.
· Frequently occurring types of data with very little exposure. For example, maintenance-type transactions such as name & address changes.
· Low frequency errors that have very little consequence.
· Low frequency errors resulting in heavy losses. For example, the printing of a few additional zeros in a financial report.
Once the data source has been identified, you have to determine the files and sizes of files to use. Always try to use any known standards or conventions for databases and files. Data are then extracted using the appropriate methods.
The key to successful testing is to state the expected results when defining the test conditions and documenting the test scripts. Documentation should include:
· What is being tested
· How is testing done (procedures)
· Where to find the test data
· The expected results.
The keys to reducing the effort to create test data are ... REUSE and RECYCLE!
7. Test Procedure Set Up
The test procedures describe the step-by-step process to conduct the tests. The primary goal is to define the sequence and contents of tests and to establish pass-fail criteria for each. In so doing, you will also be establishing the entry and exit criteria for each test.
As part of the process of setting up entrance criteria, what we are in fact doing is defining the pre-requisites and co-requisites. For example, if we want to test the printing of certain audit reports in batch, we may include the entry of on-line transactions and the bringing down of the on-line system as entry criteria. Again, if we are testing for a specific error condition to be generated and reported on the audit report as part of our test, we may specify a particular entry appearing on the report to be the exit criteria. We cannot exit until that entry appears as expected..
Typically, test procedures include instructions to set up the test facility. For example, if files have to be restored before a test is run, this becomes the first step in the test procedure. If certain tests fail, back out steps may have to be performed to undo what was done up to that point. In that case, some restart and recovery steps may be required once the problem has been fixed. All of these steps are included as part of the test procedure.
Some times it may not be possible to anticipate all error conditions. In that case, steps to handle an unexpected error situation should be described.
Procedures set up may also involve writing the Job Control Language (JCL), or platform-specific equivalent, to be used in production. This is usually tested during the System Test and certified in the UAT and Operability Test Environment. If the JCL is being set up for a new job, it will actually be tested during Integration and System testing before being promoted to UAT. Please follow all relevant JCL standards for setting up the production JCL procedures.
8. Managing and Controlling the Test Effort
8.1 Introduction
Although testing is a part of the development process, the activities that must be managed by the test manager and staff can be quite distinct and critical to the overall success of the project. Outlined below are the major factors that are most important to testing. While some are exclusive to testing, others are shared as part of overall project management.
8.2 Project Plans and Schedules
In larger projects, the test manager will assume responsibility for the portion of the project plans that relate specifically to testing activities. It is important for the test team to institute a tracking and control system to measure the schedule's progress and identify potential bottlenecks. As an example, test plans are written to schedule and control test runs. These can be as complex and extensive as those that are used by the product developers. This activity can demand a level of planning skill on par with a project manager.
It is critical that the testing plans (schedule, costs, and resources) be fully integrated with the development plans and be managed as an overall plan for all sizes of projects. This ensures that coordination is achieved throughout the project.
8.3 Test Team Composition & User Involvement
A key step in managing a successful test project is to identify the key participants who will be involved in testing. It is good to use the concept of a Test Manager. Typically the Test Manager will be from the development team but can come from someone in the users' shop with experience in this or related areas. The role of the Test Manager is to:
· Customize the generic testing methodology
· Define test responsibility / tasks that need to be performed
· Identify individuals who can perform those responsibilities / tasks
· Plan, coordinate, manage and report results of the testing process
The test team staff is responsible for:
· Producing test package material (test cases, test data, test procedures)
· Executing test cases, reporting results, and retesting fixes
The technical staff is responsible for:
· providing the technical guidance required to test and fix the new application
Personnel involved in testing activities can be typically drawn from any of the following groups:
· Developers
· Users
· User management
· Operations
· Independent test group specialists
Effective managing and control of the test effort requires that a test team leader and staff be assigned and the roles and responsibilities defined. For each level of testing, clearly communicate and document the following:
· who creates the test data
· who runs the tests or enters the data
· who checks the results
· who makes the decisions on what corrections are to be made, and
· who makes the required corrections
· who builds and maintains the test facilities
· who operates the testing facilities and procedures (e.g., output handling)
8.4 Measurement and Accountability
You can't manage what you can't measure!

The test manager will track the typical project measures such as schedules and resources. Project teams are all accustomed to those. There should also be an objective set of measurements to track and control the progress of the testing activities and their success. Many of these measurements are just plain common sense and have been validated by many major project experiences. They can be applied at each level of testing.
Among the parameters the test team must know are:
· What parts have I completed testing successfully? (coverage)
· What parts have I attempted to test? (coverage)
· How many variances can I expect? (defect prediction)
· How many variances have I already discovered? (problem management)
· How many variances have I fixed and have left to fix? (problem management)
· What is the impact (seriousness) of the outstanding variances? (problem management)
· At what rate is the team still finding variances? (problem management)
While the list of questions may not be exhaustive, it does represent the most useful information the test team needs to know.
Using the measures that answer the above questions and the history of previous projects, the test planner can make more reliable schedule, resource, and test plans, and use the on-going results to control and report on the success of testing. The accountability of the team will rest on the outcome of these measures but the risk will be managed over the full project life cycle.b
At this point, the specific measures will not be discussed. The methodology training will refer to some of the more popular measures recommended. The measures selected by your project will depend on your needs.
8.5 Reporting
Progress reporting to management must be regular and useful. The format and the frequency of the reports are the responsibility of the test manager or, in the absence of that position, the project manager.
To be useful, progress should be reported in terms of factual and objective measures. The measures, which answer the questions posed in the measurement section above, could be presented or combined and used to answer other related management questions. Graphic representation of the data improves the clarity and impact of the information. The level of detail should be appropriate to the audience to which you are reporting.
The Test Team should contribute to and be recipients of testing progress reports. Success of testing then becomes a visible incentive.
8.6 Problem Management
Once a work product is transferred from its author and becomes part of the test process it becomes subject to problem management procedures. This means that any problem reported will be documented, tracked, and reported until it becomes resolved. There is a tendency not to report very small problems. This must be resisted since the impacts of the problem or its fix may not be fully analyzed and the problem may lose visibility for retest and subsequent analysis.
The status of problems discovered and outstanding must be tracked and reported regularly. Using backlog measures (where outstanding problems are accumulated, counted and reported); you can assess the ability of the project team to cope with the volume and/or severity of problems. If the backlog continues to increase in relation to the numbers of problems discovered, then this information can assist in determining the need to change the project and test schedule. Similarly, if severe problems remain unresolved for prolonged periods this might indicate that there are design issues, which may need addressing.
Another benefit of maintaining a history of problems is the capability of the organization to improve the testing or development processes through causal analysis. For every problem, cause category codes can be assigned and later analyzed for patterns of failures on which to base corrective actions.
Causal analysis is a process whereby every problem's cause is tracked and categorized. Very often a situation may cause multiple problems that manifest themselves in different forms and in different places in a system. If these causes are not tracked, significant time can be wasted throughout a testing exercise. If they are recorded and categorized, identification of common problem-causing situations is facilitated.
If the project has a Problem Management system, it should be used. Otherwise, one should be created and maintained whether supported with automated tools or not.
8.7 Change Management
When the system requirements or specifications are approved at the phase ends, they become candidates for change control.
As part of the evaluation process for a change, the Test Team should be involved since the change may have a significant effect on the direction and cost of testing. It may also affect the test build strategy.
Failure to strictly adhere to formal change management has been the reason for the failure of many projects. Inadequate change control leads to runaway requirements, elongated test schedules and inflated project costs.
If the project has a Change Management system, it should be used. Otherwise, one should be created and maintained whether supported with automated tools or not.
8.8 Configuration Management (Version control)
As development progresses, the interim work products (e.g., design, modules, documentation, and test packages) go through levels of integration and quality. Testing is the process of removing the defects of components under development at one testing level and then promoting the new defect-free work product to the next test level.
It is absolutely critical that the highest level of test not be corrupted with incomplete or defective work products. It might require the retest of some or all of the work products at that level to regain the state of confidence in their quality.
A test team member usually (but not always) acts as the "gatekeeper" to the next level of test and assures the quality of the interim work products before they are promoted. Update access is restricted from the authors. Control may be enforced by procedures and/or automation (such as software library control systems).
These controls are key to avoiding rework or defective work slipping by uncontrolled. A configuration or version control process and tools must be present and proactively managed throughout the project.
8.9 Reuse
This testing methodology assumes a process of creating an initial work product and then refining it to subsequent levels of detail as you progress through the stages of development. A Master Test Plan is first created with a minimal level of detail known at the time and is used as the template and source of basic information for the Detailed Test Plans. The test sets, cases, and data are created and successively refined as they are used through the design, code/unit test, and integration test for use in the System Test and the UAT. Even then, they are not discarded but are recycled as the System Test and/or UAT regression test packages.
The deliverables produced at one phase by the test team should be considered as candidates to be re-used fully or in part by subsequent phases. The final product of all the testing should be the regression test package delivered as part of the implementation to those who will maintain the system.

Appendix B. Integration Approaches
Introduction
A critical decision which impacts the implementation phase of the development process more than any other is the strategy to determine the sequence for developing and integrating system components such as modules, programs, and sub-systems. There are three strategies:
· Top down
· Bottom up
· A combination of both
A conscious decision must be made by the project / test team, early in the development life cycle to merge the development and testing strategies so as to optimize the project resources, and deliverables.
Some opportunities to exploit advantages through an integration approach when you consider system units which:
· Form a stand-alone subsystem and those, which interface with other units.
· Must exist before other units can be tested.
· Must be delivered first (which are critical to the success of the project) or which are on the critical path of the project.
Top Down
The Top Down Approach is a strategy, which starts with construction of the top level or control modules and then integrates lower level modules within the structure. In order to test Top Down, you will need to construct program and module stubs.
Stubs
The purpose of a stub or dummy program is to simulate the existence of a program or module until it can be replaced by the real module. This use of temporary code is an example of "scaffolding". Each module integrated into the program or job stream requires stubs to represent the modules it calls until built. Testing can occur on the overall structure of modules itself and on individual modules as part of the whole structure as they are integrated and come on-line.
Advantage
· Tends to encourage the overlapping of design and testing.
· Performs incremental testing on whole unit at each integration
Disadvantages
· Extra construction of stubs (dummies) is required.
· Design deficiencies may not become apparent until lower level modules are tested.
Bottom Up
In the Bottom Up approach, modules or program units are written and tested completely starting at the lowest level. Successively, higher levels are added and tested until the program or programs are complete.
Drivers
A driver is a program that simulates the calls to one or more modules under development for the purpose of testing those modules. For example, a driver may generate the data required for the module under test or, read a file with test data, which is then formatted and passed to the module. Usually, but not always, a trace or debug report may be created showing the data sent to and returned from the module. If the module being tested and any modules that it calls are fully coded then the use of drivers is an effective approach to testing.
Development of driver modules or programs is sometimes thought to be a tedious process, which is often avoided. However, if such facilities are developed and fully documented they can provide payback in the future for subsequent test exercises.
Advantage
· Encourages reusable code
· Can be modified for other "driver" situations
Disadvantage
· Requires the construction of drivers (additional programming).

Top Down and Bottom Up
The best features of Top Down testing and Bottom Up testing can sometimes be combined.
· Top Down testing is effective in iterative or incremental development strategies. Bottom up is effective when a number of common modules are used by different systems and are normally coded and tested first.
· A combination of top-down and bottom-up strategies may be best depending on above considerations.
Appendix C. Black Box/White Box Testing Techniques
Introduction
Two techniques are commonly used in the development of test conditions. They are Black Box and White Box techniques. An overview of the two techniques is given below. More detailed information on how and when to use each technique are covered in the Test Process Model.
Black Box Testing
In the Black Box approach, the testers have an "outside" view of the system. They are concerned with “what is done” NOT “how it is done".
Black Box testing is requirements oriented and is used at all test levels.
The system is defined and viewed functionally. Inputs are entered into the system and the outputs from the system are examined. All possible inputs and/or input combinations are entered to test the system. Both valid and invalid inputs are used to test the system.
Black Box testing uses the following sub-techniques:
1. Equivalence Partitioning
Equivalence partitioning is a method for developing test cases by analyzing each possible class of bvalues. In equivalence partitioning you can select any element from:
· Valid equivalence class
· Invalid equivalence class
Identify equivalence classes by partitioning each input or external condition into valid and invalid classes.
Example 1. First character must be alphabetic
Valid class Invalid class
Any one of the 26 Any special character
Alphabetic characters Any numeric character
It is not necessary to test with each one of the valid or invalid classes.
Example 2. Input X will range in value from 0 to 40
Valid class Invalid class
0<= X <= 40 X <> 40
2. Boundary Value Analysis
Boundary value analysis is one of the most useful test case design methods and is a refinement to equivalence partitioning. Boundary conditions are those situations directly on, above, and beneath the edges of input equivalence classes and output equivalence classes. In boundary analysis one or more elements must be selected to test each edge. There is a focus on both input conditions and output results.
Example: If amount greater than $1.00...
Test case 1: Amount = 0.99 (false)
Test case 2: Amount = 1.00 (false)
Test case 3: Amount = 1.01 (true)
White Box Testing
In the White Box approach, the testers have an inside view of the system. They are concerned with "how it is done" NOT "what is done".
White Box testing is logic oriented. The testers are concerned with the execution of all possible paths of control flow through the program.
The "White Box" approach is essentially a Unit test method (which is sometimes used in the Integration test) and is almost always performed by technical staff.
Each technique of White Box testing offers only one facet to the overall complex solution to the problem of completely testing all the aspects of logic paths and their content.
Techniques for White Box testing are the following:
1. Statement Coverage
Statement Coverage is a White Box technique used to verify that every statement in the program is executed at least once. This ensures complete logic coverage. Consider the following program segment:
A + B
If (A = 3) Then
B = X + Y
End-If-Then

While (A > 0) Do
Read (X)
A = A - 1
End-While-Do
If "(A = 3)" is tested, only the true condition is followed but not the false condition "A not=3". Statement coverage will only test that the statement itself has been reached; it will not test the coverage if the condition is false. Both true and false outcomes must be considered.
2. Decision Coverage
Decision Coverage is an improvement over Statement Coverage in that all true and false branches are tested. Decision Coverage is also known as Branch Coverage. Decision Coverage is achieved when all branches have been traversed and every entry point has been invoked at least once. But once a condition is satisfied it won't necessarily test other branch conditions included in the statement.
Example:
If A <> 20 Then
B = X + Y
If "A <> 20". This would require another decision test.
3. Condition Coverage
The objective of Condition Coverage testing is that all conditions within a statement, not just each decision outcome, are tested to ensure full coverage.
Consider the following program segment:
A = X
If (A > 3) or (A < b =" X"> 0) and (Not EOF) Do
Read (X)
A = A - 1
End-While-Do
Condition coverage is achieved when each condition has been true at least once and false at least once, and every point has been invoked at least once.
4. Multiple Condition Coverage
The purpose is to ensure that all combinations of conditions are tested. Decision tables are a good tool to use to determine the number of unique combinations in a program.
5. Path Coverage
This ensures that all possible combinations of flow paths, based upon all possible combinations of condition outcomes in each decision from all points of entry to all exit points, are invoked at least once.
Selection Guidelines
1. Black Box testing
· Consider each possible class of values
· Consider inside and outside edge of boundaries of the range
· Consider expected input and output values

2. White Box testing
· Select appropriate techniques
· Identify all decisions, conditions and paths
· Determine the number of test cases needed
· Identify the specific combinations of conditions needed
· Determine variable values required
· Select specific test case values (input and output)
· Inspect for completeness and correctness
Appendix D. Test Case Design
A general list of test conditions is shown below. These can be used as an aid to develop test cases.
Transaction Test Cases
The following section includes some of the key considerations to keep in mind while designing test cases based on the type of processing (on-line, batch or transaction flow) and the level of testing (unit or system test and so on). Also, consider the use of:
· Inspections and walkthroughs to check for completeness and consistency.
· Automation to aid in administration, documentation, regression testing and operation.
General Conditions
· Adding records
- Add a record using valid data.
- Add an existing record.
· Updating records
- Update a record using valid data.
- Update a record that does not exist.
- Update a closed record.
- Make multiple updates to the same record.
· Closing records
- Close (delete flag or move to history) using valid data.
- Re-close a closed record.
- Close a non-existent record.
· All records
- Add, update and close a record in the same test.
- Use all possible data entry methods.
For Batch Transactions
· Try transactions singly and in combinations
· Pass duplicate transactions
· Pass combinations of valid and invalid values, multiple errors
· Pass data set with sequence errors
· Pass batches with invalid dates, duplicates, missing headers/trailers, incorrect totals/control data
· Run with no data at all
· Run two or more days data as one run
For On-line Transactions
· Process null request for all transactions
· Try interrupting the transaction without completing it
All Transactions - Invalid Data
· Test all types of transactions using valid and invalid data.
· Set up test records so that balances will go high, low, zero, and negative, negative then positive.
· Try transactions against closed and non-existent records.
· Check ranges high and low, within and outside.
· Check arithmetic fields for overflow, negative values, rounding, truncating, zero divide, alignment.
· Try numeric fields with alphanumeric, blanks, leading blanks, trailing blanks, and embedded blanks.
· Try date fields with invalid dates (months, days, and years).
· Try special characters or keys - *,?,/1/2,1/4,EOF, etc.
Screen/Panel/Window Test Cases
· Screen flow.
· Test the number of display lines for 0 entries, minimum, maximum, minimum - 1, maximum + 1 number of entries. For example, if an order display screen can display up to 20 items, test it against orders with no items, 1 item, 20 items and 21 items.
· Force continuation of all output display screens.
· Test that all program function keys perform the desired action on all applicable screens.
· Consistency between screens, format, use of colors, use of function keys and of action codes is identical on all screens.
· Availability and consistency of help screens.
· Appropriate return from help screens.
Glossary of Testing Terms and Definitions
Term
Definition
Acceptance Criteria
The definition of the results expected from the test cases used for acceptance testing. The product must meet these criteria before implementation can be approved.
Acceptance Testing
(1) Formal testing conducted to determine whether or not a system satisfies its acceptance criteria and to enable the client to determine whether or not to accept the system. (2) Formal testing conducted to enable a user, client, or other authorized entity to determine whether to accept a system or component.
Acceptance Test Plan
Describes the steps the client will use to verify that the constructed system meets the acceptance criteria. It defines the approach to be taken for acceptance testing activities. The plan identifies the items to be tested, the test objectives, the acceptance criteria, the testing to be performed, test schedules, entry/exit criteria, staff requirements, reporting requirements, evaluation criteria, and any risks requiring contingency planning.
Adhoc Testing
A loosely structured testing approach that allows test developers to be creative in their test selection and execution. Adhoc testing is targeted at known or suspected problem areas.
Audit and Controls Testing
A functional type of test that verifies the adequacy and effectiveness of controls and completeness of data processing results.
Auditability
A test focus area defined as the ability to provide supporting evidence to trace processing of data.
Backup and Recovery Testing
A structural type of test that verifies the capability of the application to be restarted after a failure.
Black Box Testing
Evaluation techniques that are executed without knowledge of the program’s implementation. The tests are based on an analysis of the specification of the component without reference to its internal workings.
Bottom-up Testing
Approach to integration testing where the lowest level components are tested first then used to facilitate the testing of higher level components. This process is repeated until the component at the top of the hierarchy is tested. See "Top-down."
Boundary Value Analysis
A test case selection technique that selects test data that lie along "boundaries" or extremes of input and output possibilities. Boundary Value Analysis can apply to parameters, classes, data structures, variables, loops, etc.
Branch Testing
A white box testing technique that requires each branch or decision point to be taken once.
Build
(1) An operational version of a system or component that incorporates a specified subset of the capabilities that the final product will provide. Builds are defined whenever the complete system cannot be developed and delivered in a single increment. (2) A collection of programs within a system that are functionally independent. A build can be tested as a unit and can be installed independent of the rest of the system.
Business Function
A set of related activities that comprise a stand-alone unit of business. It may be defined as a process that results in the achievement of a business objective. It is characterized by well-defined start and finish activities and a workflow or pattern.
Capability Maturity Model (CMM)
A model of the stages through which software organizations progress as they define, implement, evolve, and improve their software process. This model provides a guide for selecting process improvement strategies by determining current process capabilities and identifying the issues most critical to software quality and process improvement. This concept was developed by the Software Engineering Institute (SEI) at Carnegie Mellon University.
Causal Analysis
The evaluation of the cause of major errors, to determine actions that will prevent reoccurrence of similar errors.
Change Control
The process, by which a change is proposed, evaluated, approved or rejected, scheduled, and tracked.
Change Management
A process methodology to identify the configuration of a release and to manage all changes through change control, data recording, and updating of baselines.
Change Request
A documented proposal for a change of one or more work items or work item parts.
Condition Testing
A white box test method that requires all decision conditions be executed once for true and once for false.
Configuration Management
(1) The process of identifying and defining the configuration items in a system, controlling the release and change of these items throughout the system life cycle, recording and reporting the status of configuration items and change requests, and verifying the completeness and correctness of configuration items. (2) A discipline applying technical and administrative direction and surveillance to (a) identify and document the functional and physical characteristics of a configuration items, (b) control changes to those characteristics, and (c) record and report change processing and implementation status.
Conversion testing
A functional type of test that verifies the compatibility of converted programs, data and procedures with the “old” ones that are being converted or replaced.
Coverage
The extent to which test data tests a program’s functions, parameters, inputs, paths, branches, statements, conditions, modules or data flow paths.
Coverage Matrix
Documentation procedure to indicate the testing coverage of test cases compared to possible elements of a program environment (i.e. inputs, outputs, parameters, paths, cause-effects, equivalence partitioning, etc.).
Continuity of Processing
A test focus area defined as the ability to continue processing if problems occur. Included is the ability to backup and recover after a failure.
Correctness
A test focus area defined as the ability to process data according to prescribed rules. Controls over transactions and data field edits provide an assurance on accuracy and completeness of data.
Data flow Testing
Testing in which test cases are designed based on variable usage within the code.
Debugging
The process of locating, analyzing, and correcting suspected faults. Compare with testing.
Decision Coverage
Percentage of decision outcomes that have been exercised through (white box) testing.
Defect
A variance from expectations. See also Fault.
Defect Management
A set of processes to manage the tracking and fixing of defects found during testing and to perform causal analysis.
Documentation and Procedures Testing
A functional type of test that verifies that the interface between the system and the people works and is usable. It also verifies that the instruction guides are helpful and accurate.
Design Review
(1) A formal meeting at which the preliminary or detailed design of a system is presented to the user, customer or other interested parties for comment and approval. (2) The formal review of an existing or proposed design for the purpose of detection and remedy of design deficiencies that could affect fitness-for-use and environmental aspects of the product, process or service, and/or for identification of potential improvements of performance, safety and economic aspects.
Desk Check
Testing of software by the manual simulation of its execution. It is one of the static testing techniques.
Detailed Test Plan
The detailed plan for a specific level of dynamic testing. It defines what is to be tested and how it is to be tested. The plan typically identifies the items to be tested, the test objectives, the testing to be performed, test schedules, personnel requirements, reporting requirements, evaluation criteria, and any risks requiring contingency planning. It also includes the testing tools and techniques, test environment set up, entry and exit criteria, and administrative procedures and controls.
Driver
A program that exercises a system or system component by simulating the activity of a higher level component.
Dynamic Testing
Testing that is carried out by executing the code. Dynamic testing is a process of validation by exercising a work product and observing the behavior of its logic and its response to inputs.
Entry Criteria
A checklist of activities or work items that must be complete or exist, respectively, before the start of a given task within an activity or sub-activity.
Environment
See Test Environment.
Equivalence Partitioning
Portion of the component’s input or output domains for which the component’s behavior is assumed to be the same from the component’s specification.
Error
(1) A discrepancy between a computed, observed or measured value or condition and the true specified or theoretically correct value or condition. (2) A human action that results in software containing a fault. This includes omissions or misinterpretations, etc. See Variance.
Error Guessing
A test case selection process that identifies test cases based on the knowledge and ability of the individual to anticipate probable errors.
Error Handling Testing
A functional type of test that verifies the system function for detecting and responding to exception conditions. Completeness of error handling determines the usability of a system and ensures that incorrect transactions are properly handled.
Execution Procedure
A sequence of manual or automated steps required to carry out part or all of a test design or execute a set of test cases.
Exit Criteria
(1) Actions that must happen before an activity is considered complete. (2) A checklist of activities or work items that must be complete or exist, respectively, prior to the end of a given process stage, activity, or sub-activity.

Expected Results
Predicted output data and file conditions associated with a particular test case. Expected results, if achieved, will indicate whether the test was successful or not. Generated and documented with the test case prior to execution of the test.
Fault
(1) An accidental condition that causes a functional unit to fail to perform its required functions (2) A manifestation of an error in software. A fault if encountered may cause a failure. Synonymous with bug.
Full Lifecycle Testing
The process of verifying the consistency, completeness, and correctness of software and related work products (such as documents and processes) at each stage of the development life cycle.
Function
(1) A specific purpose of an entity or its characteristic action. (2) A set of related control statements that perform a related operation. Functions are sub-units of modules.
Function Testing
A functional type of test, which verifies that each business function, operates according to the detailed requirements, the external and internal design specifications.
Functional Testing
Selecting and executing test cases based on specified function requirements without knowledge or regard of the program structure. Also known as black box testing. See "Black Box Testing."
Functional Test
Types
Those kinds of tests used to assure that the system meets the business requirements, including business functions, interfaces, usability, audit & controls, and error handling etc. See also Structural Test Types.
Implementation
(1) A realization of an abstraction in more concrete terms; in particular, in terms of hardware, software, or both. (2) The process by which software release is installed in production and made available to end-users.
Inspection
(1) A group review quality improvement process for written material, consisting of two aspects: product (document itself) improvement and process improvement (of both document production and inspection). (2) A formal evaluation technique in which software requirements, design, or code are examined in detail by a person or group other than the author to detect faults, violations of development standards, and other problems. Contrast with walk-through.
Installation Testing
A functional type of test which verifies that the hardware, software and applications can be easily installed and run in the target environment.

Integration Testing
A level of dynamic testing which verifies the proper execution of application components and does not require that the application under test interface with other applications.

Interface / Inter-system Testing
A functional type of test which verifies that the interconnection between applications and systems functions correctly.
JAD
An acronym for Joint Application Design. Formal session(s) involving clients and developers used to develop and document consensus on work products, such as client requirements, design specifications, etc.
Level of Testing
Refers to the progression of software testing through static and dynamic testing.
Examples of static testing levels are Project Objectives Review, Requirements Walkthrough, Design (External and Internal) Review, and Code Inspection.
Examples of dynamic testing levels are: Unit Testing, Integration Testing, System Testing, Acceptance Testing, Systems Integration Testing and Operability Testing.
Also known as a test level.
Lifecycle
The software development process stages. Requirements, Design, Construction (Code/Program, Test), and Implementation.
Logical Path
A path that begins at an entry or decision statement and ends at a decision statement or exit.
Maintainability
A test focus area defined as the ability to locate and fix an error in the system. Can also be the ability to make dynamic changes to the system environment without making system changes.
Master Test Plan
A plan that addresses testing from a high-level system viewpoint. It ties together all levels of testing (unit test, integration test, system test, acceptance test, systems integration, and operability). It includes test objectives, test team organization and responsibilities, high-level schedule, test scope, test focus, test levels and types, test facility requirements, and test management procedures and controls.
Operability
A test focus area defined as the effort required (of support personnel) to learn and operate a manual or automated system. Contrast with Usability.
Operability Testing
A level of dynamic testing in which the operations of the system are validated in the real or closely simulated production environment. This includes verification of production JCL, installation procedures and operations procedures. Operability Testing considers such factors as performance, resource consumption, adherence to standards, etc. Operability Testing is normally performed by Operations to assess the readiness of the system for implementation in the production environment.
Operational Testing
A structural type of test that verifies the ability of the application to operate at an acceptable level of service in the production-like environment.
Parallel Testing
A functional type of test, which verifies that the same input on “old” and “new” systems, produces the same results. It is more of an implementation that a testing strategy.
Path Testing
A white box testing technique that requires all code or logic paths to be executed once. Complete path testing is usually impractical and often uneconomical.
Performance
A test focus area defined as the ability of the system to perform certain functions within a prescribed time.
Performance Testing
A structural type of test which verifies that the application meets the expected level of performance in a production-like environment.

Portability
A test focus area defined as ability for a system to operate in multiple operating environments.
Problem
(1) A call or report from a user. The call or report may or may not be defect oriented. (2) A software or process deficiency found during development. (3) The inhibitors and other factors that hinder an organization’s ability to achieve its goals and critical success factors. (4) An issue that a project manager has the authority to resolve without escalation. Compare to ‘defect’ or ‘error’.
Quality Plan
A document which describes the organization, activities, and project factors that have been put in place to achieve the target level of quality for all work products in the application domain. It defines the approach to be taken when planning and tracking the quality of the application development work products to ensure conformance to specified requirements and to ensure the client’s expectations are met. A
Regression Testing
A functional type of test, which verifies that changes to one part of the system have not caused unintended adverse effects to other parts.
Reliability
A test focus area defined as the extent to which the system will provide the intended function without failing.
Requirement
(1) A condition or capability needed by the user to solve a problem or achieve an objective. (2) A condition or capability that must be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed document. The set of all requirements forms the basis for subsequent development of the system or system component.
Review
A process or meeting during which a work product, or set of work products, is presented to project personnel, managers, users or other interested parties for comment or approval.
Root Cause Analysis
See Causal Analysis.
Scaffolding
Temporary programs may be needed to create or receive data from the specific program under test. This approach is called scaffolding.
Security
A test focus area defined as the assurance that the system/data resources will be protected against accidental and/or intentional modification or misuse.
Security Testing
A structural type of test which verifies that the application provides an adequate level of protection for confidential information and data belonging to other systems.
Software Quality
(1) The totality of features and characteristics of a software product that bear on its ability to satisfy given needs; for example, conform to specifications. (2)The degree to which software possesses a desired combination of attributes. (3)The degree to which a customer or user perceives that software meets his or her composite expectations. (4)The composite characteristics of software that determine the degree to which the software in use will meet the expectations of the customer.
Software Reliability
(1) The probability that software will not cause the failure of a system for a specified time under specified conditions. The probability is a function of the inputs to and use of the system as well as a function of the existence of faults in the software. The inputs to the system determine whether existing faults, if any, are encountered. (2) The ability of a program to perform a required function under stated conditions for a stated period of time.
Statement Testing
A white box testing technique that requires all code or logic statements to be executed at least once.
Static Testing
(1) The detailed examination of a work product's characteristics to an expected set of attributes, experiences and standards. The product under scrutiny is static and not exercised and therefore its behavior to changing inputs and environments cannot be assessed. (2) The process of evaluating a program without executing the program. See also desk checking, inspection, walk-through.
Stress / Volume Testing
A structural type of test that verifies that the application has acceptable performance characteristics under peak load conditions.
Structural Function
Structural functions describe the technical attributes of a system.
Structural Test Types
Those kinds of tests that may be used to assure that the system is technically sound.
Stub
(1) A dummy program element or module used during the development and testing of a higher level element or module. (2) A program statement substituting for the body of a program unit and indicating that the unit is or will be defined elsewhere. The inverse of Scaffolding.
Sub-system
(1) A group of assemblies or components or both combined to perform a single function. (2) A group of functionally related components that are defined as elements of a system but not separately packaged.
System
A collection of components organized to accomplish a specific function or set of functions.
Systems Integration Testing
A dynamic level of testing which ensures that the systems integration activities appropriately address the integration of application subsystems, integration of applications with the infrastructure, and impact of change on the current live environment.
System Testing
A dynamic level of testing in which all the components that comprise a system are tested to verify that the system functions together as a whole.
Test Bed
(1) A test environment containing the hardware, instrumentation tools, simulators, and other support software necessary for testing a system or system component. (2) A set of test files, (including databases and reference files), in a known state, used with input test data to test one or more test conditions, measuring against expected results.
Test Case
(1) A set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement. (2) The detailed objectives, data, procedures and expected results to conduct a test or part of a test.
Test Condition
A functional or structural attribute of an application, system, network, or component thereof to be tested.
Test Conditions Matrix
A worksheet used to formulate the test conditions that, if met, will produce the expected result. It is a tool used to assist in the design of test cases.
Test Conditions Coverage Matrix
A worksheet that is used for planning and for illustrating that all test conditions are covered by one or more test cases. Each test set has a Test Conditions Coverage Matrix. Rows are used to list the test conditions and columns are used to list all test cases in the test set.
Test Coverage Matrix
A worksheet used to plan and cross check to ensure all requirements and functions are covered adequately by test cases.
Test Data
The input data and file conditions associated with a specific test case.
Test Environment
The external conditions or factors that can directly or indirectly influence the execution and results of a test. This includes the physical as well as the operational environments. Examples of what is included in a test environment are: I/O and storage devices, data files, programs, JCL, communication lines, access control and security, databases, reference tables and files (version controlled), etc.
Test Focus Areas
Those attributes of an application that must be tested in order to assure that the business and structural requirements are satisfied.
Test Level
See Level of Testing.
Test Log
A chronological record of all relevant details of a testing activity
Test Matrices
A collection of tables and matrices used to relate functions to be tested with the test cases that do so. Worksheets used to assist in the design and verification of test cases.
Test Objectives
The tangible goals for assuring that the Test Focus areas previously selected as being relevant to a particular Business or Structural Function are being validated by the test.
Test Plan
A document prescribing the approach to be taken for intended testing activities. The plan typically identifies the items to be tested, the test objectives, the testing to be performed, test schedules, entry / exit criteria, personnel requirements, reporting requirements, evaluation criteria, and any risks requiring contingency planning.
Test Procedure
Detailed instructions for the setup, operation, and evaluation of results for a given test. A set of associated procedures is often combined to form a test procedures document.
Test Report
A document describing the conduct and results of the testing carried out for a system or system component.
Test Run
A dated, time-stamped execution of a set of test cases.
Test Scenario
A high-level description of how a given business or technical requirement will be tested, including the expected outcome; later decomposed into sets of test conditions, each in turn, containing test cases.
Test Script
A sequence of actions that executes a test case. Test scripts include detailed instructions for set up, execution, and evaluation of results for a given test case.
Test Set
A collection of test conditions. Test sets are created for purposes of test execution only. A test set is created such that its size is manageable to run and its grouping of test conditions facilitates testing. The grouping reflects the application build strategy.
Test Sets Matrix
A worksheet that relates the test conditions to the test set in which the condition is to be tested. Rows list the test conditions and columns list the test sets. A checkmark in a cell indicates the test set will be used for the corresponding test condition.
Test Specification
A set of documents that define and describe the actual test architecture, elements, approach, data and expected results. Test Specification uses the various functional and non-functional requirement documents along with the quality and test plans. It provides the complete set of test cases and all supporting detail to achieve the objectives documented in the detailed test plan.
Test Strategy
A high level description of major system-wide activities which collectively achieve the overall desired result as expressed by the testing objectives, given the constraints of time and money and the target level of quality. It outlines the approach to be used to ensure that the critical attributes of the system are tested adequately.
Test Type
See Type of Testing.
Testability
(1) The extent to which software facilitates both the establishment of test criteria and the evaluation of the software with respect to those criteria. (2) The extent to which the definition of requirements facilitates analysis of the requirements to establish test criteria.
Testing
The process of exercising or evaluating a program, product, or system, by manual or automated means, to verify that it satisfies specified requirements, to identify differences between expected and actual results.
Testware
The elements that are produced as part of the testing process. Testware includes plans, designs, test cases, test logs, test reports, etc.
Top-down
Approach to integration testing where the component at the top of the component hierarchy is tested first, with lower level components being simulated by stubs. Tested components are then used to test lower level components. The process is repeated until the lowest level components have been tested.
Transaction Flow Testing
A functional type of test that verifies the proper and complete processing of a transaction from the time it enters the system to the time of its completion or exit from the system.
Type of Testing
Tests a functional or structural attribute of the system, e.g., Error Handling, Usability. (Also known as test type.)
Unit Testing
The first level of dynamic testing and is the verification of new or changed code in a module to determine whether all new or modified paths function correctly.
Usability
A test focus area defined as the end-user effort required to learn and use the system. Contrast with Operability.
Usability Testing
A functional type of test which verifies that the final product is user-friendly and easy to use.
User Acceptance Testing
See Acceptance Testing.
Validation
(1) The act of demonstrating that a work item is in compliance with the original requirement. For example, the code of a module would be validated against the input requirements it is intended to implement. Validation answers the question "Is the right system being built?” (2) Confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use have been fulfilled. See "Verification".
Variance
A mismatch between the actual and expected results occurring in testing. It may result from errors in the item being tested, incorrect expected results, invalid test data, etc. See "Error".

Verification
(1) The act of demonstrating that a work item is satisfactory by using its predecessor work item. For example, code is verified against module level design. Verification answers the question "Is the system being built right?” (2) Confirmation by examination and provision of objective evidence that specified requirements have been fulfilled. See "Validation".

Walkthrough
A review technique characterized by the author of the object under review guiding the progression of the review. Observations made in the review are documented and addressed. Less formal evaluation technique than an inspection.
White Box Testing
Evaluation techniques that are executed with the knowledge of the implementation of the program. The objective of white box testing is to test the program's statements, code paths, conditions, or data flow paths.
Work Item
A software development lifecycle work product.
Work Product
(1) The result produced by performing a single task or many tasks. A work product, also known as a project artifact, is part of a major deliverable that is visible to the client. Work products may be internal or external. An internal work product may be produced as an intermediate step for future use within the project, while an external work product is produced for use outside the project as part of a major deliverable. (2) As related to test, a software deliverable that is the object of a test, a test work item.

Testing web performance

Friday, May 11, 2007

ISTQB sample exam papers

Monday, February 26, 2007

EDS to buy RelQ for $40 million

Friday, February 23, 2007

Testing Books

Testing Companies

Testing Certification

Testing Concepts

About Me

Welcome to Your own tetsing site

Blog Archive