Commonwealth of Pennsylvania

Department of Environmental Protection

Bureau of Information Technology

Project Management Office

System Test Plan

For

<Project Title>

Prepared by: <Name of Person Preparing Report>

Date: <January 1, 2005>

Project Manager: <Name of Project Manager>

This page is left intentionally blank.

Document Control

Author

Document Name:	System Test Plan
Version Number:
Version Date:
Project Manager:
Date Issued:

Approvals

The Commonwealth of Pennsylvania, Department of Environmental Protection (DEP) approves the System Test Plan within this document. Approval of this document signifies that this System Test Plan is a complete, quality implementation of the deliverable requirements as specified for the authorized work.

Roles	Name	Signature	Date
Quality Manager
Project Management Officer
Project Sponsor
Chief Information Officer

Change History

Version	Date	Revision Descriptions
V.01
V.02
V.03

TABLE OF CONTENTS

Document Control	i

1.	Introduction	1

1.1	Scope and Objectives	1

1.2	Approach	1

2.	Test and Acceptance Process	2

2.1	Application Test and Acceptance Process	3

2.2	Data Conversion Test and Acceptance Process	6

2.3	Documentation Test and Acceptance Process	7

2.4	Training Test and Acceptance Process	8

2.5	Maintenance Test and Acceptance Process	9

3.	Test Responsibilities	10

3.1	Deliverable Developer	10

3.2	QA Manager	10

3.3	Project Manager	11

4.	Specific Testing Procedures and Associated Documentation	12

4.1	Module Testing	12

4.2	Integration Testing	13

4.3	Acceptance Testing	13

Appendix A - Glossary of Abbreviations, Acronyms and Terms	15

1.	Introduction

1.1	Scope and Objectives

This System Test Plan provides guidance for the quality assurance (QA) in development and delivery of the <fill in project name in previous field and delete this field> project. This document outlines the QA procedures and methods for the development through delivery of the software. The purpose of this plan is to ensure that project requirements are met, quality is maintained, and procedures be verified.

The System Test Plan specifically documents the strategy and procedures that will be used in the comprehensive testing of the Deliverables.

In addition this document includes standards for the creation, documentation, and execution of development and acceptance tests.

The system has been broken down into small units called modules for the purposes of quality assurance. Modules are usually easily distinguishable functional units such as mapping to a specific point, survey screens, or a report, but may also be parts of the system such as menus, toolbars, and the help sub-system. Most QA procedures will be implemented on an individual module level until just before formal acceptance testing when module integration testing and procedures will be more common.

1.2	Approach

Our approach to QA uses system requirements, development standards and procedures, testing, and traceable communications as the basic elements to ensure quality.

System requirements are used as the blueprint for the module functional performance, usability, and system security that can be reviewed and refined in user interface prototype sessions. Prototype sessions allow user representatives to actually define the final look and feel of the system as they begin to learn how to use the tools created to meet their job requirements.

Standards and procedures provide rules and validation that result in consistency of the end system. The consistency of the system increases usability and decreases system lifecycle maintenance time.

Testing a system is a process that continues throughout a system’s life cycle. The system will undergo several levels of testing: informal checks by the designer and developer during development, formal tests by the developer during development, QA tests by the QA staff at the end of the development phase, stress testing by the technical staff and a final system acceptance test by the customer. This document describes the formal tests performed by developers, technical and the QA staff during the design and build stages of a project.

The system will be tested for both performance and functionality. Performance tests will ensure that the modules act correctly: i.e. menus call the correct screens; the cursor moves to the correct field; list of values windows perform correctly; help and hint messages are available and informative and navigation, provided keys and buttons perform the correct function. Functionality tests will verify that the modules perform the functions specified in the Functional Requirements Specification.

In addition, Stress tests will be performed. Stress testing will verify that, when accessed by multiple users during the same time, the system will continue to function within the access time standards.

Finally, the project team uses traceable (written) communications in all significant matters to facilitate clear understanding and historical record keeping. Documents that support the test process are described in the relevant section of this test plan.

Errors and problems reported as a result of the testing and acceptance process, including any found during the maintenance/warranty period, are tracked in the Error Log, maintained as a Microsoft Access database.

2.	Test and Acceptance Process

The test and acceptance process for each deliverable is dependent on the type of deliverable. Specific processes are described for:

1.	Application (includes screens and reports)

2.	Data Conversion

3.	SDM Documentation

4.	Training

5.	Maintenance

2.1	Application Test and Acceptance Process

The Application, which includes mapping along with the screens and reports for purposes of this test plan, will be tested and accepted using the following strategy:

1.	Development Analyst develops a specific module based on a Functional Specification. (The Functional Specification, also referred to as a report or screen specification, is developed based on the relevant portions of the Requirements Document and provides specific details necessary for functional development). As an integral part of this process, the development analyst performs iterative testing and debugging. This development testing is not formally tracked.

2.	There is a Testing Template with space to record six checks: three by the developer, and three by QA. The form is specific to screens or reports, as there are differences in what needs to be checked in each.

3.	When the Development Analyst believes the module is complete, the Developer Test #1 is completed for the module and submitted to QA by the developer. Errors reported starting with this checklist, and during all subsequent steps, will be entered and resolution tracked in the Error Log.

4.	The QA Manager reviews the checklist as submitted by the developer and completes QA Test #1 with a focus on standards implementation and application functions. The QA Manager reviews any discrepancies and/or problems uncovered with the Development Analyst.

5.	The Development Analyst corrects any discrepancies and/or problems.

6.	When all functionality and database transactions are working properly, the Development Analyst completes Developer Test #2 and submits it to QA. (For reports that do not modify the database, this checklist may be skipped at the discretion of the QA Manager.)

7.	The QA Manager reviews the checklist as submitted by the developer and completes QA Test #2 with a focus on the database transactions. The QA Manager reviews any discrepancies and/or problems uncovered with the Development Analyst. (For reports that do not modify the database, this checklist may be skipped at the discretion of the QA Manager.)

8.	The Development Analyst corrects any discrepancies and/or problems.

9.	When the entire module (screen or reports) is working properly, the Development Analyst completes Developer Final Test and submits it to QA. Execution of this iteration of the checklist also constitutes the completion of the Developer’s Unit Testing.

10.	The QA Manager reviews the checklist as submitted by the developer and completes QA Final Test, which verifies complete functionality of the module (screen or report), and completes Unit Testing. If the QA Manager discovers any discrepancies and/or problems, they are reviewed with the Development Analyst. The module is not passed through QA Final until all problems have been resolved to the satisfaction of the QA Manager.

11.	Error Reports are issued to project manager based on the Error Log on an as‑needed basis. The Error Log is maintained by the QA Manager and/or designate to support the entire process (assignment, correction testing, and resolution).

12.	Modules that pass the Unit Test are released to the User Representatives in the test database for pre-acceptance testing. The intent of this process is to provide users a mechanism to perform early acceptance testing -- testing both the acceptance process and the module. This early testing does not replace the formal acceptance testing and is not considered a “formal” step in the process. Problems and/or clarifications reported by users as a result of testing in this environment are tracked through closure using the Error Log. Scheduling of these corrections is managed by the development team so as not to disrupt the overall flow and time schedule of the project. No other Team test management is provided for this early user testing.

13.	The documentation development and training team works on modules in both the development stage and those that have passed Unit Testing. Problems and/or clarifications discovered as a result of their work are tracked through closure using the Error Log. Scheduling of corrections is managed by the development team so as not to disrupt the overall flow and time schedule of the project.

14.	Integration testing is a systematic technique for constructing the program structure while at the same time conducting tests to uncover any errors associated with interfacing. The objective is to take unit‑tested modules and build a program structure that has been dictated by design. Integration testing will be done after the modules are linked through the menu and to each other. Integration testing will verify the menu operations, parameter passing and transfers between modules, and data integrity between modules.

15.	Whenever a module is enhanced or changed, the QA team will re‑test the module to determine if the change had created any unwanted alteration in functionality, corrected errors as intended, and did not introduce any new errors. This testing will be recorded and placed on file with the module’s checklists. If integration testing was previously done, portions or the total may also be repeated. The error log will be updated based on test results.

16.	The Initial Acceptance Test is the formal process for the QA and development team to test the system functionality using the users developed User Acceptance Test Plans. Problems with the system are noted and managed using the Error Log. Any problems with the test plans are reported to and discussed with the users.

17.	Once development is complete, the Project Manager will declare the entire application ready for formal acceptance testing, and the users will perform the formal Acceptance Test using the user developed Acceptance Test Plans and the User Guides. Users will manage and conduct the Acceptance Testing for a period of 60 calendar days. Problems will be reported to the Project Manager or Lead Analyst for correction as identified. Once corrected, the users will re‑test the applicable software. The Error Log will be used to track all reported problems and resolutions.

18.	Upon successful conclusion of this 60 day acceptance testing, the application software is declared complete and the Maintenance period begins.

Note: At any point in the process, errors may also be reported by those not “formally” involved in testing. These errors are also tracked and managed through closure using the Error Log.

2.2	Data Conversion Test and Acceptance Process

Data Conversion will be tested and accepted using the following strategy:

1.	Development Analyst develops a specific conversion script based on a Data Conversion specification. (The Data Conversion specification is developed based on the relevant portions of the Requirements Document and provides specific details necessary for functional development.) As an integral part of this process, the Development Analyst performs iterative testing and debugging. This development testing is not formally tracked.

2.	The Project Manager and users working on the data conversion effort hand checks the data for conformance with the specification. Errors found starting with this step, and during all subsequent steps, will be entered and resolution tracked in the Error Log.

3.	The Development Analyst corrects any discrepancies and/or problems and re‑submits for data conversion effort testing.

4.	When the scripts are working properly as determined by the data conversion team, the conversion error reports are delivered for review and data cleanup.

5.	The End User Representatives report any errors they find to the staff working on the data conversion effort. These errors are tracked through the Error Log and resolved, as appropriate.

6.	The Development Analyst corrects any discrepancies and/or problems. Updated conversion error reports are re‑submitted to the End User Representatives.

7.	The End User Representatives submit their Acceptance Test Plan for review. Problems with the conversion scripts are noted and managed using the Error Log. Problems with the test plan are reported to and discussed with the users.

8.	Upon completion of the entire application, the Initial Acceptance Test is the process for the QA team to test the system’s functionality using the DEP developed User Acceptance Test Plans. These are executed to verify both the tests and correct script functionality.

9.	Once development is complete for the entire application, the Project Manager will declare the data conversion ready for formal acceptance testing, and the users will perform the formal Acceptance Test using the user developed Acceptance Test Plans. The users will manage and conduct the Acceptance Testing for a period of 60 calendar days. Problems will be reported to the Project Manager or Lead Analyst for correction. Once corrected, the users will re‑test the applicable software. The Error Log will be used to track all reported problems and resolutions.

10.	Upon successful conclusion of this acceptance testing, the data conversion software is declared complete and the Maintenance period begins.

Note: At any point in the process, errors may also be reported by those not “formally” involved in testing. These errors are also tracked and managed through closure using the Error Log.

2.3	Documentation Test and Acceptance Process

Documentation will be tested and accepted as it is developed over the course of the project. Each deliverable document will be provided to users for review and comment, and once agreement is reached on contents, for sign‑off by the users. Once a document is accepted through this sign‑off process, it is subject to update and/or correction to address deficiencies and/or agreed upon changes, as part of the maintenance process. Documentation is not subject to re‑acceptance during the formal 60 day acceptance period for changes other than those described above. Documentation that will be accepted in this manner includes, but is not limited to:

Requirements Specifications

Functional Check Lists (screen and report specifications)

Data Conversion Specifications

User’s Guides

Training Plan

Course Outline and Training Materials

System Test Plan (this document)

Design Report

On‑line help is a special case of documentation, which is provided by linking accepted User’s Guides to the applicable modules. While testing on‑line help will not be part of the formal written ATP, users are expected to test and verify on‑line help functionality as part of the acceptance test process (contents to match the accepted User’s Guide). The help screens and messages will be developed following standards set forth for the project. To test these screens and messages, the designer and QA staff will review the screens and messages to ensure that the text is complete and informative, and that the appropriate standards were followed. The tester will document the results in a given procedure and place them on file with the module.

Errors and discrepancies reported in signed documents and/or on‑line help will be tracked in the Error Log and maintained as a Microsoft Access database.

2.4	Training Test and Acceptance Process

The steps to develop and deliver training are:

a)	Development of a Training Plan

b)	Development of Course Outlines based on the Training Plan.

c)	Development of Training Materials in accordance with the Course Outlines.

d)	Scheduling and delivery of the courses.

Training will be delivered near the end of the project and, ideally, will be scheduled such that it overlaps acceptance testing and start‑up.

The test and acceptance strategy for the training is as follows:

1.	Project Manager or Lead Analyst will work with users to develop a Training Plan that details the courses to be delivered. The training plan will list the courses and include a Course Outline for each. This Training Plan will be subject to testing and acceptance as described for documentation.

2.	Based on the Training Plan for each subset, a Project Team member or designee will develop Training Materials for each course described in the Training Plan. The Training Materials will be subject to testing and acceptance as described for documentation prior to the delivery of the specific course for which they were developed.

3.	Project Team member or designee will deliver the course in accordance with the accepted Course Outline and Training Materials. At the conclusion of each course session (complete course), a Project Team member or designee will distribute a Training Evaluation Questionnaire to the user students. The questionnaire will include review of the course content, course materials, and the effectiveness of the instructor.

4.	Project Team member or designee will prepare and furnish to the users a summary of the evaluations, along with details of any issues which must be addressed and corrective measures which will be instituted.

5.	If the majority of DEP attendees, upon completion of the course, deem it insufficient to provide for DEP self‑support, then the DEP will report to Project Manager the deficiencies in writing for resolution. If the Project Manager does not receive such written notification, then the course will be deemed accepted as delivered.

6.	Training is accepted when all courses detailed in the Training Plan, including any agreed upon modifications to that plan, are complete and accepted.

2.5	Maintenance Test and Acceptance Process

Testing and acceptance of Maintenance will be based on the users review of project records reflecting that:

1.	Project Team will provide maintenance support during the 90 calendar days, beginning upon acceptance, and

2.	Project Team will provide a Help Desk during those same 90 calendar days, and

3.	Project Team will provide a Log of User Requests and updated the application and documentation as authorized by the user’s representative(s).

3.	Test Responsibilities

The responsibilities for testing will be divided between QA members and other members of the Team, with ultimate responsibility for a quality deliverable resting with the Project Manager and Client Project Manager. Specific responsibilities are detailed below for various team members.

3.1	Deliverable Developer

The deliverable developer may be a designer, the DBA, a documentation developer, a development analyst or any other member of the project team. The deliverable developer shall:

Develop the work product in accordance with specifications and standards.

Test that work product to confirm adherence to those specifications and standards.

Perform test activities detailed elsewhere in this System Test Plan.

Correct any deficiencies and/or problems, whether or not they are obvious to others involved in the testing and review process.

Seek help from QA Manager, Project Manager, and/or Client Project Manager as necessary to complete responsibilities.

3.2	QA Manager

The QA Manager and/or designate is responsible for overall quality assurance and test coordination across the team. These responsibilities include assisting team members that are having difficulty with their assignment, and hands‑on correction as needed. Specific QA Manager responsibilities include:

Test each deliverable in accordance with the procedures in this System Test Plan.

Review users’ Acceptance Test Plans for conformance to requirements and functional specifications.

Maintain an Error Log, and assign/track problems to completion.

Work with the other team members to resolve any deficiencies and/or problems.

Work with users to coordinate and review results of Acceptance Testing, and with team members to get problems and modules re‑submitted for re‑test.

Review of test documentation, including detailed user test plans, test data, and results of testing.

Prepare the integration test plan.

Monitor overall test progress and adherence to schedules. Assign schedules to individual module developer.

Maintain a Log List for each module. The Log List will show all actions taken by developers and QA staff during the lifecycle of a module. These actions may be repeated several times, as needed. All actions are recorded on this log.

Escalate any non‑resolved issues to the Project Manager.

3.3	Project Manager

The Project Manager is responsible for technical design and development of the deliverables in accordance with the project plan and that design. Specific QA responsibilities for the Project Manager includes:

Exercise a general supervision over the activities of the QA Managers and developers, and ensure that testing is thorough and complete.

“Spot check” test results to confirm adequacy of test.

Clarify any standards that are unclear and need resolution for QA and the team members.

Insure that the standards are complete and unambiguous for development and QA, and that they conform to user’s standards.

Escalate any non‑resolved issues to the Project Steering Committee.

Approve all test packages.

Assign individuals to appropriate tasks in test plan and ensure sufficient resources are allocated to conduct a quality test.

Ensure that all work is thoroughly tested and that problems and any escalated issues are resolved.

Review problems with users when requested by QA to determine whether the issue is a bug or a potential enhancement.

Coordinate release of completed deliverables to users for acceptance testing.

4.	Specific Testing Procedures and Associated Documentation

4.1	Module Testing

The Technical Checklist is used to validate the module’s technical performance against project standards and conventions, as well as the outline for Unit Testing (in coordination with the module’s specification). The checklist includes performance, navigation, integration, and some access control functions. This checklist does not cover items that have been programmatically set in the Forms Template (e.g., color and font).

The Technical Checklist provides the format to record the results and testing from the Developers Checklist #1, QA Checklist #1, Developers Checklist #2, QA Checklist #2, Developers Checklist Final and QA Checklist Final. When errors are found during the test that don’t “fit” the checklist format, an explanation is appended to the form.

Unit Testing is complete upon completion of the Final checklists. Units (triggers, procedures, and functions) are low‑level sections of code included in a module in order to perform a specific function. The form developer will test units for syntax and some logic errors using tools within the development software. Some of the functional logic of the units can be tested in independently by the developer during the informal testing stage, but any unit code that affects the overall functionality of the module should be included in the module testing phase.

The following are typical of unit test considerations that will be included in the Unit Tests:

Parameter and argument attributes match.

Global variable definitions consistent across form or module.

Constraint passes as arguments.

Format specification matches I / O Statement.

Buffer size matches item size.

Any textual errors in output information.

Improper or inconsistent typing.

Incorrect (misspelled or truncated) variable names.

Inconsistent data types.

Overflow or value errors data values.

QA is responsible to maintain a hardcopy file of completed checklists. Electronic copies are also on the development system (where available).

4.2	Integration Testing

Integration testing is performed to test the functionality of a group of units (modules) that have been combined together into larger segments. The QA team will perform the integration testing.

Integration testing consists of internal integration (menu/module and module/module interfaces), and external integration (interface to outside systems).

During module testing, the functionality of each module was tested. The QA team will validate internal system integration.

The test team also needs to test the system’s interactions with outside systems. All interfaces to systems that exchange data with and are in scope of this project will be identified, and the data transfers will be tested and the results documented.

4.3	Acceptance Testing

The objective of user acceptance testing is to verify whether the application conforms to the signed requirements and specifications. Acceptance Testing will be based on users’ developed plans for each module, including screens, reports, and data conversion. The test plans will include specific data to be used in the test and expected results for each test.

The users will give the Project Manager and QA team a copy of their Acceptance Test Plan during development for review and comment. The Project Manager and QA team will provide one set of comments and discuss them with the users as requested.

Upon completion of the Acceptance Test Plan, the users will place the “final” User Acceptance Test Plan in the shared file directory. Project Manager will use these plans for “initial acceptance testing” prior to release of it to users for formal Acceptance Testing. This “initial acceptance testing” is a final test by the Project Team before declaring the system ready for acceptance.

Once the Project Manager declares the application ready for formal acceptance testing, the users will perform the tests using User Acceptance Test Plans. The users and QA Manager will report problems to the Project Manager and/or Lead Analyst for correction. Once corrected, the users will re‑test the applicable software (may be all or a portion of the application). The Error Log will be used to track all reported problems and resolutions.

The final Acceptance Test must be executed on a system provided by the OIT’s Department of Environmental Protection. The dedicated system must replicate as closely as possible the hardware and software setups of the production systems that will be in place at the time of system conversion. Project Manager will assist the users to establish system configuration requirements prior to the Acceptance Test. The Acceptance Test system must support no other activity during the time of the test.

Appendix A - Glossary of Abbreviations, Acronyms and Terms

Term	Definition
ATP	A formal evaluation of a hardware product performed by the customer to verify that the product is performing according to specifications.
BIT	Bureau of Information Technology
CIO	A Chief Information Officer is the department’s most senior IT manager, responsible and accountable for the Department’s performance in that area.
DBA	One who manages a database. The administrator determines the content, internal structure, and access strategy for a database, defines security and integrity, and monitors performance.
Deliverable	A deliverable is a physical item to be delivered for a project. It may include organization attributes, reports and plans, as well as physical products or objects. It can apply to the project itself, such as a risk management plan, or to a system, such as software code.
DEP	Department of Environmental Protection
I/O	The complementary tasks of gathering data for a computer or a program to work with, and of making the results of the computer’s activities available to the user or to other computer processes. Gathering data is usually done with input devices such as the keyboard and the mouse, while the output is usually made available to the user via the display and the printer. Other data resources, such as disk files and communications ports for the computer, can serve as either input or output devices.
Project Champion (PC)	The Project Champion is the person who is responsible for acting as the project’s owner/delegate. This person approves the project’s deliverables and coordinates integration of the project’s outcomes into the sponsoring organization.
Project Management Body of Knowledge (PMBoK)	PMBoK is the Project Management Institute’s body of knowledge that includes knowledge of proven, traditional practices, which are widely applied, as well as knowledge of innovative and advanced practices, which may have seen more limited use.
Project Management Methodology (PMM)	The system of practices, techniques, procedures, and rules used by those within the profession of Project Management.
Project Management Office (PMO)	The Project Management Office is an organized body or entity assigned various responsibilities related to the centralized and coordinated management of those projects under its domain. The responsibilities of a PMO can range from providing project management support functions to actually being responsible for then direct management of a project.
Project Manager (PM)	Project Manager. The person assigned by the performing organization to achieve the project objectives.
Project Sponsor (PS)	The person or group that provides the financial resources, in cash or in kind, for the project.
Project Stakeholder	Persons and organizations such as customers, sponsors, performing organization and the public, that are actively involved in the project, or whose interests may be positively or negatively affected by execution of completion of the project. They may also exert influence over the project and its deliverables.
QA	A system of procedures carried out to ensure that a product or a system adheres or conforms to established standards. Also called: quality control.
SDM	The process of defining, designing, testing, and implementing a new system using a proven methodology.
Subject Matter Expert (SME)	A Subject Matter Expert (SME) is a person who is viewed as an expert in a particular field. This person can be a software expert who knows how a software application works or a business expert who understands business forces and requirements. An SME designation is not particular to a discipline; an SME is someone to whom a project manager can go to for accurate information.

References:

A Guide to the Project Management Body of Knowledge (PMBOK® Guide) Third Edition

Governor’s Office of Administration

http://www.oit.state.pa.us/oaoit/cwp/view.asp?a=671&q=194302

Microsoft Computer Dictionary, Fifth Edition.

Commonwealth of Pennsylvania	4.A_STP_Final

Department of Environmental Protection

System Test Plan

<Enter Project Name Here>

16	12/21/2005

<Location of Project "file path">

Commonwealth of Pennsylvania

mahockenbe

2005-10-13

strotman