Management Dynamics, Inc

Management Dynamics, Inc.

Testing Policies and Procedures

Objectives:

Our objective is to ensure the highest possible quality for each release. Our software is our product. Therefore, all efforts to assure the quality of our product are paramount to our continued success as a company and maintaining our market leadership position.

Why we need testing policies and procedures:

These policies and procedures are intended to help us organize and manage the activities necessary for quality assurance. The goal is to make the effort easier and more efficient. As we implement these procedures we may continue to discover ways to refine them and make them serve us better. Any suggestions are always welcome.

Testing phases and their objectives:

Our objective is to limit the scope of each testing phase and thereby make each one easier to manage. Think of the testing as being done in layers. As we find and correct problems in one layer, we pave the way for more effective and thorough testing in the next layer.

We have defined three distinct testing phases. Although Phase One and Phase Two may occasionally overlap, each phase has specific objectives. Ideally, thorough Unit testing by the engineer should catch errors that would otherwise be discovered in Phase One and Phase Two. Phase Three testing begins only after all issues from Phase One and Phase Two are resolved and the release is “locked down”.

Phase One: Mechanical / GUI Testing

-Program abort or other unmanaged errors

-Functions (e.g. Find, Filter) operate normally and consistently

-GUI interface functions normally

Phase Two: Functional Testing

-The project requirements function as designed and documented

-Production program functions are not disrupted (unless intended)

-Database additions, changes and integrity are verified

-Questions and possible future enhancements are also documented

Phase Three: System / Regression Testing

-Verify that no unintended or unpredicted changes have been caused to other Production system processes or to the database

-This testing phase must necessarily include verification of all system life cycle processes and functions, from creation, through release, amendment and expiration

-Normal function of all system components, such as Tariffs, Contracts, Commodities, TLIs, and Rate Tables must be verified

-Questions and possible future enhancements are also documented

Testing Assignments and Resources:

Phase One mechanical / GUI program testing is assigned to each engineer in an Excel worksheet. The worksheet is also used as a checklist to document each program function tested. The worksheet is named for each engineer and is found on the ‘Development on “netsrv”’ network drive in the Issues folder / current release subfolder (e.g. IssuesV83).

Resources:

Documentation of standards and common functions may be found on the intranet link titled ‘Standards Documentation’.

Phase Two functional testing is assigned by task and will pair the developing engineer with a CSM or other manager when available. When a CSM or manager is not available, another engineer may be assigned. Parallel testing with the Production system is required.

Resources:

Design documents should be available for most, if not all, tasks to guide the testing objectives. These documents may be found in the current release ‘Documentation’ folder on the ‘Development on “netsrv”’ network drive, in a subfolder named for each current release task. The document should provide a non-technical description of the project and clearly explain the expected results. It should also identify all programs, display panels and database tables affected by the project.
The Navigation Model consists of Excel worksheets that show the navigation path or calling relationship between menus and program names/descriptions. It may be found on the intranet link titled ‘Testing Documentation’. This is useful for finding and testing all navigation paths for a given program to be tested.
The Data Model shows the relationship between database tables. One version shows only the table names/descriptions and relationships. The other version also shows the data elements and key names. It has been developed with the Erwin data-modeling tool and may be found on the ‘Development on “netsrv”’ network drive, in a folder named ERWMM. A printed hard copy of each is available for ease of reference. This is useful for finding tables that may have been affected by any database changes and verifying database integrity.
Future resource to be acquired: A tool that integrates an application dictionary over modules, programs, database and all other related objects, and, most importantly, provides graphical views of data relationships, program relationships, and program to database relationships. Pathfinder by Hawkeye is one such tool for the iSeries.

Phase Three system / regression testing assignments are based on knowledge and experience with specific parts of the system. Where possible, this testing phase will also pair engineers with Customer Service Managers or others with good system knowledge, thus providing an additional training opportunity. This requires an in-depth understanding of how the current release modifications are intended to affect the system functions assigned for testing. Parallel testing with the Production system is required.

Resources:

To be developed: Test scripts/procedures to ensure that all relevant system processes are tested and verified.
To be gathered and organized: System documentation explaining how to perform system functions and what results should be expected.
Possible future resource to be acquired: A tool that automates testing and assists with evaluation of results. One such tool to evaluate is ‘TestBench for iSeries’ by Original Software, which is integrated with the Aldon Computer Group’s Change Management System (CMS).

Documenting Test Issues:

Issues from all testing phases are first logged into an Excel worksheet.

The worksheet is named for each engineer and is found in the Issues folder / current release subfolder. For example, my Issues worksheet for Release 8.3 testing is found via Windows Explorer on Development on ‘netsrv’ / Issues / IssuesV83 / Test Issues / S Huete Test Issues. For Customer Service Managers the Issues worksheets are found on Development on ‘netsrv’ / 8.3 Testing / (CSM name).

The first four worksheet columns must always be entered so as to provide a unique value for collection into a master worksheet. Note that Column C (‘Issue’) is a simply a sequential Issue number beginning with the value of 1. Column D (‘Reported By’) should always contain the name of the tester, always spelled exactly the same.

Note that Column N (‘MPP’) should not be used by the tester. This column is used when the issue is collected, after which the value is ‘Yes’. Likewise, the tester should not enter any data in Columns H through M, as they are used only in the Master Issue worksheet.

To assist in prioritizing and assigning issues logged, the tester should determine whether the issue was introduced in the current release or not. Column O of the Issue worksheet (‘Production’) is used to log this ‘Yes’ or leave it blank. An easy way to determine that the issue was introduced by the current release is to check whether or not the program object exists yet in the REV8PROD production program library. If it does not, the tester would leave the Production column blank for the program. If the program object does exist in REV8PROD then the tester must also test for the issue in the Production system using the same test data. If the issue is duplicated with the Production version of the program then the tester would enter ‘Yes’ in the Production column.

Each issue logged must also have a supporting Word document to provide more details and assist in the duplication and correction of the issue. The naming convention is initials_mmddyy_hhmm.doc (e.g. SH_103102_1530.doc).

This document name is referenced in Column B (‘Document’) of the issue worksheet. More than one issue may be documented in one Word document if desired. A template for this document titled ‘Issue Doc Template’ is being developed and will be found in folder IssuesV83 / A Testing Procedures. All issue documents should be developed from this template and placed in the root of the current Issues folder (e.g. IssuesV83). It is imperative that the naming convention for these documents be followed for ease of reference.

In addition to screen captures, the tester should collect and document relevant information from the Display Job panel into the Word document (see template). This includes the complete Call Stack and any error messages from the job log. Any program dump or other relevant spool files should be referenced in the document and moved to Output Queue REL83TEST. For assistance with displaying the emulator panel and entering the ‘DSPJOB’ command onto a system command line, see the document titled ‘Retrieve Program Name from Emulator’ in the folder IssuesV83 / A Testing Procedures.

Documenting Test Questions and Enhancements:

Both questions and enhancements may be entered into a Word document and placed in the root of the current Issues folder (e.g. IssuesV83). The naming convention for this document is ‘TEST_task name_mmddyy.doc’. The tester is encouraged to document any questions about program test results if unable to determine whether or not they are valid. Also, any ideas for future enhancements may be documented. These documents will be evaluated for possible issues and enhancements during the testing of the release.

For paired testing during Phase Two, it may be helpful to first create and open this document, using it as a workspace to collect all issues, questions and comments. Later, any issues may be entered into the Issues worksheet and relevant documentation cut and pasted into the supporting Word document for the Issue. Then those issues may be purged from this document, leaving only the unresolved questions and enhancements.

Collecting Documented Test Issues:

Issues logged into each worksheet are collected periodically into a master worksheet. When each issue is collected, Column N (‘MPP’) is populated in the tester’s worksheet with the value ‘Yes’.

Assigning Collected Test Issues for Resolution:

Management assigns collected issues for resolution by engineers based upon whether or not the issue is related to the current release and the issue’s priority. Where possible, issues affecting the same program objects will be assigned together to the same engineer for more efficient work.

Engineers should monitor the Excel worksheet used for assignments. Choose intranet link ‘Testing Documentation’ and then click on the link titled ‘Rate Search Test Status.xls’. The Scoreboard worksheet shows total issues assigned to each engineer. The Issues worksheet contains each issue assigned. Filter on Column H ‘Assigned To’ to display only your own assigned issues. You may also filter on Column I ‘Fixed By’ to show only issues that are “Open”, or not yet logged as resolved.

An effort will be made to balance the engineers’ time spent working on testing assignments and resolving issues assigned for resolution. The goal is to keep testing activities moving forward while issues are being resolved. Non-critical and low priority issues may be deferred for assignment while engineers are testing. For this to create more efficiency, it is important that testing activities be approached with the same sense of urgency as assigned issues.

Verifying Test Issues after Resolution:

The tester who logged the issue is expected to verify that the issue has been satisfactorily resolved. After the corrected objects have been promoted to the Test environment, the engineer who resolved the issue should send an email the tester. The Issue worksheet Column D (‘Reported By’) contains the name of the tester who should receive the email, triggering the verification activity.

Summary:

Our desire is to make the testing process easier and more productive while assuring the highest possible quality. When teamed with others for testing, it is a good opportunity to share system knowledge and ask questions. Productive testing should help all of us to grow our knowledge of the system and add increased value through our experience.

Revised 11/1/2018Page 1