EVITALS PROJECT

Independent Verification and Validation

FinalReport – Including Activity forAugust– October,2007

Submitted November15, 2007

Submitted by

10555 Montgomery Northeast

Building 2 Suite 110

Albuquerque, NM 87111

“Quality is the Difference”

New Mexico Department of Health

EVITALS Project

Final IV&V Report – August/November 2007

Executive Summary:

This Independent Verification and Validation(IV&V) report for the EVitals project provides a final updateon the statusof this very special and important project. It also provides final observations of the project in its entirety.

The project team has been successful in implementing the birth and death registration and issuance system. The project team also implemented an interface to the Social Security Administration, and addedan unscheduled task to the plan for the interface for Office of the Medical Investigator (OMI). The birth record conversion was officially removed from the EVitals project via a Change Order, but the Project Manager, Bette Perry was retained to continue to coordinate the death record data conversion that was completed November 11, 2007. This completed every scheduled task for the EVitals project.

This report will provide a summation of what should be considered a “Model Project.” Key factors for its success are simple:

  • There was a strong, interested and active Steering Committee who took their role seriously and stepped in to expedite authorizations and provide resolution when requested. Additionally, they had knowledge of processes in order to provide guidance to the team on how to proceed in various situations.
  • There was an experienced, excellent Project Manager who came with a comprehensive set of project management ‘tools’ in order to effectively guide, develop, monitor and report on the project.
  • There was an extremely supportive Bureau Management team who was directly involved in every aspect of implementing the system. Their willingness to actively participate in such a positive manner served as an exemplary model for their project staff.
  • There were team leaders and members who were the most consistently dedicated people ever observed, and who gave 150% effort to their every task.
  • There was a tremendous amount of support from ITSD who stepped in to assist with problems at critical times and who also provided excellent support for testing activity, Help Desk activity, network, hardware setup and conversion development, just to name a few.
  • The selected vendor had a good product in its ‘youth’ and was willing to dedicate as many staff as it took to assure the software was customized, installed and functional in a timely manner.The exception was the DTS load functionalitywhich was several weeks overdue near the end of the project. This in particular caused the death record data conversion to run slightly over schedule. Earlier versions were more ‘buggy’ than was preferred, but there finally came a point where there were only minor issues to fix in version patches. The Birth Module software had significantly fewer issues than the Death Module. Remarkably, considering the complexity of edits and rules the system contains, the vendor delivered a good, basic product that was made better through the testing and QA/QC process, issue tracking and additional enhancements provided at no cost to the State.
  • Last, the working relationship and cooperation between all parties was admirable, professional and sprinkled with humor. The team dynamics were the most positive ever observed.

The challenges on the EVitals project were external to the project team itself. The primary reason the project went to the final moments was the arduous procurement process for every item purchased for the project, e.g., hardware, software andcontract staffing. This was further exacerbated by the funding issues that were at the root of many challenges. The difference between the estimated cost of the project and allowable funding was one million dollars. This caused delays that impacted deadlines and resulted in the loss of grant funds earmarked for items on the project. Team members had to serve dual roles by integrating the project demands into their full-time responsibilities resulting in a staff that spent two years working extra hours.

In addition, the SHARE implementation schedule was an unknown when the Evitals Project Plan was developed. This major change impacted some acquisitions as did other procurement-related issues that existed prior to SHARE and are expected to continue.A case in point is the fact that a software invoice approved and submitted in Julywas not paid to the vendor until October. The vendor met their obligations and should have been compensated in a timely manner. This is a systemic issue for the entire State and these observations are not meantto pass judgment on any particular area. Any process requiring more than four signatures will take additional time with a greater risk of documentation getting lost between stops (this has been observed as quite common across all departments, not just DOH.)

Observations on weaknesses include the following, but are not limited to this list:

  • There was a lack of general communication and coordination of processes.
  • There was a lack of procurement procedures and input onproper contract vehicle/templates for the procurement activity. It was also observed that a lack of communication regarding procedure or template changes resulted in re-initiating procurement processing. Since this process crosses departmental lines (individual departments, DFA, DoIT) and is complicated at best, the current solution for project planners is to build in extremely long periods of time for procurement activities.With the current process it is recommended that one should begin the procurement four to six months in advance of when the item/service is needed or up for renewal.
  • System performance issues were experienced due to network and infrastructure issues. The DOH network is saturated. The ITSD staff has done an admirable job with ‘band-aids and baling wire’, but improvements will be ongoing and expensive for some time to come. Keep in mind that there have been many installations of sophisticated systems within DOH that rely heavily on real-time data transfers with increased communication demandsacross all points in the state and beyond.While not knowing the ultimate, long-term goals of DOH or the State, cost allocation for ITSD is a positive move and will hopefully result in additional, adequate funding to upgrade and expand their communications network and hardware.

A detailed discussion by the project team regarding lessons learned is provided in Attachment D. This document is very important in that it is an honest and complete, retrospective review of the accomplishments, strengths and challenges the project faced and states it better than could have been done by this reviewer. Please refer to that document for a more detailed analysis.

Despite all obstacles the project team persevered and delivered a high quality product of great benefit to the citizens of New Mexico. One can now obtain birth and death certificates at sixteen offices with an additional seven offices issuing the certificates for funeral homes. This is added with the other interfaces that were developed, and the major death record conversion was recently completed.

Lastly, Bency & Associates is pleased to have been a part of this most important and successful system implementation. Without question this project has been one of the best managed and documented projects this IV&V Specialist has ever had the privilege to review, which helped guide it through all the challenges and roadblocks that came along the project path.It is with great pleasure (and a twinge of sadness) that I provide this final report on a model project, thus fulfilling the IV&V agreement.

Final Summarized Analysis of Project Status

In August, the focus was on completing the OMI interface that was implemented without any major problems, and the death record conversionthat spanned all weekends for processing through mid-November. The good news is that final reports on the conversion are very positive, and all records are now contained in the DAVE system.

For this final review, there is not much improvement to report in the remediation of system performance issues in the production environment, though purchase of parts and needed items is underway. The Bureau continues to host the production application, and it is not possible at this time to turn on replication. The good news is ITSD has set up to issue self-signed certification until the SSL feature could be purchased. The final word is that the SSL is in and is scheduled for installation within the next week.

For the final status update, the discussion of details for accomplishments and technical issues will be provided in the IV&V Analysis of Project Status section below. This information provides who, how, and where project information resides and responsibilities for each as the project closeout commences.

Risks Noted:

The final risk items for the EVitals project now dwindle to maintenance support funding.

Activities Completed this Period
  • Attended the EVITALS Project Steering Committee meeting for August, 2007.
  • Reviewed minutes for six project status meetings.
  • Attended the final Steering Committee meeting held on November 9, 2007. (Many thanks to Dr. Armitage for her specialapple cider and fancy plastic glasses for the ‘final toast’.)
  • Reviewed the Steering Committee meeting minutes.
  • Reviewed fiveISD project status reports provided by the PM.
  • Reviewed the project budget and dashboard submitted to the OCIO for August and September.
  • Reviewed the ITSD Application Support plan.
  • Reviewed the Lessons Learned document provided by the project team.
  • Reviewed the Conversion meeting notes and Conversion test plan, issue logs and status minutes from August through mid-November.
  • Prepared the final monthly report.
  • Participated in multiple update sessions on project status with the Project Manager throughout this final reporting period.
Significant Findings and Pertinent Recommendations:

New Findings and Recommendations

There were no new findings or issues during this reporting period.

Analysis of Budget Status

Table 1: Budget Status as of September 10, 2007 taken from project lifetime for the final budget review.

Total
Budget / Project To-Date / Estimate to Complete
$1,800,000 / $1,536,636 / $263,364

Funds have been encumbered from last fiscal year in order to pay license/maintenance costs. Additionally, Bette Perry will be updating the expenditures with all final invoices in the closeout report for the project.

Statistics on Prior Findings and Recommendations

Table 2 provides a statistical summary of previous findings/issues (excluding current period findings). Findings/issues and recommendations from prior reporting periods are contained in AttachmentD: IV&V Issues Log.

Table 2: Statistical Summary of Prior Findings/Issues

Potential Impact / # Open / # Closed
# Severe / 1 / 7
# Moderate/Mild / 1 / 10

The open issues are few and should be monitored post-project until resolved.

  • DOH Network/EVitals server configuration (system redundancy)
  • DOH toolset for stress testing systems prior to implementation
  • Adequate funding for maintenance.

END of EXECUTIVE SUMMARY

IV&V Periodic Review:

Analysis of Project Status

The project team made great strides during this final reporting period to meet all project objectives. It was a challenge as issues arose in the conversion process and contributed to the conversion going to the deadline. Some issues were systems related such as data base limits set too lowor not properly adjusting automatically for the volume of records being loaded. Other issues were procedural such as the invoking of applications in the conversion process (‘triggers’ that cause the next application in the processing flow to begin or calls to external modules to begin). However, all of these issues were resolved, and it was simply a matter of continuing the load process first to the conversion data base while the new version is installed instead of directly to the production database, which is the ultimate goal. All was completed smoothly, and the final issues from an IV&V perspective have been resolved with the exception of system fail-over/redundancy and stress test mechanisms. Following is the final update/disposition of all project related tasks as provided by Bette Perry for the most complete status possible while still meeting the report deliverable deadline.

Status of Project Deliverables

The EVitals Project has successfully implemented the birth and death registration and issuance system, an interface for Social Security Administration, and an additional project deliverable, an interface between the DAVE™ application and the Office of the Medical Investigator’s X-Files system.

All major tasks in the scope of workhave been completed. The conversion of legacy birth data was formally removed from the scope of work several months ago via a Change Request.

Data Conversion

As of this writing, approximately 100% of the legacy death data has been converted in the NM Production environment.

There are two primary reasons that the death data conversion was not completed untilmid-November:

1) Late delivery by the vendor of the DTS load functionality, which was promised before the end of July, was received approximately the third week of August. The vendor agreed to develop this back-end load functionality after it was determined that the DAVE application job scheduler could not handle the large file sizes.

2) Issues (database not expanding automatically, and some application trigger issues) arose after the conversion of several loads into the production database. These issues were resolved.

Due to critical bug fixes in DAVE™ release 7.1.4 and new birth module functionality in 7.2, these releases had to be in the production environment by the end of September. Since the window to complete the data conversion by the end of September was missed, the plan is to temporarily suspend conversion loads into the production environment until 7.1.4 and 7.2 are deployed into production. These releases were deployed to production at the end of September.

Contingencies to Complete Data Conversion

Kemtah Contract 4445 for project management services for the EVitals Project could not be renewed as Kemtah was not awarded a new State Pricing Agreement (SPA) for project management. Two contingency plans were developed to continue the conversion of legacy death data and close out the project:

1) A Purchase Order for staff augmentation for Bette Perry was routed through the system. This PO would be through another vendor who does have an SPA for project management.

2) If the staff augmentation purchase order was not approved, Debbie Dickerson was prepared to place another DOH project management resource with the EVitals Project to complete the data conversion.

Contingency #1 was preferred and utilized with Bette remaining until the end of the project.

In order to maintain momentum on the data conversion, the EVitals Project continued to do conversion loading in the conversion database against release 7.1.4/7.2 until the project determined that the system was ready to resume loads to the production environment.

Status of Payment of Vendor Invoices

A purchase requisition for VitalChek Network, Inc. in the amount $190,743.00 was submitted on July 20, 2007. This purchase order is being funded from carryover funds from the computer enhancement budget. As of this September, a purchase order had not been issued. This purchase requisition is for payment of the remaining FY08 deliverables for VitalChek. The vendor cannot submit invoices until the purchase order is completed. (Reference Contract 05/665.0009.0030 A1, Vendor #42160). This issue was finally resolved in October.

Kemtah submitted a final invoice for project management services for the month of September. The purchase order for these services is in place. Reference Contract #4445, Vendor #49201

Bency will submit a final invoice for IV&V services. The purchase order for these services is in place. Reference Contract #2041, Vendor #55484.

Status of Technical Issues

SSL - Permanent SSL solution – A temporary solution (self-signed certificates) was implemented prior to the rollout of the birth module in July. Permanent SSL certificates have been received from the vendor and are awaiting installation by DOH ITSD in November.

Replication between fail-over and production – Replication between the fail-over and production environments was turned off early in the EVitals implementation when the users began experiencing severe system performance issues. Due to the fact that there was not a timely resolution to the performance issues in the Runnels Building, a temporary production environment was established in the Vital Records Building. The former main production environment in the Runnels Building was taken off-line pending resolution of the performance issues. System performance in the Vital Records Building is far superior to that experienced when the production environment was in the Runnels Building. A third environment, testing/training/conversion was brought up in the Runnels Building. (These environments, testing/training/ conversion, experience the same slow response time as the former main production environment experienced in the Runnels Building.)

An update from Rod Skiver was received on September 25,2007. The parts required for the infrastructure work in the Runnels Building have been received from the vendor.

Ultimately, at some point in time the main production environment will be moved to the Simms Building. Several months ago when the project began experiencing severe system performance issues, ITSD was going to move the servers to the Simmsbuilding. Unfortunately, the power was not adequate for the Dell 6850 database servers. Per information from Rod Skiver on September 25, 2007, a power conversion is scheduled for the Simms Building in late October. No additional servers will be moved to the Simms building until that work is done. There was no final status as to whether this was accomplished.