Increased Reliability and Availability

• Increased reliability and availability
• Reduced IT management burden
• Enhanced security and scalability
• Increased responsiveness to patients
• Reduced costs

Situation

When patients with complex and advanced-stage cancer need a dedicated team to join them in their fight, many turn to Cancer Treatment Centers of America (CTCA) for comprehensive cancer care. CTCA searches for and provides innovative therapies to heal the whole person, improve quality of life, and restore hope. The organization’s cancer hospitals and clinics include facilities in Philadelphia, Pennsylvania; Zion, Illinois; Tulsa, Oklahoma; Goodyear, Arizona; and Seattle, Washington. It also plans to open a hospital in Newnan, Georgia, in 2012.

To ensure that CTCA provides the highest levels of patient care, every aspect of CTCA needs to run efficiently, including its IT infrastructure. One of the IT systems that employees depend on most is email, which they use to exchange clinical and operational information, communicate with current and prospective patients, schedule meetings, and more. “Our messaging system is used very heavily, both for widespread communications and impromptu collaboration by nurses, doctors, and other employees,” says Eric Sato, Director of Infrastructure for Information Technology at Cancer Treatment Centers of America. “For example, our oncology information specialists constantly send email to potential patients, often with scanned documents attached.”

“We definitely consider messaging a mission-critical, tier-one application,” adds Scott Johnson, Data Center Manager at Cancer Treatment Centers of America. “On the IT side, we do a lot of alerting through email, such as notifying employees of planned system downtime. On the treatment side, our doctors and nurses definitely use email to communicate about and with CTCA patients.”

For some time, CTCA had relied on successive versions of Microsoft Exchange Server to satisfy its messaging needs. By 2010, CTCA was quickly outgrowing its messaging environment, which was based on Microsoft Exchange Server 2003, and recognized that the environment could not meet some of its emerging needs, such as support for mobile workers, encryption consistent with regulations under the Health Insurance Portability and Accountability Act of 1996 [HIPAA], and comprehensive disaster recovery.

Reliability Concerns

The existing CTCA messaging environment was becoming unreliable, and it also had never been set up to provide adequate redundancy, which compounded the problem. “With no redundancy setup, we had no protection against users experiencing downtime,” says Victor Sanchez, Senior Server Administrator for Information Technology at Cancer Treatment Centers of America. In fact, the IT team avoided making software updates for more than a year because the server setup was so difficult to manage. “We tried not to touch the messaging environment,” continues Sanchez. “We did not want to risk system downtime in our number-one means of communication.”

The organization needed to set up a more consistent email recovery mechanism to address the outages and unreliable email delivery that employees experienced. Space to store records of all email activities was limited. If the accumulated records were not removed promptly, it would negatively affect system performance. “When the drives filled up, employees could go without email backup for several days, and two or three times, the system went down entirely,” says Sanchez. Certain IT staff members had to closely monitor the system, clearing the backup drives a minimum of every two days to avoid downtime.

In addition, employees had to accommodate CTCA mailbox size limitations, which typically meant dealing with the hassles of using .pst files to store and retrieve their email. For example, one CTCA employee had a 2-gigabyte mailbox and a 20-gigabyte .pst file, which slowed the performance of Microsoft Office Outlook 2003 messaging and collaboration client, which the organization used on its workstations at the time. Using .pst files was not only impractical; it was also risky because there was no backup for those stored messages.

The CTCA IT team dedicated a tremendous amount of time to maintaining the Exchange Server 2003 infrastructure and helping employees deal with issues. “From a support perspective, it took valuable time to restore a mailbox or an individual message because the entire mail store for that particular user had to be restored,” says Sato. “That inefficiency affected employees as well, who had to wait without email while the IT team resolved the issue.”

The Need for Disaster Recovery Capabilities

The previous messaging system design at CTCA had resulted in the lack of a proper disaster recovery environment. “As CTCA started to expand, we sought to ensure high availability for all employees, which included gaining more flexibility across multiple data centers,” says Sato. “We knew that we needed to upgrade our messaging system to bring us closer to our goal of maintaining 100 percent messaging-system uptime and availability and also support our projected growth.”

The organization’s previous messaging environment consisted of stand-alone production servers located in Schaumburg and Zion, Illinois. The CTCA backup servers were in Zion, Illinois, and had to be manually brought online in the event of an outage. “Our traditional setup did not have any disaster recovery solution in place, other than manually restoring from backup to an alternate Exchange Server and mounting the mail datastore,” says Johnson. “This was a cumbersome process that was very time-consuming and disruptive. We needed something that would better prepare CTCA for emergency situations.”

Support for More Flexible Use

CTCA also wanted to boost workforce efficiency by making it easier for employees to gain access to email from home and elsewhere. Physicians, administrators, and other employees encountered issues when trying to use smartphones for work; long delays and difficulties when synchronizing calendars and contacts among their various computers and mobile devices were common. “CTCA doesn’t follow a typical 9-to-5, five-days-a-week work schedule, and as smartphones grew in popularity, employees’ expectations of being able to work from anywhere also rose,” says Sato. “They wanted straightforward mobile communication abilities and easier remote access to their email.”

CTCA doctors, nurses, and others had to go through an often cumbersome, time-consuming process involving the use of a virtual private network (VPN) to check their email remotely. CTCA had set up the messaging infrastructure so that some employees could use Microsoft Office Outlook Web Access while on-site—to send and receive email messages from a hospital’s mobile workstation, for example—but not outside the organization’s firewall.

The organization also wanted its Mac users, severalCTCA executives among them, to have an easier time accessing email. “We wanted to streamline the messaging experience for all employees so that they could make even more effective use of their time,” says Sato.

Solution

CTCA determined that the time had come to find a new messaging system, and it looked no further than Microsoft Exchange Server 2010. “CTCA has a very strong partnership with Microsoft. We use a number of its enterprise solutions, and many of our engineers have expertise and certifications in Microsoft technologies, so it did not make sense for us to go with anything other than Exchange Server 2010 for our email,” says Sato. “We were especially excited to take advantage of the Database Availability Groups and the opportunity to split our server resources among multiple data centers to give us a highly available environment.” Database Availability Groups are groups of up to 16 mailbox servers that host a set of databases and provide automatic database-level recovery from failures that affect individual servers or databases.

CTCA wanted to ensure that it designed the optimal messaging solution, so it interviewed multiple potential partners and selected New Resources Consulting, a member of the Microsoft Partner Network, to provide guidance during the Exchange Server 2010 implementation. “Our New Resources Consulting expert already had conducted more than 20 Exchange Server 2010 upgrade projects. He was extremely knowledgeable and highly organized, so we felt good about our choice,” says Sanchez.

In December 2010, CTCA began working with both New Resources Consulting and a Microsoft technical specialist to make key decisions, draw up a project plan, and follow Microsoft best practices. The consultant from New Resources Consulting prepared the organization’s Active Directory service structure and helped with troubleshooting. “We encountered some problems with command lines in Exchange Server 2010, but our consultant knew exactly what to do,” says Sanchez. The consultant used the Windows PowerShell command-line interface to resolve the issues. New Resources Consulting then worked with CTCA IT staff members to transfer his knowledge to them, at which point they built the servers and began to move mailboxes to the new system.

CTCA upgraded 100 to 200 user accounts per day to the Exchange Server 2010 environment. “We chose to gradually move email accounts to the new solution because we were worried that our help-desk staff could not support the number of calls that might result from an overnight upgrade,” explains Sanchez. “We expected 20 percent of employees to need help initially using Exchange Server 2010, but only 2 percent ended up calling for assistance.” Even during the transition, when some mailboxes had been upgraded and others were still on the old system, employees had no difficulty with email communication between systems.

As of December 2011, CTCA has moved 90 percent of its mailboxes to the new environment. IT staff members are checking the remaining accounts to be sure that they are no longer in use before decommissioning the remaining two servers running Exchange Server 2003.

Part of the project also involved establishing greater redundancy in the organization’s messaging environment, to help with availability and disaster recovery. The previous infrastructure had included four servers running Exchange Server 2003 in Schaumburg and two more in Zion. Each ran as a production mail service, and there was limited disaster recovery capability in place. Now the infrastructure includes a primary data center in Madison, Wisconsin, where the organization has four physical servers and two virtual ones, and a disaster recovery setup in Schaumburg with two physical and two virtual servers. “We now have a dynamic messaging environment,” says Sanchez. “If one server fails, service automatically gets transferred to our facilities in Schaumburg or to a different server in Madison, so employees have continual access to their email.”

CTCA also switched from using a storage area network (SAN) to using local, direct-attached storage that uses Exchange 2010 to replicate data from site to site, which makes it faster and easier to shift accounts between locations.

As another part of the upgrade project, CTCA consolidated seven voice over Internet Protocol (VoIP) systems into just one and integrated that consolidated system with its Exchange Server 2010 solution, to give employees the ability to receive voicemail messages as email messages and vice versa.

In addition, the organization now uses the Microsoft Office Outlook Web Access web client to minimize the amount of software on its shared workstations, such as those used by mobile nurses in CTCA hospitals. The CTCA IT team created generic user accounts for each mobile computer so that the computer remains logged on at all times and can be shared by nurses for convenient access to information and key applications. If nurses need to get to their specific accounts, they can quickly launch Outlook Web Access from the browser and enter their credentials to ask a doctor or other healthcare worker key questions and receive rapid responses via email.

Benefits

CTCA successfully designed and implemented a robust messaging infrastructure that operates at the same high level at which the rest of its systems function. “End users and IT staff no longer have to deal with messaging system downtime. Our physicians and other employees can quickly and easily access their email and voicemail messages from wherever they happen to be, using whatever devices they choose. And overall worker efficiency and mobility have increased,” says Sato. “Since we moved to Exchange Server 2010, we definitely are benefitting from using the range of opportunities to enhance messaging system reliability, security, and productivity.”

Increased Reliability and Availability

Because CTCA now has an up-to-date messaging system with redundancy built into the architecture, employees enjoy greater uptime. “Since moving to the Exchange Server 2010 messaging environment, I no longer am plagued by my former daily worries about the system going down and wreaking havoc on hospital communications,” says Sanchez. “IT staff members have no more dread—if there’s an issue, the server automatically switches employee accounts to another of our servers with no interruption of service. We count it as a huge measure of success that there has been no messaging system downtime since the upgrade to Exchange Server 2010.”

Employees throughout CTCA can trust that messages will get delivered as soon as they send them, and they have confidence that they are receiving their email in a timely manner. And, in case of catastrophe, the IT team has a fully operational disaster recovery infrastructure that will continue to support “business as usual.” “If CTCA experienced a disaster, a crucial facet of our disaster recovery plan is to use email to send out vital information to our hospitals,” says Sato. “So having Exchange Server 2010 up and active in multiple locations simultaneously is extremely important.”

Reduced Messaging System Management by 75 Percent

Whereas in the past, IT staff members had to carefully watch the organization’s messaging servers for issues, they now can dedicate themselves to more proactive IT tasks. “Bringing in the capabilities of Exchange Server 2010 has made it possible to extend the reach of our existing IT team,” says Sato. “We keep a lean staff, so having fewer support issues on a daily basis and the ability to resolve them more quickly means that IT staff members have more time to tackle other, more strategic projects and initiatives.”

When difficulties do arise, the IT team is able to resolve them more quickly than it could in the past, due to the use of the Windows PowerShell command-line interface for scripting management changes. For example, an employee accidentally sent a particular email message to every CTCA mailbox, and the IT team identified a way to use a single Windows PowerShell command to delete the message from every account. “We estimate that we will be able to cut 75 percent of the time we currently spend supporting our Exchange Server environment,” says Sanchez. “Plus, we’re able to deliver faster time-to-resolution for email-related issues.”

Enhanced Security and Scalability

Although CTCA uses a third-party tool to encrypt email messages, it now has an additional safeguard for the vast amounts of email sent and received among Exchange Server 2010 users, both within and outside of the organization. “With the additional layer of encryption from Exchange Server 2010, we’ve gained more than peace of mind about the security of our messages,” says Sato. “We have bolstered our ability to comply with regulatory requirements, such as those in HIPAA, which mandate high levels of protection when sending and receiving patient information.” CTCA implemented Transport Layer Security and established policies to place mail accounts in legal hold for electronic discovery situations that may arise from time to time.

In addition, CTCA can conduct regular software updates because the IT team is no longer worried that doing so will result in unwanted downtime for employees. “We now do regular updates every two months, as opposed to occasional updates about every two years, which means that we have a far healthier, more secure system,” says Sanchez. CTCA also gained more efficient management of its mobile environment and therefore optimized security over the organization’s information. For example, when employees leave CTCA, IT staff can immediately disable their user accounts and wipe all email from their mobile devices.