Technology Consultancy Speeds Patch Deployment by 4,000 Percent

Situation

Avanade is the leading global technology integrator specializing in the Microsoft® enterprise environment. Avanade’s approximately 2,200 employees in 30 locations around the world help customers maximize their IT investments and create comprehensive solutions that drive business results.

After several years of rapid growth, Avanade found itself straining to keep its growing collection of servers updated with the latest security patches. Regular patches containing critical updates to Microsoft Windows® operating systems, the Microsoft Internet Explorer browser, Microsoft SQL ServerTM, and other key server applications were eating up more and more IT staff time.

“Server patching involved putting many IT projects on hold while we gathered the majority of our IT staff into conference rooms, where we would frantically patch our production servers,” says Virginia Knight, Project Manager at Avanade.

Euphemistically called “patch parties,” these sessions would occur either after hours (from 5:00 P.M. to 10:00 P.M. or midnight) or during the workday for a critical update. During those five to seven hours, 8 to 10 IT staff people would use the Windows 2000 Server Terminal Services feature to remotely access each server, apply the patch, and then restart the server. With an average staff pay rate of U.S.$46 per hour, the intensive labor requirements made the cost per patch approximately $1,840.

“Even the process of coming up with a list ofservers requiring a patch was time-consuming and had to be repeated for each patch,” says Greg Petersen, IT Systems Engineer at Avanade. “We’d dump a list of ‘all computers’ from our monitoring system and start sorting from there.”

As the number of security threats has increased, Avanade’s patches to its Microsoft technologies have also increased in frequency, from one or two per quarter to approximately one per month. Avanade needed a way to deploy patches faster, secure the enterprise sooner, and decrease IT costs and staff interruptions.

Solution

Eager to pioneer new Microsoft technologies that might be useful in solving problems thatits own customers might be facing, Avanade enrolled in the Microsoft Systems Management Server 2003 (or SMS 2003) Early Adopter Program in mid-2003. SMS 2003 provides a comprehensive change and configuration management solution for the Microsoft Windows ServerTM operating system, enabling organizations to quickly and cost-effectively provide relevant software and updates to users. SMS 2003 is part of the Microsoft Windows Server SystemTM integrated server software.

Avanade deployed SMS 2003 across three data centers and 18 offices in just three months. The configuration involves just four servers running SMS 2003 across one primary site and two secondary sites and a distribution point share in each of the local offices:

The primary site server, at Avanade’s datacenter near Seattle, Washington, consists of two servers. One Dell PowerEdge 6450 four-processor database server runs Windows Server 2003—the foundation of Microsoft Windows Server System—SQL Server 2000, and SMS 2003. The second server, a Dell PowerEdge 2450 dual-processor server running Windows Server 2003 and SMS 2003, is used as the SMS 2003 management point for clients and a distribution point server. The IT staff creates packages and advertisements for all updates on the primary server, which automatically distributes the updates to specified distribution points within the primary and secondary sites.

Two secondary site servers are located in Avanade’s U.K. and Singapore data centers. Each is a Dell PowerEdge 2450 server running Windows Server 2003 and SMS 2003.

Distribution points are located in each remote office to prevent multiple servers at a site from pulling a package across the wide area network (WAN) link. The distribution points are created as server shares on the existing file servers in the offices.

“We used the Active Directory® system discovery interface in SMS 2003 to help manage our system discovery for client installation,” says Brent Sommerseth, IT Systems Engineer at Avanade. “The entire client installation process was automated and wizarddriven from end to end and extremely simple. One person deployed SMS 2003 clients to all 250 servers.”

Avanade also uses Microsoft Operations Manager 2000 to complement SMS 2003 by providing automated, proactive server monitoring. Also part of Windows Server System, Operations Manager 2000 (or MOM 2000) provides automated server monitoring and alerts when critical hardware and software thresholds have been exceeded. For example, using management packs from Dell and Microsoft, MOM monitors CPU temperatures and fan and power supply voltages and sends an alert (by e-mail or pager) when a processor is overheating or a fan is about to fail. Microsoft Operations Manager Management Packs for specific Windows Server System applications help organizations to monitor the well-being of applications such as SMS 2003, SQL Server, and Microsoft Exchange Server.

Avanade uses the management packs for SMS 2003 and SQL Server to monitor its SMS 2003 implementation. The management packs provide alerts when server applications aren’t distributed or performing properly. For example, MOM flags disk space problems with SQL Server.

Lastly, Avanade takes advantage of several Windows Server 2003 features to provide improved systems management:

Avanade uses the Windows Update feature to help manage the deployment of critical updates to desktops. PCs automatically download updates, and users are prompted to installthe updates once theyare downloaded.

The Windows Server Group Policy Management Console (GPMC) helps administrators to better deploy and manage policies that automate key configuration areas such as users’ desktops, settings, security, and roaming profiles. Avanade uses GPMC to manage its Windows Terminal Server profile settings and help enforce network security settings such as complex passwords and forced password resets.

MOM works with the Active Directory service to locate machines that need attention of some kind.

Benefits

With the implementation of Systems Management Server 2003 and the development of a proactive patch management process, Avanade has streamlined patch management, realized dramatic savings, andreduced the risk to the enterprise by shortening the amount of time it takes to move critical patches into production. The company has also taken steps to eliminate the possibility of human oversight during patch rollout.

97 Percent Greater Efficiency in Patch Deployment

The deployment of SMS 2003 has generated a huge labor savings for Avanade. Today, it takes one person one hour to deploy a patch, whereas before 8 to 10 people spent five hours doing so—a whopping 97 percent efficiency improvement. With IT staff pay averaging $46 per hour, the elimination of 39 hours per patch saves Avanade a significant amount of money.

“We estimate that SMS 2003 is saving us $1,800 per patch, and we’re rolling out two to three patches per quarter,” Knight says. “It’s exponentially more efficient than the way we were operating before.”

The new process works like this: A single Avanade IT staff member uses the Microsoft Software Update Services integrated feature pack to download needed updates and set up the patch package. With a few mouse clicks, the individual can designate the group of servers needing the patch (all mail servers, for example, or all servers in a particular region) and create an update schedule for each group. Avanade IT staff can create these group lists once and reuse them for subsequent patches, something they couldn’t do before. The wizard-driven process of selecting the updates, downloading them to SMS 2003 servers, and combining them into an update package usually takes about 20 minutes.

The tremendous efficiencies provided by SMS 2003 have put an end to patch parties at Avanade. “People aren’t pulled away from their normal jobs to do patches,” Petersen says. “These people are able to continue working on projects that are critical to the business rather than doing maintenance work. The productivity savings from not disrupting our staffs’ work for patching is a huge savings.”

Faster Closure of Security Gaps

Of course, the other huge “soft” savings to Avanade is the value of implementing security patches quickly, which helps avert the potentially catastrophic consequences of damage from viruses and hackers. “Everyone knows when a big vulnerability is discovered,” Petersen says. “It becomes kind of a race to see how quickly you can get your enterprise covered. There’s no way the value of enterprisewide protection can be measured.”

Proactive Asset Management

In addition to greatly simplifying and accelerating security patch deployment, SMS 2003 gives Avanade a highly efficient asset management tool. SMS 2003 makes it easy to create reports showing the software that is installed on servers and how frequently each application is used.

“SMS 2003 has allowed us to make a more proactive assessment of what we have and be less reactive,” Sommerseth says. “Because certain vulnerabilities tend to target certain software, it’s a big help to know which servers are running which applications.”

“Installing SMS 2003 was a very enlightening experience,” Petersen adds. “We were able to see many opportunities for application and server consolidation.” With the better view of IT assets provided by SMS 2003, Avanade was able to see glaring inefficiencies and get out of the “one application per server” model that pervades many data centers.

“We were able to eliminate about 10 servers right away, and the savings here adds up to far more than the cost of the hardware,” Petersen says. “It costs roughly $500 a month in operational support costs to manage a server, so we probably saved morethan $50,000 in server management expenses in just one year.”

In general, SMS 2003 helps Avanade focus its IT resources more efficiently. “SMS 2003 helps you notice operational weaknesses that need to be strengthened,” Knight says. “It highlights areas we need to focus on and allows us to develop a coherent process for patching. Deploying reactionary patches is easy, though time-consuming. It’s a much more complex task to take a hard look at your applications and services and develop an intelligent plan for keeping software up-to-date and servers in service. SMS 2003 has given us an easy way to develop such an intelligent, proactive plan.”

Microsoft Windows Server System

Microsoft Windows Server System integrated server infrastructure software is designed to support end-to-end solutions built on Windows Server 2003. It creates an infrastructure based on integrated innovation, Microsoft’s holistic approach to building products and solutions that are intrinsically designed to work together and interact seamlessly with other data and applications across your IT environment. This allows you to reduce the costs of ongoing operations, deliver a more secure and reliable IT infrastructure, and drive valuable new capabilities for the future growth of your business.

For more information about Windows Server System, go to: