Customer Solution Case Study
/ / Large Title Insurance Company Supports12,000 Users of Mission-Critical Application
Overview
Country or Region:United States
Industry:Financial Services
Customer Profile
Based in Santa Ana, California, First American Title Insurance Company is the second largest title company in the United States with 27 percent of the market.
Business Situation
First American Title Insurance Company needed a faster and more stable infrastructure to support its mission-critical First American Software Technology (FAST) application that replaces a number of disparate systems.
Solution
First American upgraded its FAST infrastructure to Microsoft® Windows Server™2003, Datacenter Edition, Internet Information Services (IIS) version 6.0, and Microsoft SQL Server™ 2000 Enterprise Edition.
Benefits
Increased business agility
Lower IT costs
Faster performance
25percent reduction in database maintenance time
Better user experience / “With Windows Server 2003 and the 64-bit version of SQL Server running on HP Superdome, we’ve been able to tighten our SLAs to provide much better service.”
Sue Binks, Vice President of Application Development, First American Title Insurance Company
First American Title Insurance Company, a subsidiary of The First American Corporation, is the second largest title insurance company in the United States, with 2004 revenues of U.S.$6.7 billion. In 2002, the company deployed First American Software Technology (FAST), an integrated title and escrow system that unified disparate systems into a centralized database hosted on a 32-bit computer with 32 processors. But with the rapid growth of the FAST database, CPU utilization soon surpassed 80 percent. The company upgraded its mission-critical FAST to a 64-bit solution with Microsoft® Windows Server™ 2003, Datacenter Edition and Microsoft SQL Server™ 2000 Enterprise Edition (64-bit), running on an HP Integrity Superdome with 32 Intel Itanium 64-bit processors. Today, the centrally hosted SQL Server database, with more than 1 terabyte of information, supports 12,000 users in 1,300 offices.
Situation
First American Title Insurance Company, the largest subsidiary of The First American Corporation, has some 15,000 employees and more than 1,300 offices across the United States. The company is a highly decentralized organization—one that has grown largely by acquisition. By the late 1990s, the company had more than 50 different title and escrow systems, which ran on a number of platforms including the MS-DOS® operating system, Novell, UNIX, and the Microsoft® Windows® operating system. Some of these systems were stand-alone applications running on a single desktop computer, while others were server-based and supported dozens or hundreds of users.
Having such a distributed and heterogeneous IT environment presented several disadvantages for the company:
Higher costs: Having 50 disparate systems prevented First American from realizing economies of scale in IT infrastructure costs, data processing, and system administration.
Inefficient reporting: Financial and performance data had to be rolled up manually—a full-time job for several people. And by the time data reached upper management, it was three to fourweeks old.
Slow reaction times: Because there were so many platforms, applications, and data centers to address, IT projects were complex and costly.
To address these obstacles, the company developed First American Software Technology (FAST), an integrated title and escrow system—and the largest IT initiative in the company’s history. Built from the ground up over a period of three years, the new solution replaced all 50 title and escrow systems across the company with a state-of-the-art, Web-based, centrally managed solution.
When First American began to roll out FAST in late 2001, the system’s central database ran on a server running Microsoft Windows 2000 Advanced Server operating system, Internet Information Services (IIS) version 5.0, and Microsoft SQL Server™2000 Enterprise Edition (SQL Server 2000 and Windows 2000 Advanced Server are part of Microsoft Windows Server System™ integrated server software). The 32-bitserver had eight Intel Xeon processors MP and 8 gigabytes (GB) of RAM—the maximum that it could support.
Unanticipated business growth increased the workload on the system faster than expected—due partially to the refinancing boom that began at about the same time as the FAST rollout. In mid-2002, before the rollout was even halfway finished, the database server had run out of capacity. CPU utilization was at 90 to 95 percent, and users were complaining about slow system response. First American completed the rollout of FAST by moving the database onto another server, this one with 32 Intel Xeon processors and 32 GB of RAM, running Microsoft Windows 2000 Datacenter Server (part of Microsoft Windows Server System integrated server software) and SQL Server 2000.
This 32-bitsolution served the company well but, again, the company’s growth began overwhelming its FAST infrastructure, as CPU utilization exceeded 80 percent. An even more powerful infrastructure was needed. FAST is mission-critical, so the company needed an immediate, and enduring, solution.
Solution
First American Title Insurance Company upgraded FAST to Microsoft SQL Server 2000 Enterprise Edition (64-bit) and IIS 6.0 running on Microsoft Windows Server™ 2003, Datacenter Edition operating system, the foundation of Microsoft Windows Server System integrated server software. The solution is hosted on an HP Integrity Superdome with 32 Intel Itanium 64-bit processors.
SQL Server 2000 Enterprise Edition (64-bit) is a database management system optimized to run on Itanium-based servers and Windows Server 2003. The 64-bit version of SQL Server 2000 provides a scalable data platform optimized for the Itanium processor to help First American analyze large quantities of data and make informed decisions. It takes advantage of advanced memory-addressing capabilities for essential resources including caches, thereby reducing the need to perform multiple input/output (I/O) operations to bring data in and out of memory from disk, slashing processing time.
Gaining processing capacity without the penalties of I/O latency driving up processing time was important to the company. When First American was using 32-bit versions of SQL Server 2000 and the Windows operating system, complex queries that analyze extremely large amounts of data had to be broken down into smaller data sets that fit into the 4-GB memory space, with excess data temporarily stored on disk. With SQL Server 2000 (64-bit), First American has the space to process an entire query without breaking down the data into smaller sets. This increase in system memory was crucial to giving First American the fast, efficient, and sustained application performance it required for FAST.
FAST Architecture
FAST is a multitiered intranet Web-based application that services FAST users across a wide area network (WAN) infrastructure. The multitier structure includes:
Client Tier.Accessed using Microsoft Internet Explorer, FAST screens are rendered using HTML pages where custom components are automatically downloaded to manage client-side processing logic. Client tier functions include proxy interfaces to the middle-tier business objects, and client-side data validation. The client tier also includes Adobe Acrobat Reader so users can preview FAST documents.
Web Application Tier.The Web application tier uses IISto serve HTML and Active Server Pages (ASP), Microsoft JScript®development software files, image files, cascading style sheets, and cabinet (.cab) files to the Internet Explorer client browser. The Web application tier also serves as a middle-tier application server, using Windows 2000 Component Services, to encapsulate business logic and data. The tier provides abstracted access between the business objects layer and FAST SQL Server database using ActiveX® Data Objects (ADO). Message Queuing, a component of the Windows Server 2003 operating system, supports asynchronous interfacing with the document delivery tier. FAST Web services supports delivery of FAST data to non-FAST systems.
Document Delivery Tier. When a user requests a document to be printed, e-mailed, faxed, or previewed, the request is dispatched to a documentdelivery server to bind user data with a FAST user-defined document template. The binding of the data and the user-defined template is done using Microsoft Office Word 2003. The rendered document is then delivered to the requested destination (client’s printer, FAST mail server, FAST fax server, or client’s Adobe Acrobat Reader for previewing). This process uses the same set of business objects as used in the Web application tier to retrieve and resolve user data in a document.
Interfaces Tier. The interfaces tier enables FAST to interoperate with a number of non-FAST First American systems. Interfaces include:
−FastWebgives First American customers the ability to enter title/escrow orders and view status of these orders using the Internet.
−FastSearch provides title search data to FAST. The FAST interface system retrieves the FastSearch data automatically as a background process upon creation of a new order.
−WinTrack is used by the company’s Lenders Advantage division to process title orders. These orders are transmitted to FAST, which is then used by Lenders Advantage for the closing process.
−TrustNet provides trust accounting support for FAST escrow orders. This is a one-way interface that supports the transmission of both the original and adjusted trust accounting transactions.
−Datafaction is a general accounts receivable system for the First Americanoffices. This is a one-way interface that supports the transmission of invoiced items to the Datafaction system.
Database Tier. FAST uses a single instance of Microsoft SQL Server 2000 for its relational database. The database exceeds 1 terabyte and grows by 4 GB a day. The largest single table is 72 GB, with more than 120 million rows. SQL Server 2000 stored procedures are used to access the FAST database. The development team decided to use stored procedures because they offer excellent performance, ease of management, and are compiled and can be performance tuned as necessary. The FAST SQL Server 2000 database supports some 130 other servers, including 48 Web servers, 60 document-delivery servers, six check-printing servers, and three fax servers.
All tiers are supported by the 32-way Superdome server. Message Queuingis used to connect the solution’s subsystems in a highly reliable, fully integrated manner by providing guaranteed message delivery, efficient routing, security, and priority-based messaging. An EMC Symmetrix storage area network with 5 terabytes of usable disk capacity provides external storage. To maximize availability for its FAST solution, First American maintains a second, identical infrastructure at another First American data center in Dallas, Texas.
Data Warehouse Architecture
Having all data stored in one place made it easy for First American to take the next step—creating a data warehouse to improve its reporting capabilities. Basic elements of the solution include:
Data Warehouse Tier.Using Windows 2000 Advanced Server and the 64-bit version of SQL Server 2000, the company built a data warehouse that receives a feed from the FAST database server and aggregates 32 days of detailed history for each of the company’s 1,200 offices—approximately 700 GB of data in total.
Analytics Tier. Every night, First American builds online analytical processing (OLAP) cubes against the data warehouse to generate daily reports on business performance—reports that are updated with all the previous day’s data and are waiting on the managers’ desktops when they arrive for work in the morning.
“Executives can make better and faster business decisions, because the data that they require is immediately available—and in a consistent format across all offices and regions,” says Larry Godec, Vice President and Chief Information Officer for First American Title Insurance Company. “There’s no way we could have delivered the same level of business intelligence with the 50 different systems that we used to have.”
Mission-Critical Support Model
Windows Server 2003, Datacenter Edition running on the HP Superdome provides First American with more than just a scalable and reliable platform. The Datacenter High Availability Program, a partnership between Microsoft and server manufacturers like HP, complements the server and operating system with a mainframe-style support model that helps First American keep its mission-critical solution up and running.
Benefits
First American Title Insurance Company is enjoying a number of benefits since upgrading to the 64-bit version of Windows Server 2003, Datacenter Edition and SQL Server 2000 Enterprise Edition (64-bit), including increased business agility, lower IT costs, faster performance, a 25percent reduction in database maintenance time, better user experience, and support for changing business processes.
Increased Business Agility
With all its data in one place, First American is able to provide business decision makers with consistent and accurate information on the company’s performance—and with information updated to include all of the previous days’ transactions.
“Being responsive to the marketplace is critical to us, because the housing market and refinance boom could dry up as fast as it exploded,” says Godec. “With FAST and the data warehouse that it feeds, we have practically real-time insight into how the business is operating. And we only have one system to change when we need to address a new business need or a change in market conditions.”
Lower IT Costs
Consolidating 50 systems down to one system significantly decreased First American’s IT costs. System administration and software upgrades are now done in one location and are immediately available to users across the country.
“Consolidating to one system significantly increases our operational efficiency,” says Godec. “We’ll save 100,000 hours of IT effort per year—equivalent to 50 full-time people. Total IT cost savings are expected to be at least [U.S.]$5 million per year.”
Faster Performance
Performance had been a continuing problem with the earlier hardware and software infrastructures supporting FAST. Like rush-hour traffic, system demand is especially heavy in the morning hours, with an additional spike in usage as the company’s West Coast offices come online. CPU utilization frequently exceeded 80 percent with earlier solutions, slowing responses to unacceptable levels.
“During peak hours, we have some 12,000 concurrent users,” says Sue Binks, Vice President of Application Development at First American Title Insurance Company. “Slowdowns would reach the point where help-desk calls were generated. Windows Server 2003 and the 64-bit version of SQL Server allow us to take full advantage of our 64-bit hardware. Performance is no longer an issue, not even at month’s end when users remain logged on to FAST for hours at a time.”
Performance has been enhanced by the greater memory addressability of Windows Server 2003, which has eliminated the company’s need for using the Windows 2000 AddressWindowing Extensions (AWE) application programming interface (API) to support very large memory sizes.
“With Windows 2000 Server, we were limited to addressing 4 GB of memory space,” says Priti Desai, Director of Production Systems at First American Title Insurance Company. “To go beyond 4 GB, we had to use AWE, which is not as effective as the flat address space provided by Windows Server 2003 and the 64-bit version of SQL Server 2000; we can address up to 512 GB of memory directly, without using AWE.”
The ability to directly address up to 512 GB of memory has greatly increased the size of the process cache. “Having direct access to 64 times as much memory with Windows Server 2003 means we can cache our complete SQL Server project plan procedures in memory, which decreases the time required for recompilation,” Desai says. “The combination of the larger procedure cache and faster recompilation has significantly enhanced performance. This is a huge win that enables our systems to run much faster.”
Additional performance improvements come from the 64-bit support of Windows Server 2003, Datacenter Edition and SQL Server 2000 Enterprise Edition (64-bit), which take advantage of the 64-bit architecture of the Itanium CPUs. “Because SQL Server uses the 64-bit capabilities of Itanium, SQL Server can parallel-process six instructions at one time,” says Desai. “This is a huge benefit of the Itanium architecture that we couldn’t capture without our move to the 64-bit version of SQL Server.”
Twenty-Five Percent Reduction in Database Maintenance Time
The direct addressability of 512 GB of memory that has enhanced throughput on the operations side has reduced database maintenance time. “It used to take 12 hours to run our database maintenance—and with our growing database it was pushing 14 hours,” says Desai. “It now takes just nine hours, so we’ve seen at least a 25percent reduction in maintenance time since moving to Windows Server 2003 and the 64-bit version of SQL Server running on the HP Superdome. This time savings is important, because we estimate that we are losing one milliondollars for every hour that FAST is down.”
Consolidated Web Servers
The company reports additional savings from consolidating Web servers. “We’ve have been able to achieve a 40percent consolidation, reducing the number of our servers hosting IIS from 80 to 48,” Binks says. First American estimates that the server consolidation provided a one-time savings in hardware and software of $640,000, and provides an ongoing $9,000 a year savings in reduced maintenance costs.