Increases Scalability

• Avoids costs
• Increases scalability
• Speeds time-to-market
• Improves data stability
• Enhances competitiveness

Situation

Started in 1999, Flavorus is a primary ticket seller, sellingtickets to individuals on behalf ofits customers, which include venues, artists, and promoters. Its main revenue source is service fees on ticket sales. The company is the largest ticket seller of electronic music events in the United States. “We have advanced client services,” says James Reichardt, Chief Technology Officer (CTO) and Lead Programmer at Flavorus. “We can handle festivals, but we’re still growing. We’re not as big as Ticketmaster yet.”

Flavorus is a midsize company withsales, marketing, customer service and support, design, and development departments. On its website, the company hosts a complete ticketing solution that makes it possible for customers to create an engaging and easy-to-use ticket sales page for an event in minutes. Selling tickets on Flavorus is free, and the company provides flexible ticket-payment options and real-time management tools where customers can view a breakdown of sales and projected sales.

Flavorus provides ticket buyers with a variety of services, including low service fees and multiple ticket types. The company supports traditional tickets, print-at-home tickets, and mobile tickets. “As far as ticket buyers are concerned, usability is the biggest challenge,” says Reichardt. “It is one thing to buy a ticket, but it’s another thing to be able to get into the venue easily—especially when it comes to large events. We offer services that aid concert-goers in getting into the event as fast as possible.” Flavorus recently introduced paperless ticketing that features a customizable card that fits into a wallet like a credit card. It can hold multiple tickets and is scanned at the entrance, just as a paper ticket would be.

Flavorus also makes it possible for itscustomers to connect with ticket buyersby using marketing tools. “We offer added functionality for both our customers and theirs,” says Reichardt. “We provide built-in social networking tools and tie-ins to popular social networks, and we support the posting of videos, photos, and blogs.”

Ticketing is a unique problem on the Internet because demand is only high for a short time. In the past few years, some ticketing companies have had major issues with large-scale events. For example, in 2010, the ticketing site for Comic-Con International experienced crashes. In January 2011, the Burning Man organization, which uses a queuing method (ticket buyers wait in a virtual line), encounteredissueswhen its ticketing vendor’s website could not handle demand after tickets went on sale. “When big events go on sale, ticket sales can hit some of the highest peak bandwidth, even higher than that experienced by Facebook or Google,” says Reichardt. “As many as 300,000 buyers can hit a ticketing website at one time, vying for 80,000 tickets. An event can sell out in a few seconds.”

Flavorus hosts its own on-premises data center in Los Angeles that features the capacity to handle midsize events. It is traditionally a very technology-focused company. “For a company our size, we have very nice servers with cutting-edge firewalls and load balancing,” says Reichardt. “We have had slowdowns in the past, but our servers can handle a pretty large workload. For example, for a large event at the Hollywood Palladium in Los Angeles, we sold 2,000 tickets in two hours—about 20 orders a minute.”

In early January 2011, the company started looking into how it would handle ticket sales for the largest paid music festival in the United States—expected to attract as many as 150,000 attendees per day—to be held in Las Vegas, Nevada, in June 2011. In 2010, Flavorus sold tickets for the same event as a secondary ticketing company, and in 2011, because of a venue change, it would be acting as the primary agent. It knew that its servers couldn’t handle the load. “We had never sold tickets for an event that came close to the demand of this event,” says Reichardt. “We didn’t know if the event was going to sell out in five seconds or 10 hours, but when we looked at the scalability of our hardware, there was just no way we could handle it.”

In addition, the company would be servicing ongoing ticket sales for hundreds of other events at the same time—all of which put demands on its data center for compute time and database querying. Flavorus considered handling the event’s large-scale traffic by bolstering its on-premises solution with greater database capabilities, additional web servers, and higher bandwidth. “From a business perspective, it didn’t make sense to buy hardware that we might not need otherwise,” says Reichardt. “On the flip side, this was our biggest customer, and we had to do it right. We were up against a wall on how to approach scaling for this particular event.”

Flavorus started reviewing options for hosting ticketing sales in the cloud with a comprehensive ticketing solution presented as a set of applications and services accessed over the Internet. The solution would need to be easy to maintain and scale to handle ticketing for individual events that faced a large demand on the day tickets go on sale. Flavorus also had to make sure that any cloud-hosted solution was highly stable and would provide seamless customer service. “The way the Internet is now—with social networking and everything else—if one customer has a bad experience, he has a very strong voice,” says Reichardt. “We needed the system to be stable enough that all of our customers would have a good experience buying tickets and checking into the event.”

Additionally, a primary consideration for Flavorus—and for ticketing companies in general—is to make sure that every ticket sold from a database maps to a physical ticket. The company’s relational database structure has to be flawless. Flavorus needed a solution that would enable high-volume reading and writing to its ticketing database and an application that was optimized for the cloud. It also needed to launch a fully functioning ticketing application within a short time frame to accommodate its marquee customer.

Solution

Flavorus considered hosting its service with a company that offers a cloud solution for websites. However, Flavorus did some basic testing using that model, and its engineers determined that the solution couldn’t handle the scale of traffic required. “The hosting company offers scalable commodity hardware in that they have a server and run virtual machines in the cloud,” says Reichardt. “You can host a traditional website in the cloud, but the application itself doesn’t scale. That’s a problem when you have the amount of traffic we were facing.”

Instead, Flavorus chose to host its ticket application on the Windows Azure platform. Developers can use Windows Azure to host, scale, and manage web applications on the Internet through Microsoft data centers. The platform includes a set of development tools, services, and management systems familiar to Flavorus. The company builds solutions by using the Microsoft .NET Framework, software that provides a comprehensive programming model and set of application programming interfaces for building applications and services. “We’re a .NET shop and I’m a .NET programmer,” says Reichardt. “Microsoft solutions are much more rich and easy to program.”

Sharding the Data

In early February 2011, Flavorus started working closely with product representatives for the Windows Azure platform. At first, the company wrote its ticketing application—which was built by using Microsoft SQL Server 2008 data management software—for a traditional single-instance database in Microsoft ASP.NET, a web application framework for building dynamic websites, web applications, and web services. It evaluated storage options on the Windows Azure platform and found that Microsoft SQL Azure, a cloud-based relational and self-managed database service, was the fastest option for its performance needs.

Flavorus used the .NET Framework to rework the ticket-processing application—which the company calls Jetstream—to run on Windows Azure, the Microsoft cloud services development, hosting, and management platform. The Jetstream application is very simple: it collects the customer’s data and reserves a ticket, while a dynamic webpage calls a database and processes the customer data.

It was easy for Flavorus to port its solution to the Windows Azure platform. The company developed and tested its initial application from February 7 to February 20 and then ran the initial high-volume tests on February 21 and February 22.

The first volume tests on the Windows Azure platform failed. One SQL Azure database did not provide enough computational capacity to process large numbers of tickets and high user traffic. The database automatically restricted connections when the load was exceeded. For its ticketing application, Flavorus needed to scale beyond what a single SQL Azure database could provide.

The company decided to adopt sharding in SQL Azure to take advantage of the massive scale that a SQL Azure cluster provides.Sharding is a horizontal partitioning technique that spreads data across multiple databases. To this end, the company could turn its single-database ticketing service into an application with data that is split across multiple SQL Azure database servers (Figure 1).

The company was able to scale its on-premises, single-database solution into a sharded solution within two weeks. By doing this, Flavorus gained higher availability by engaging many databases. “By using sharding with SQL Azure, we can have tons of customers on the site at once trying to buy tickets,” says Reichardt. “If any one of the database servers goes down, it doesn’t really matter.”

Flavorus realized that the ticketing application was ideal for sharding because the tickets are all the same. “With sharding, you can put a few tickets on each database if you want,” says Reichardt. “It doesn’t matter which database the ticket is on. You just need to distribute the load. It’s a beautiful thing.”

From February 22 to February 24, Flavorus ran initial high-volume tests using sharding. The ticketing application was able to handle a large amount of ticket sales but hit a limit because of how Internet Information Services 7.5, the default web server role on the Windows Azure platform, handles traffic. To get around this limitation, the company had to rewrite the data-dependent routing logic.

For debugging purposes, Flavorus needed to access logs to troubleshoot issues with the compute nodes and databases. “I was able to retrieve the logs and track down the line of code that was causing the web servers to have difficulty managing connections to the databases,” says Reichardt.

From March 2 to March 24, Flavorus ran tests using the new logic, gradually increasing the number of sharding instances. By March 25, Reichardt was running volume tests with 7,000 general-admission tickets available in each of the 550 SQL Azure databases. During load testing, he successfully sold 150,000 tickets in 10 seconds. “When we reached that speed, we realized we could fulfill ticketing for any festival in the United States,” says Reichardt. “The Jetstream application literally scaled as I added servers. We were ready to go.”

A few days before event tickets went on sale, Flavorus learned that the promoters wanted to sell eight ticket types at different prices. To accommodate this request, the company sharded the ticket data into eight groups of servers, with each group serving a specific ticket type. For example, it used 20 databases for the VIP tickets and 400 databases for the most popular tickets—the general-admission, three-day tickets. Flavorus allocated ticket data by using projections based on the previous year’s ticket sales. “Sharding gives us infinite flexibility in terms of sending traffic to different databases,” says Reichardt. “It was an eye-opening experience for us in terms of SQL Azure capabilities.”

Launching Jetstream in the Cloud

To deploy Jetstream to production on the Windows Azure platform, Reichardt simply ran a command in the Microsoft Visual Studio 2010 Professional development system, and the application—which comprises 1,391 lines of code—launched in the cloud. He then assigned the application and data to the required number of web servers and database servers.

Tickets went on sale for the event on April 23, and Jetstream worked flawlessly. To support the spike in first-day ticket sales, Flavorus used the Windows Azure platform to host 750 web role instances of the Jetstream application and 550 SQL Azure databases. The application wrote chunks of customer data to multiple sharded SQL Azure databases in parallel. Jetstream was live for two days, after which the company returned the event data to its on-premises servers and continued selling tickets. “For an on-sale event, we really need scalability for a small amount of time—about two hours,” says Reichardt. “Abouthalf an hour before sales start, traffic ramps way up. Then you get the latecomers who go to the website to find out if tickets are still available.”

The company continues to use its on-premises environment for day-to-day ticketing sales. It offers the Jetstream service only for customers promoting large-scale events. “By using the cloud model, we can isolate the ticket sales of one event,” says Reichardt. “We can run Jetstream on demand and point the ticket-sales traffic to that application. It doesn’t affect any of the other events on our website. Then, we can shut down Jetstream and pull the data back into our system to process it—with no interruption in service.”

Benefits

By designing its Jetstream application to scale for large events by using SQL Azure, Flavorus benefits from greatly reduced costs while increasing the ability to handle high ticket-sales traffic exponentially. The company used Windows Azure technologies to ramp up capacity within a short time frame, easily meeting the deadline for its largest customer. It also benefits from highly reliable data stability. Thanks to cloud capabilities, Flavorus finds itself in a new competitive landscape in which it can host ticket sales for the largest events in the country.

“Part of the beauty of SQL Azure is that you can tinker with its capabilities if you have unique needs—like the way we adopted sharding to scale ticketing on the on-sale day,” says Reichardt. “But for the most part, the Windows Azure platform handles everything for you. It’s a really beautiful model.”

Avoids Costs

With its ticketing application in the cloud, Flavorus can sell tickets for big events without investing in infrastructure. The company avoids purchasing bandwidth anda variety of hardware and software that would need to be replaced in a few years anyway. It also bypasses the costs of administering and maintaining a large on-premises data center. Flavorus estimates that it is saving more than U.S.$100,000 annually in hardware, software, and bandwidth.

When putting Jetstream into operation on the Windows Azure platform, Flavorus pays only for each hour of use. The cost of running Jetstream on Windows Azure for two days was minimal, and the on-sale ticketing event was profitable. “Using sharding with SQL Azure is so cost-effective, it is mind-blowing,” says Reichardt. “Plus, we can turn Jetstream off when we’re not using it—and there’s no cost to us.”

Increases Scalability

Thanks to the sharding capabilities of SQL Azure, Flavorus has gone from knowing its servers would crash during high-volume on-sale events to being able to sell 900,000 tickets in a minute. The company has more scalability than it needs. “Sharding with SQL Azure turned out to be the golden ticket for us,” says Reichardt. “It’s almost infinitely scalable. You could continue to ramp it up with more databases and get faster and faster times. It is like night and day compared to what we had done before.”