Azure Logic Apps Streamline Marketing Data Integration

Page 1 | Azure Logic Apps streamline marketing data integration

Azure Logic Apps streamline marketing data integration

Microsoft IT supports the big data platform that supports many business scenarios for ourmarketing team at Microsoft. To support these scenarios, data is pulled from more than120 data sources from many transactional systems. This data pull has been managed and executed by a variety of custom-designed solutions in the past. Typically, these solutions have been developed with specific data sources in mind. They’re usually developed on a one-by-one basis, which makes the development and management process time-consuming, but it’s the only option we’ve had until now. Recently, we’ve begun touse Azure Logic Apps to create modular logical data flows that supplement and often replace data orchestration mechanisms. Using Logic Apps, we can quickly create logical data movement tasks that are modular, reusable,configurable, and easy to assemble. With the increased efficiency, we’re better able to deploy and manage data movement solutions within the marketing environment.

Understanding data integration in marketing

Microsoft Azure is the default platform for infrastructure management at Microsoft. We anticipate that 80 percent of our computing resources will be hosted in Azure by the end of the 2017 fiscal year. By hosting our apps and solutions on Azure, we have broad access to data sources—and a storage and data management platform that we use as our central repository for business data.

Our design for marketing data integration is to provide our customers with the best purchasing opportunities, based on the data we have about these customers. This solution is the core of our bulk data transform and load processes. Most of the 120 data sources we pull from are hosted in Azure. We’ve been developing Azure-based solutions to move, manipulate, and analyze the data for use by a variety of apps and services. One of our solutions for data integration in Azure has been a custom-coded solution on the Azure Cloud Services platform using Web Jobs. However, we’ve identified some issues:

Logical data flow is sometimes hard to define and manage between multiple data sources.
Data connectors must be custom created individually for each data source. Often, these connectors must be created from scratch and require developer time for both creation and maintenance.
Integration with Azure components, such as Azure Blob storage and Azure Data Lake can be complex and require additional code or modifications to the data integration logic.
Additional functionality meant having to build on existing functional code,

Overall, demand was high for engineering resources to maintain the solution.

Understanding Logic Apps

Logic Apps provide a way to simplify and implement scalable integrations and workflows in the cloud. It provides a visual designer to model and automate a process as a series of steps known as a workflow. There are many connectors across the cloud and on-premises to quickly integrate across services and protocols. A logic app begins with a trigger and after firing can begin many combinations actions, conversions, and condition logic. The advantages of using Logic Apps include:

Reducing integration challenges by using out-of-the-box connectors.
Saving time by designing complex processes using easy-to-understand design tools.
Seamlessly implementing patterns and workflows that would otherwise be difficult to implement in code.
Getting started quickly from templates.
Customizing your logic app with your own custom APIs, code, and actions.
Connecting and synchronizing disparate systems across on-premises and the cloud.

Building off of API Management, Azure Functions, and Azure Service Bus with embedded integration support.

A decision tree When a record is updated Transform XML Register in Datamart If record requires action If yes Create a new record Slack Post Message If no Execute stored procedure

Figure 1. A Logic Apps data flow diagram

Logic Apps is a fully managed iPaaS (integration PlatformasaService). Therefore, developers don’t have to worry about building, hosting, scalability, availability, and management. Logic Apps will scale up automatically to meet demand.

Extending data integration with Logic Apps

We recognized Logic Apps as a solution to our data integration needs when evaluating it for large data pull transactions between multiple data sources. The modular nature of Logic Apps would allow us to build out a solution in a short timeframe; there were several out-of-the-box data connectors available in the Logic Appsgallery that would allow us to integrate SaaS (software as a service) and enterprise applications. As we started using Logic Apps further, we realized its potential as a solution for many of our data integration needs.

Our big data platform is used to drive marketing efforts across the entire organization. We frequently need to pull data from a variety of data sources, including on-premises resources like SQL Server databases, flat files, and SharePoint, as well as cloud-based resources hosted in Azure and other online locations by third-party providers. Logic Apps gave us the flexibility and reusability to integrate any of these data sources using a single solution.

Diagram that shows the following all leading to the Marketing big data platform Corporate network on premises SQL files SharePoint Azure Services SQL Azure Blob Storage Through SaaS APIs third party Saas

Figure 2. Logic Apps data integration scenarios for marketing

Working with Logic Apps

Our engineers used the integrated web-based designer and Visual Studio Online to deploy Logic Apps into our Azure environment. This provides a code view for the JavaScript Object Notation (JSON)–based templates, and a drag-and-drop visual editor for constructing logical data flows. Because Logic Apps are JSON-based, redeployment, minor customizations, and even complete repurposing of a Logic Apps deployment are easy to accomplish.

Making data integration more efficient with Logic Apps

Logic Apps was used to implement data integration tasks for two large data sources in marketing. Both data sources contained important business and marketing data that was used to create sales opportunities and identify customers who could benefit from the products and services that Microsoft provides.

In both cases, the data integration was relatively complex. It required data pull and movement between several endpoints, and retry logic that ensured data integrity was kept intact. Our engineers used the built-in error and retry logic components in Logic Apps to ensure data integrity:

Retry policies. Retry policies define if and howactions should be retried when the initial request fails. The policy defines both the timing and reoccurrence of retries if a failure occurs.

Azure diagnostics and telemetry. Azure diagnostics provide a way to send all workflow events to an Azure storage account or Azure Event Hub. Stream Analytics provides creation for real-time query creation for analysis or output to other analysis tools.

The built-in tools saved our engineers considerable time. Without Logic Apps, they would have needed to create the retry logic and other components from scratch in the solution code. On average, our engineers have been spending 50 percent less time developing data integration solutions with Logic Apps compared to other solutions that are in place.

Assessing the benefits of Logic Apps in data integration

We’ve had the chance to use Logic Apps for several data integration tasks within the marketingteam.The integration of Logic Apps into the Azure environment makes it a natural choice for working with data coming from Azure-based data sources or going to Azure-based data storage mechanisms. Additionally, we realized some benefits, including:

A huge reduction in time required to build data integration. Overall, the data integration build process has been greatly simplified with Logic Apps. Engineers can use built-in data connectors to connect to data sources rather than building them from scratch. The Logic Apps designer provides a simple—but capable—toolset that our engineering teams can use to quickly develop and produce functioning Logic Apps. They’re also able to reuse existing Logic Apps for other solutions.
A cost reduction from engineering time. Our engineers spend 50 percent less time working on data integration tasks when using Logic Apps. This provide us with a reduced requirement for dedicated time on data integration and frees our engineers to focus their time on other projects.

Agility in building new scenarios. Logic Apps are highly modular and reusable. Because of the open nature of JSON templates, our engineers can reuse code blocks and data logic in other Logic Apps solutions.And the simplicity of the code makes it easy for an engineer to understand and work with Logic Apps solutions created by other engineers.

Logic Apps adoption continues to grow at Microsoft IT. The modular nature of design and the efficiency that it brings to our data management platform has provided immediate benefits that we expect to expand and extend into the future. We’re using it for many current data integration tasks and plan to use it heavily across our data management platforms.

For more information

Microsoft IT

© 2016 Microsoft Corporation. All rights reserved. Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.

IT Showcase Article

microsoft.com/itshowcaseDecember 2016