Azure Logic Apps makes simple integration scenarios a breeze. Securely integrating two web applications that expose API endpoints can be done in a matter of hours, not days or weeks. In this post, I will walk through a basic Azure Logic Apps workflow of retrieving paged data from one REST API endpoint, transforming and preparing the data, and posting the prepared data to another REST API endpoint. The most difficult part of this process is requirements and mapping – making sure you can authenticate to both APIs and you know which fields should be mapped.
Testing Authentication and Request/Response Format
I recommend using Postman or your favorite API tool to test the following:
- Authentication: Most APIs use a basic API token passed in the header. Some require a base64 encoded header. Some require initiating a session by hitting an authentication endpoint to get an expiring token. Be sure to allocate time for this, as many vendors’ API documentation is not straightforward.
- Request: Test whether the request is a GET vs. POST method and whether it expects querystring parameters, route parameters, or a JSON body.
- Response: Test the response format (text, JSON, e.g.) and the schema. Which fields will you need to translate to the destination? Are the fields easy to access or are they nested or an array? Test both successful responses and error responses. In some cases, errors are still reported as HTTP status code 200 but a custom status code is present inside the body. Note the schema or copy a sample response – you will need it in the Parse JSON step of the logic app.
Field Mapping
I won’t dwell here, other than to say this is a very important part of the planning process. You need to understand if there are 5 fields or 100. You need to understand whether the fields are a 1:1 mapping or they need to be transformed (splitting first name/last name, e.g.). This will add to the complexity of the integration.
Workflow Planning
I remember my first programming class in junior college. There was no coding – just logical flow diagrams. I loved it! Azure Logic Apps is a more powerful version of that. You point and click and design a logical flow. Along the way, you have to wire things up because each step references the previous, but getting the flow right is half the battle. I’ll paste the basic flow below and then discuss the components.
It looks like there is a lot here, but let’s break it down.
- Schedule Trigger: This kicks off the process and allows the app to run every 24 hours. You can also set it by Minute, Hour, Day, Week, or Month.
- Set Variables: I ended up needing three variables (one is not picture here) to capture the total records to process, the number of processed records, and a third variable for the current time. Because the logic app might take a few minutes to run, you want every step to use the same time.
- Loop Until: This is used to process all records in a ‘paged’ API setting. Most APIs limit the number of records that can be retrieved in a single call and provide what’s called a continuation token that points to the next page. The loop allows us to increment the processed records and ensure we’ve processed all the records.
- Get Source Data: Call the source vendor’s API (typically a GET request) to get the paged records.
- Parse Source JSON: This step is critical because it converts the generic “body” property of the HTTP response to a concrete JSON structure that can be referenced easily in future steps. This is where you’ll need the sample response body from Postman or another API testing tool. Be careful – Azure Logic Apps will interpret any non-empty field as “required” so you may need to go through and change the schema. I recommend just removing anything from the schema that you don’t need.
- Validation Condition: You need to determine if the data is valid. You could throw an error here, but for this example it just skips the record and goes to the next.
- Send Data to Target: This is the REST API (typically a POST or PUT) method to send the data to the target vendor.
- Parse JSON: Not pictured here, but you likely want to parse the JSON response of the second HTTP request as well so you can log it.
- Success/Failure Condition: This goes back to the Postman testing. You need to know what a failure looks like. Is it a 4XX or 5XX HTTP response code? Is it a 200 response code with an error code/message in the body? You’ll need that in a future step to log or send the correct email.
- Log the Result in a Table: Plenty of options here for logging (Azure Log Analytics, Table Storage, Blob Storage). I like Azure Table Storage because it is cheap and easy to query. Choose what’s right for you.
- Send Success/Failure Email: Setting up a SendGrid account takes about 5 minutes and the free tier allows 25,000 emails a month – more than enough for our use case. You can set it up through the Azure Portal and then click ‘Manage’ to dive into the SendGrid account. Capture the API Key and paste it into the Logic App and you’re up and running. If you want to send from a corporate account, you’ll need to register the SendGrid mail server with your corporate DNS – outside the scope of this document but your IT team should know what to do. You can alternatively use a Gmail account, which is supported in Azure Logic Apps as an Action – just enter your Gmail credentials directly into the Logic App step.
Wrapping Up / Best Practices
Overall I really like Azure Logic Apps for simple integration scenarios like this. It doesn’t replace things like Azure Data Factory or Event Hub for complex mapping and big data integration, but it should definitely be a tool in your toolbox. You might say “I’m a developer. I don’t need Logic Apps.” And my response would be…why develop something for 3 weeks that you could configure in 3 hours?
I had the following takeaways/best practices:
- Use Postman for planning and initial testing
- Create a field-by-field mapping document in advance
- Create a workflow diagram in advance
- Understand what error scenarios exist and log them appropriately
- Use Azure Tables for cheap logging
- Use SendGrid for easy email capability
- At critical points, grab the source code and put it in Git for version control. Azure Logic Apps has a built-in version history, but it’s difficult to see the changes, and there are no advanced features like tagging.
I’m interested in your thoughts and experiences with Logic Apps for data integration. Leave feedback in the comments!