In today’s landscape, customers expect their applications to have integrations with popular third parties (Slack, Salesforce, Hubspot) and real-time data from those integrated third parties. Building these integrations for your SaaS application will involve webhook listeners that receive data in real time to deliver a seamless experience where your customers can do more within your application.

Why integrate third-party webhooks?

Customers generally have an ecosystem of software products that they use for their business. They might use Slack for messaging, Salesforce for their CRM, Monday for their project management, just to name a few. When building a multi-tenant SaaS product, you may want to use data from your customers’ ecosystems - loading data in real time, triggering automated tasks, and producing a tailored experience for each customer.

Imagine your application uses leads data from your customers’ Hubspot accounts. Rather than have your customer input data they already have and import them to your application every time a lead is created, your application should automate that process and receive that Hubspot data immediately without user interaction.

Webhooks enable this type of interaction, where a third-party application (Salesforce in this example) pushes data to your application in real time whenever an action is taken on their end.

What are webhooks?

Traditionally, an application may use the polling pattern where your application “polls” a third party by sending their API a request for new data at a set cadence (i.e. every 10 minutes, send a GET request asking for any new messages from Slack). Webhooks are a different pattern that reverses this flow, where the third party API (the webhook producer) sends a request to your application (the webhook consumer) whenever new data is available. This not only enables real time data to be reflected in your application, but it also reduces the amount of requests that are involved as the polling pattern sends requests regardless if new data is introduced (also called an “empty trip”).

General steps for building webhook consumers

For your application to receive data from third-party application, you will need to go through these general steps in building webhook consumers.

1. Set up your webhook endpoints and implement logic for your endpoints that accepts the third-party data and processes them for use

2. Ensure your endpoint is in accordance with your third-party API’s authentication and authorization practice

3. Design and deploy the infrastructure that will handle scaling, error handling, and monitoring

4. Repeat for each integration

General challenges of building webhook consumers

Let’s go through the challenges involved in each step of building webhook consumers.

Setting up and implementing logic

There is no standard webhook producer implementation and as a result, each third party will have different formats for their data and different ways they send data to your webhook consumer. For example, some third-party API’s will send an individual message to your consumer for each event that occurs, while other third-party API’s will batch messages together. It takes a trained eye to go through their documentation to find out their standards.

Even after setting up your webhook logic correctly and verifying your logic is processing their data, the third-party API could introduce breaking changes with updates that will force engineering effort from your team.

Authentication and authorization

Each third-party API may use different practices for properly authenticating messages. For example, Slack requires a configuration handshake between your endpoint URL and their API, where your endpoint will receive a request and will need to respond with a challenge for Slack to acknowledge your endpoint. Again, it takes going through their documentation to properly build a webhook consumer for each third party.

It may now be apparent that unlike standard authorization practices where you have OAuth and SAML that are generally industry standard, there is more variety for webhook authentication (Even OAuth has its challenges across integrations, see https://www.useparagon.com/blog/challenges-with-integration-authentication). Authentication can be at an app level or a user level; different strategies are used from Java Web Tokens (JWT) to Hash-based Message Authentication Codes (HMAC); documentation thoroughness also varies from third party to third party.

Design and infrastructure

With any data intensive system, the design and infrastructure to accommodate scaling and faults while having observability can be challenging. Webhook consumers may require scalable designs such as message queues for storing messages, horizontal scaling of webhook consumer instances, implementing retry policies, having dead-letter queues for messages that can’t be processed, enforcing data idempotency in case of duplicate messages, and using occasional polling to verify messages.

Unlike the “polling” pattern, webhooks can be less resilient to errors as once a message is sent by the producer, the onus is on your application to properly process that message. That’s why queues and monitoring become paramount. Retaining messages and errors can help with debugging and remedying logic.

Building webhook consumers for multiple tenants across multiple integrations

Despite the challenges laid out, building a webhook integration for a third-party integration doesn’t have to be arduous with a proper level of engineering effort. What can become harder is building and maintaining fleets of webhooks needed for each tenant across multiple third-party integrations.

When working with multiple third-party integrations, it means staying on top of multiple organizations’ documentation, implementing logic for different message formats, familiarizing with different authentication strategies as each third-party webhook producer works differently.

When working with multiple tenants, lifecycle management of their webhooks also comes into play as customers may decide to un-integrate to certain third parties or opt out of webhook dependent features. Quickly creating, modifying, and tearing down webhooks programmatically will reduce unnecessary work done by your application.

Paragon alleviates many of the challenges that come with building webhook consumers for any integration your customers may use. Using Paragon’s own webhook consumer infrastructure and developer-friendly workflows, the challenges around webhooks are largely abstracted away.