Why Your Product Needs Webhooks
Every product gets to the point where they’re missing a key element to continue their growth trajectory. Integrations.
Integrations with other popular services helps you acquire customers as well as retain them as your product becomes embedded in their workflows. However, creating these integrations consumes valuable engineering resources. It’s especially painful when you need to keep diverting engineering time to the next integration (and the next, and the next, and…).
Enter webhooks.
Webhooks, also known as http callbacks, are a way for APIs to notify applications that a specific event has occurred without receiving a request. Webhooks allow users to specify an endpoint URL where they can receive real time event updates, eliminating the need to send constant requests to their API providers looking for updates.
For example, Zapier, a popular low/no code integration platform, hates API polling:
Polling is the process of repeatedly hitting the same endpoint looking for new data. We don't like doing this (it's wasteful), vendors don't like us doing it (again, it's wasteful) and users dislike it (they have to wait a maximum interval to trigger on new data). However, it is the one method that is ubiquitous, so we support it.
Many products have APIs with a well documented webhook service that make receiving real time updates a breeze for their users.
Why webhooks
Here are 4 reasons why your procuct should offer a webhooks feature and stop relying on API polling:
Easy Integration
Webhooks enable both custom integrations as well as listing on no-code integration platforms.
Zapier, IFTTT, and Make (formerly Integromat) are very common ways for products to tap into a network of app integrations. All can work using API polling and webhooks (and we already know how we feel about polling).
Integrating with these workflow automation platforms has many benefits:
Integrate with 3000+ apps currently available in the marketplace
Continued integration with any new apps added
Increased visibility to 3.5M+ users
In addition to no code platforms, custom integrations become more straightforward and repeatable. For example, Hubspot has a public API with full documentation on how to set up a webhook listener to receive event notifications from them so that anyone can integrate with their platform.
Resource Efficiency
Generally, a webhook system will be much less resource intensive than a polling system. This is true for both the API provider and consumer.
For example, if 1,000 users poll the API every 5 seconds, your API will have to be able to handle up to 1,000 requests per second depending on the timing of the requests.
Zapier estimates that only 1.5% of polling requests find an update. That implies that with a webhook solution, you would only need to send up to 15 responses per second while your users don’t have to send any requests at all. That’s < 1% of the resources needed to sustain the polling solution (1000 requests + 1000 responses)
Real Time Updates
Our example scenario only accounts for 1 request every 5 seconds. What if customers want updates in real time? They would need to send requests at least every second if not more. This increases the load on your system and increases operational complexity.
If the API provider has rate limits, you could end up blocked from sending requests due to exceeding the specified rate limit. If the customer has implemented polling at frequencies that do not exceed the rate limit, they end up polling so infrequently that their updates are stale. This can decrease customer satisfaction and lead to churn as your API could no longer be a valid solution.
Webhooks get around this issue by sending updates right as events happen.
Customer Experience
Webhooks also offer a better customer experience. Forcing your users to integrate by constantly polling your API for updates means they need to do the heavy lifting of maintaining and comparing state to determine if there have been any changes and what actually changed.
Think about the common situation of a family on a road trip. Polling is equivalent to the kids constantly asking “Are we there yet?” and the parents having to say “no” over and over again. Wouldn’t you prefer the webhook scenario where the kids are quietly reading a book or watching a movie while the parents get to drive in peace?
Developers Expect Webhooks
Major API providers like Stripe and Plaid have very well documented webhook features and they are becoming commonplace with the most popular APIs. That means developers are used to working with the webhook pattern as they are fairly simple for customers to ingest. The more others start offering webhooks, the more developers will expect webhooks to integrate with.
Webhooks also improve the developer experience. In fact, a poll conducted by Wufoo (now part of SurveyMonkey) found that 82% of developers preferred working with webhooks over API polling.
So Why Aren’t More People Offering Webhooks?
There are some cases where polling would be a preferable solution to webhooks. When updates are more frequent than the polling interval, it’s more efficient to get the updates in batches via polling. If you also don’t need the updates in real time, polling is a great solution.
The main thing blocking most people from offering webhooks is that they’re really hard to implement at scale. Unreliable user endpoints means you need automatic retries to ensure deliverability. Monitoring deliverability and notifying customers when their endpoints are broken is critical. There are many potential security vulnerabilites like server-side request forgery (SSRF) and replay attacks that you need to defend against. You need a way for your users to authenticate webhook events to ensure they are coming from you. You need a UI for your users to test, debug, and monitor their implementations.
You can see all the potential components and services needed to build a secure, reliable and scalable webhook system in this architecture diagram we created for Svix:
Even a well funded Silicon Valley startup with a strong engineering culture like Lob recently revamped their system using our webhook service. They were able to successfully simplify the design to make their engineers’ lives easier, increase scalability, add features to their dashboard, and make their cost structure more predictable.
Summary
Webhooks are becoming a must have feature for products that want to integrate with other systems. They’re very efficient for providing real time updates, provide an excellent developer experience and enable a lot of possible integrations through no/low code platforms like Zapier and Make.
Although they can be difficult to implement at scale, there are open source and hosted solutions that make it simple to offer a great webhook experience for your users.