Should You Build a Webhooks API?(brandur.org)
brandur.org
Should You Build a Webhooks API?
https://brandur.org/webhooks
9 comments
As using webhooks for git repositories is probably the most common usage, it would be really nice if the different APIs converged here. Now everyone is inventing their own (and I'm in fact guilty to this myself for a side-project).
I guess the section on Retries is closest, but we're looking at moving from a system with straight in-process hooks to one using webhooks, and a complication is we now have to provision a webhook endpoint with very high reliability to accept the callback and (effectively) call the old hook.
The thing is, all the hook does is put a message on a queue…
The thing is, all the hook does is put a message on a queue…
I would love to see some of these APIs replaced by WebSockets. It would allow building your own client for some services much more easily while maintaining push notifications
Side note: their music is so good. Paired with their actual tech and vision, I think the music actually makes me want to work at SpaceX more than if paired with something less epic.
I swear I wrote this comment on the BFR story [1].
[1] https://news.ycombinator.com/item?id=15362999
[1] https://news.ycombinator.com/item?id=15362999
Good tips! I actually just worked on detecting dead webhook endpoints, as I've had a number of users use ngrok for local testing, create hundreds of events, and then disable ngrok--and much like Stripe, I retry with an exponential backoff over the course of a few days. Needless to say, my logs were filled with failed webhook request errors and I had to figure out how to limit that from happening in the future. Very interesting list of alternatives to webhooks as well.
When it comes to axing dead endpoints, I see two ways to handle this based on the way webhooks are configured.
* If webhooks are configured by UI, then you have to contact the user and tell them their webhook is gone, e.g. via email.
* If webhooks are configured programmatically, then the endpoints ought to be able to be swept up silently, and the receiver should know to periodically refresh the configuration for as long as it cares about it.
Ideally all webhook implementations would work the second way, but unfortunately it is not standard practice.
* If webhooks are configured by UI, then you have to contact the user and tell them their webhook is gone, e.g. via email.
* If webhooks are configured programmatically, then the endpoints ought to be able to be swept up silently, and the receiver should know to periodically refresh the configuration for as long as it cares about it.
Ideally all webhook implementations would work the second way, but unfortunately it is not standard practice.
For those interested in technically easy webhook consuming endpoint server check out https://github.com/adnanh/webhook
If you're using a web hook to notify your users of changed content, the WebSub standard (https://www.w3.org/TR/websub/; formerly PubSubHubbub; currently a candidate recommendation) is a good option. It solves a couple (though not all) of the problems of web hooks listed in the article, including:
- Uncertain security: signing notifications is in the spec
- Development and testing: ideally, with libraries that implement the spec, you can assume that the websub part will 'just work' and only need to test your handler function.
- Misbehavior is onerous: subscribers need to actively confirm their subscription every once in a while. This might seem complex, but it allows for periodic checks if the server/client is still functioning as expected, and ideally can be handled automatically by libraries.
- Retries: the spec leaves some room here, but again, a library can handle this.
As I decided to use WebSub as an alternative for a 'plain' webhook for a project recently, I wrote a library for the Flask web framework: https://github.com/marten-de-vries/Flask-Websub. I personally hope libraries for other frameworks will follow now the spec is moving less.
- Uncertain security: signing notifications is in the spec
- Development and testing: ideally, with libraries that implement the spec, you can assume that the websub part will 'just work' and only need to test your handler function.
- Misbehavior is onerous: subscribers need to actively confirm their subscription every once in a while. This might seem complex, but it allows for periodic checks if the server/client is still functioning as expected, and ideally can be handled automatically by libraries.
- Retries: the spec leaves some room here, but again, a library can handle this.
As I decided to use WebSub as an alternative for a 'plain' webhook for a project recently, I wrote a library for the Flask web framework: https://github.com/marten-de-vries/Flask-Websub. I personally hope libraries for other frameworks will follow now the spec is moving less.
how can someone stay knowledgable of all of the many reports and standards on w3.org? Do people visit there regularly?
I tried looking for websub on w3.org without using a search engine and it was a challenge (finally found it)
I tried looking for websub on w3.org without using a search engine and it was a challenge (finally found it)
I found out about this standard myself indirectly during a web search. As an outsider to the w3.org process, I think the best way is to follow working groups with themes you're interested in: https://www.w3.org/Consortium/activities. For example, WebSub is worked on by the Social Web Working Group: https://www.w3.org/TR/tr-groups-all#tr_Social_Web_Working_Gr....
We built Auth0 Extend [1] to address with many of these concerns. Our hypothesis is that the logic that implements these extensibility points should live in the host platform.
For several years now, we've had this sort of hook-via-code in the Rules engine on our Auth0 identity product. It's been such a powerful enabler for many different integration scenarios that we thought that other companies may find the technology useful for their platforms.
Since committing to building Auth0 Extend, we've seen Twilio launch their functions [2] and just today CloudFlare launched its Workers [3]. It seems like the this is an approach that is in its infancy but that is here to stay!
1: https://auth0.com/extend/developers
2: https://www.twilio.com/docs/api/runtime/functions
3: https://blog.cloudflare.com/introducing-cloudflare-workers/
For several years now, we've had this sort of hook-via-code in the Rules engine on our Auth0 identity product. It's been such a powerful enabler for many different integration scenarios that we thought that other companies may find the technology useful for their platforms.
Since committing to building Auth0 Extend, we've seen Twilio launch their functions [2] and just today CloudFlare launched its Workers [3]. It seems like the this is an approach that is in its infancy but that is here to stay!
1: https://auth0.com/extend/developers
2: https://www.twilio.com/docs/api/runtime/functions
3: https://blog.cloudflare.com/introducing-cloudflare-workers/
We are just in the process of launching something similar at Converse to allow users to extend our conversational workflow capabilities to add their own modules, events, including extending API endpoints and the core UI.
Definitely a good way of doing things, still looking for more beta testers but feedback is very positive so far.
https://get.converse.ai/v2.0/docs/plugin-development if anyone wants to kick the tires on it, all feedback welcome.
Definitely a good way of doing things, still looking for more beta testers but feedback is very positive so far.
https://get.converse.ai/v2.0/docs/plugin-development if anyone wants to kick the tires on it, all feedback welcome.
As the article alludes to APIs change over time, and so you can find the data that is submitted to you changes. In my case I setup an end-point that responded to push events from various code-hosting services.
I found that the three "main" git-hosts sent widely different data in their webhooks, and updates we often painful. (Github, bitbucket, and .. something else. I forget now.) Trying to have one end-point to handle all services was a recipe for frustration and failures. Even parsing out the basic details was hard:
* This is the SSH repository URL of the thing that was committed to
In the end I moved to self-hosting git-repositories, rather than allowing users to point at my webhook. That allowed hooks in the master-repos to fire.
(In my case I trigger DNS updates when git commits are made.)