REST Hooks - Stop the polling madness(resthooks.org)
resthooks.org
REST Hooks - Stop the polling madness
http://resthooks.org/
34 comments
Same for me i miss the How? part.
And i got oversold on an idea that sounds good but i have no clue how it works.
http://en.wikipedia.org/wiki/Webhook
And i got oversold on an idea that sounds good but i have no clue how it works.
http://en.wikipedia.org/wiki/Webhook
We are pushing some copy updates to the homepage, most of the technical details were in docs. TL;DR coming to the homepage soon, but the gist is:
REST Hooks itself is not a specification, it is a collection of patterns that
treat webhooks like subscriptions. These subscriptions are manipulated via a
REST API just like any other resource. That's it. Really.
Better. That's practically the only thing that needs to be on the home page.
You also need to explain who you are and why you're doing this, too. Right now it looks like you have an interest in selling something but you never explain what it is. It reminds me a lot of those sites which sell some miracle product, and are deliberately very long and vague, and it ends finally with some button to order an ebook or something.
You also need to explain who you are and why you're doing this, too. Right now it looks like you have an interest in selling something but you never explain what it is. It reminds me a lot of those sites which sell some miracle product, and are deliberately very long and vague, and it ends finally with some button to order an ebook or something.
At the bottom of the page I saw:
> An initiative by Zapier 2013.
Zapier is a company that makes money off sites having REST endpoints to push and pull data. It looks like Zapier would prefer that more sites have endpoints to accept push, and this marketing effort is a brand-campaign (think Public Service Announcement) to get more people to tailor their products to work with Zapier.
That explains why more emphasis was put on marketing, etc. The primary audience is not developers but PHBs and PMs to have sound bytes to parrot to developers.
(I'm not cynical, I work in advertising, most of my products already have 'REST hooks'...)
> An initiative by Zapier 2013.
Zapier is a company that makes money off sites having REST endpoints to push and pull data. It looks like Zapier would prefer that more sites have endpoints to accept push, and this marketing effort is a brand-campaign (think Public Service Announcement) to get more people to tailor their products to work with Zapier.
That explains why more emphasis was put on marketing, etc. The primary audience is not developers but PHBs and PMs to have sound bytes to parrot to developers.
(I'm not cynical, I work in advertising, most of my products already have 'REST hooks'...)
As I read through the website, I felt mislead by the title "REST Hooks - Stop the polling madness" in that I believed we were talking directly about reducing something like AJAX long polling for an alternative means of retrieving data from the backend.
Having felt that pain before, and not opting for WebSockets yet, I was hoping for some kind of simple alternative. I have a portion of code that occurs during a registration process that is currently polling the backend for status updates as to the worker queue's progress (it's a lengthy registration with several moving pieces and calls to multiple remote APIs).
Having felt that pain before, and not opting for WebSockets yet, I was hoping for some kind of simple alternative. I have a portion of code that occurs during a registration process that is currently polling the backend for status updates as to the worker queue's progress (it's a lengthy registration with several moving pieces and calls to multiple remote APIs).
Try playing with WebSockets, they're pretty easy to get going. Is there something about them keeping you away? Or would it require quite a re-architecture?
There's a lack of browser support (IE10+, no Android), and the fallbacks drain your mobile battery. It's just not quite there. I've done full presentations on WebSockets and toyed/demoed with it for a number of trivial and less than trivial applications, so I would like to think I'm more informed than most.
server sent events? although they are push only. I can't tell if you also want write capabilities
Bingo!
Good point. For us (Zapier), we just waste a lot of time and resources polling. We talked to some of the SaaS services we integrate with and it is the same on their side as well. That is our motivation.
I have no idea how this solves the fundamental push/pull networking problem.
Is a network connection kept open? Is there an assumption that the user has some port open that can be contacted?
I see that you somehow reduced your calls, but I don't see how. Please tell me in actual socket level networking terms how this is done; who establishes a connection to what, for what, and for how long?
Also how do you deal with disconnects, timeouts, and missed realtime data.
Is a network connection kept open? Is there an assumption that the user has some port open that can be contacted?
I see that you somehow reduced your calls, but I don't see how. Please tell me in actual socket level networking terms how this is done; who establishes a connection to what, for what, and for how long?
Also how do you deal with disconnects, timeouts, and missed realtime data.
I think they are just suggesting that services which are being polled frequently instead implement a rest-like subscription service which would accept an endpoint and then push the data to all of the subscribed endpoints when it changes. In many cases it is more work from the developers of those services but would reduce their bandwidth and server load immensely.
Ohhh, so it's really a server-side technology to better consume third-party APIs. For whatever reason, this wasn't at all obvious to me. I had initially assumed that this was technology between client and server sides of a single-page application.
As did I. I've built these systems (socket-like systems that work efficiently across browser in diverse environments). And if you think you can just get that right the first time, then either you have the hacking skills of which I have never seen or you are (more likely) wrong.
But this thing here they are talking about is something I've been doing for a long time. I didn't think about making a brochure site for it though.
Just follow the principle of making the parts as dumb as possible. They should be the stupidest simplest things ever. Sometimes you can't get to this model on the first try, and that's ok. But the closer you get to it, the better the product becomes. I swear this is true and part of the soul of the machine's ghost.
If you can't make things laterally connect this way, then make them vertically stack this way and then laterally connect this way. But don't obfuscate the objective.
But this thing here they are talking about is something I've been doing for a long time. I didn't think about making a brochure site for it though.
Just follow the principle of making the parts as dumb as possible. They should be the stupidest simplest things ever. Sometimes you can't get to this model on the first try, and that's ok. But the closer you get to it, the better the product becomes. I swear this is true and part of the soul of the machine's ghost.
If you can't make things laterally connect this way, then make them vertically stack this way and then laterally connect this way. But don't obfuscate the objective.
Thats what I thought too.
This is a very good summary.
Server A has events Server B wants to know about. With REST Hooks, Server A provides an API that Server B use to tell Server A "Please POST notifications about those events to this URL". Then Server B keeps a web server listening for requests to that url. There is no persistent connection.
As for timeouts, check out http://resthooks.org/docs/retries/ for ideas on how those problems can be solved.
As for timeouts, check out http://resthooks.org/docs/retries/ for ideas on how those problems can be solved.
So it's an API to configure webhooks on Server A?
I think it's more of an API to allow Server B to register its interest in receiving notifications from Server A.
Of course, Server A must "speak Rest Hook" in order to enable that, so in a manner of speaking I guess it is a way to configure Webhooks on Server A.
But, it basically exposes a common interface for Server A and Server B to establish Webhooks. Server A can tell Server B which notification subscriptions are available (i.e. for which events), and Server B can then choose to subscribe, providing the callback URL for the Webhook.
Of course, Server A must "speak Rest Hook" in order to enable that, so in a manner of speaking I guess it is a way to configure Webhooks on Server A.
But, it basically exposes a common interface for Server A and Server B to establish Webhooks. Server A can tell Server B which notification subscriptions are available (i.e. for which events), and Server B can then choose to subscribe, providing the callback URL for the Webhook.
[deleted]
Why make effort to explicitly state that it's not a specification? That makes it more confusing, as it really is defining a standard interface for exposing and establishing Webhooks--i.e. pretty much a spec. So, when I read that it's not a spec, it made me question for a second whether I truly understood it.
I also agree with a previous commenter who said it's unclear what you guys' role is. Are you just defining this "spec" to help the world, and hoping that the world gets on board? Or, are you offering some sort of service? Also, how would a dev just start using these, when it requires that the services they are consuming "speak Rest Hook?". It just seems like a good idea, but not immediately useful. If there was a simple call to action that said, "Hey devs, let's all use these when building services", then the pitch would be more understandable (assuming that's what you guys are promoting).
BTW, I do think it's a worthwhile idea. Personally, I've found Webhooks to be very simple, thus I'm not really sold on the "research" that indicates devs struggle with them more than they would Resthooks. So, that's not the sell to me. It's having a standard, programmatic way of discovering Webhook availability and consuming them across various services/APIs that's the draw in my book.
Anyway, good luck.
I also agree with a previous commenter who said it's unclear what you guys' role is. Are you just defining this "spec" to help the world, and hoping that the world gets on board? Or, are you offering some sort of service? Also, how would a dev just start using these, when it requires that the services they are consuming "speak Rest Hook?". It just seems like a good idea, but not immediately useful. If there was a simple call to action that said, "Hey devs, let's all use these when building services", then the pitch would be more understandable (assuming that's what you guys are promoting).
BTW, I do think it's a worthwhile idea. Personally, I've found Webhooks to be very simple, thus I'm not really sold on the "research" that indicates devs struggle with them more than they would Resthooks. So, that's not the sell to me. It's having a standard, programmatic way of discovering Webhook availability and consuming them across various services/APIs that's the draw in my book.
Anyway, good luck.
This is definitely better, but what I would expect is a short explanation of what happens to REST requests - why are there fewer of them? Are you caching the responses, serving them to clients (who are still polling, but they are polling you instead of the REST server) and waiting for PUSH request from server to invalidate the cache? Or is this a client-side solution too which allows a client to trigger a REST request only when data is available?
There is no middle intermediary, its just two servers. For example, Facebook & your app.
Instead of polling `facebook.com/api/posts` every 2/5/15/60 minutes, you'd set up a subscription for Facebook to ping you at `yoursite.com/hook.php`. The subscription would be managed under `facebook.com/api/subscriptions`.
Instead of polling `facebook.com/api/posts` every 2/5/15/60 minutes, you'd set up a subscription for Facebook to ping you at `yoursite.com/hook.php`. The subscription would be managed under `facebook.com/api/subscriptions`.
It would be interesting if browsers could accept HTTP requests.
[deleted]
This is how webhooks work in any API that has sufficiently embraced them (see Freshbooks for an example). Why are we pretending like this is something different?
And here I thought I was being dense.
Seems like it adds data that would normally be included in a poll response to requests that were going to be made anyway, and possibly adds requests that were normally going to be made independently to long polls, though I'm not sure about any of this.
If I am right, would be nice to explain this and how it compensates if there is not a "regular" request within a certain period of time. And well, to basically explain WTH REST hooks are....
Is the most important piece of information really: "REST Hooks itself is not a specification, it is a collection of patterns that treat webhooks like subscriptions"? That seems of tertiary importance at best. How about telling us what REST Hooks are and how they are that - not what they aren't from a marketing perspective.
This is targeting a technical audience. It shouldn't so drastically underestimate its readers.
Seems like it adds data that would normally be included in a poll response to requests that were going to be made anyway, and possibly adds requests that were normally going to be made independently to long polls, though I'm not sure about any of this.
If I am right, would be nice to explain this and how it compensates if there is not a "regular" request within a certain period of time. And well, to basically explain WTH REST hooks are....
Is the most important piece of information really: "REST Hooks itself is not a specification, it is a collection of patterns that treat webhooks like subscriptions"? That seems of tertiary importance at best. How about telling us what REST Hooks are and how they are that - not what they aren't from a marketing perspective.
This is targeting a technical audience. It shouldn't so drastically underestimate its readers.
You are not the only one. I read though it, clicked around, looked at the demos, thought about bookmarking it, got to the documentation page and saw this `/api/v1/subscription/`, thought That breaks HATEOAS then left.
I'm sure if I put more time into trying to understand what is going on I could figure it out, but to me, the message isn't that clear.
I'm sure if I put more time into trying to understand what is going on I could figure it out, but to me, the message isn't that clear.
Did you actually use HATEOAS? I'm currently designing an API and it's just way too much work and overhead to add links when everyone is just going to hardcode them anyway. Do you have any examples where HATEOAS actually prevents or solves problems? Do you have any tips on how to get people to follow the links instead of hardcoding?
The last API that I wrote used versioning in the headers and it saved us a ton of headaches when we updated our API. Our V2 upgraded some JSON fields of the API and our users who were on V1 were never affected. The funny the whole concept is that, you use it every day without even thinking about it. If you think about the web when you're looking at an PDF; What tells you if what version of the PDF file it is? Not the URL, it's the meta-data in the PDF tells you. You'll never see a link like:
Here is a more accurate description of the how you should design a REST API from the Author of the HTTP spec: http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hyperte...
server.com/files/a-cs3/important.pdf
server.com/files/a-cs4/important.pdf
server.com/files/a-cs5/important.pdf
server.com/files/important.pdf
Here is a more accurate description of the how you should design a REST API from the Author of the HTTP spec: http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hyperte...
I'm interested since my application publishes URLs just like that. What's wrong with it as far as HATEOAS is considered? How would it be better?
How does it break HATEOAS?
People argue that, with REST, versioning should be done in headers and not the url. Maybe that?
Nooe, HATEOAS is about links. The official-sounding thing that this would violate is the "server should be able to organize its own namespace."
That said, I don't think this actually mandates any URLs. I read that part as just a suggestion for how it could be done.
That said, I don't think this actually mandates any URLs. I read that part as just a suggestion for how it could be done.
Does that mean it should be possible to just have the link be "subscription"?
It means that the spec shouldn't specify URIs, and instead publish link relations.
(I'm not saying that I think it should be that way, just elaborating.)
(I'm not saying that I think it should be that way, just elaborating.)
Here's my basic understanding of it: This sounds like webhooks, but they have a set payload, so that the server that is sent the webhook just calls the rest endpoint it is given in the payload.
Basically we're taking webhooks and adding a RESTful subscription layer to it. This way users don't have to go copy and past webhook urls into other applications... the applications they use automatically create and destroy the subscriptions for them.
Yeah, it's pretty bad. I think it comes down to "assuming WebHooks, a suggested 'pattern' to set up the subscription programmatically".
Agree with you. If the audience is developers, give it to us in black in white instead of some new jargon and marketing blah.
I looked through several pages (waste of time) and I still can't figure out how this is real time without websockets, polling, server side events. No way I would use this service if they don't know who their audience is.
I looked through several pages (waste of time) and I still can't figure out how this is real time without websockets, polling, server side events. No way I would use this service if they don't know who their audience is.
Yep the missing part of the description is 'server-to-server', or 'client-with-a-listen-port (i.e. not a browser) to server'.
Unfortunately a lot of us developers see the term REST API and think a-ha, this is something I can consume in my browser. This isn't that, so confusion ensues.
Unfortunately a lot of us developers see the term REST API and think a-ha, this is something I can consume in my browser. This isn't that, so confusion ensues.
Agreed. I came here to ask if anyone actually understood anything about the product from the front page.
> REST Hooks itself is not a specification, it is a collection of patterns that treat webhooks like subscriptions.
Bummer. No matter if you like it or not, a collection of patterns with a name _is_ a specification, just possibly a poorly defined one. See the confusion in this thread. If it were a link to a spec, nobody would be confused.
> Skip the pedantic arguments about standards and implementation details
This reads to me as "everyone is going to have a slightly incompatible implementation. One library wont be good enough, I'll need to write a new one per site that uses this."
Furthermore, what about PubSubHubbub?
Finally, polling is great: Ive found few situations where it doesn't work well, people just tend to only do the most basic of implementations and blame polling. See http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-d... for a really interesting example.
EDIT: numbers were real, added a supporting link
Bummer. No matter if you like it or not, a collection of patterns with a name _is_ a specification, just possibly a poorly defined one. See the confusion in this thread. If it were a link to a spec, nobody would be confused.
> Skip the pedantic arguments about standards and implementation details
This reads to me as "everyone is going to have a slightly incompatible implementation. One library wont be good enough, I'll need to write a new one per site that uses this."
Furthermore, what about PubSubHubbub?
Finally, polling is great: Ive found few situations where it doesn't work well, people just tend to only do the most basic of implementations and blame polling. See http://roy.gbiv.com/untangled/2008/paper-tigers-and-hidden-d... for a really interesting example.
EDIT: numbers were real, added a supporting link
Hey Steve!
> See the confusion in this thread. If it were a link to a spec, nobody would be confused.
We definitely had a little confusion in the thread, I think that was mostly because we put too much marketing on the homepage for this audience, but the absence of a proper spec could have definitely contributed. We're correcting some of this.
This is all about adoption of some sort of subscription-based HTTP callback: at this point in time any flavor will do. We have no doubt that a formal spec will pop up someday (and that would make us very happy!).
> Furthermore, what about PubSubHubbub?
PubSubHubbub always seemed a little heavy and was a departure from APIs that weren't XML/ATOM based (meaning most JSON API providers wouldn't touch it). Not saying I agree with it, but that is the feedback we got.
> Finally, polling is great for 99% of use cases (I can make up statistics, too)
We posted the numbers driving our stats on the homepage. They are not made up and come straight for our Elasticsearch cluster. I'm happy to elaborate on them!
> See the confusion in this thread. If it were a link to a spec, nobody would be confused.
We definitely had a little confusion in the thread, I think that was mostly because we put too much marketing on the homepage for this audience, but the absence of a proper spec could have definitely contributed. We're correcting some of this.
This is all about adoption of some sort of subscription-based HTTP callback: at this point in time any flavor will do. We have no doubt that a formal spec will pop up someday (and that would make us very happy!).
> Furthermore, what about PubSubHubbub?
PubSubHubbub always seemed a little heavy and was a departure from APIs that weren't XML/ATOM based (meaning most JSON API providers wouldn't touch it). Not saying I agree with it, but that is the feedback we got.
> Finally, polling is great for 99% of use cases (I can make up statistics, too)
We posted the numbers driving our stats on the homepage. They are not made up and come straight for our Elasticsearch cluster. I'm happy to elaborate on them!
Its true, thats a large part of the issue. If you put so many links to /doc om your homepage, well...
Even then, this page (/docs) barely says anything. Almost anything can be 'compliant,' because there's almost nothing to say. I really think some rigor would help a lot. I can appreciate not wanting to get into full RFC2119 right away, but you need some amount of description.
PuSH is not limited to a XML/ATOM, and companies like SuperFeedr use PuSH with JSON.
In fact, I'm pretty sure PUSH is 98% of a webhooks implemenation.
I didnt understand that the numbers were from anything real, my mistake and apologies.
Even then, this page (/docs) barely says anything. Almost anything can be 'compliant,' because there's almost nothing to say. I really think some rigor would help a lot. I can appreciate not wanting to get into full RFC2119 right away, but you need some amount of description.
PuSH is not limited to a XML/ATOM, and companies like SuperFeedr use PuSH with JSON.
In fact, I'm pretty sure PUSH is 98% of a webhooks implemenation.
I didnt understand that the numbers were from anything real, my mistake and apologies.
GitHub implements the PSHB subscription API. It's fine, but doesn't really fit in well with the rest of our JSON API. For instance, parameters are sent over as form encoded values like "hub.mode".
http://developer.github.com/v3/repos/hooks/#pubsubhubbub
We also have a regular JSON endpoint for our hooks resource (which is essentially a subscription API).
http://developer.github.com/v3/repos/hooks/#pubsubhubbub
We also have a regular JSON endpoint for our hooks resource (which is essentially a subscription API).
Let's make it better :) The PubSubHubbub mailing list is still open and would welcome your comments on how to make things better!
This isn't impressive. [MQTT][1] is a much better protocol, and an open standard. It's a pub/sub that runs over TCP with variants that run over UDP & TLS. There's only a 2+ byte overhead, so it's very light. It's ideal for mobile devices because it has a QoS system built-in, so the developer doesn't have to think through the problem of missing messages where cell service is sub-optimal. MQTT is even resource-based, so it fits well with REST. I wish more people would start using MQTT instead of these crazy hacks like REST Hooks.
[1]: http://mqtt.org/
[1]: http://mqtt.org/
If for some reason you can't or don't want to implement webhooks, at least make sure you the GET endpoint for any object has a query param that supports fetching the most recently updated or created objects and supports pagination.
It sounds trivial, but you'd be surprised how many APIs don't support one or both of those features. When you're writing an API it might seem unnecessary to start (after all, who could ever have 1000s of <object>?), but if someone ends up polling your API frequently, having those two features can reduce a lot of unnecessary load for both you and the poller. And, of course, make sure you have an index on the created and/or updated dimensions.
That said, webhooks are terrific. Few things to consider when implementing them:
- Think carefully about the payload you send to the webhook. It's usually a good idea to send some related objects/data because many times when someone gets a webhook payload, that'll trigger calls to your API to get related information you could've reasonably sent them in the initial payload.
- You'll likely want to some way to keep track of errors so if an endpoint starts returning 404s or 500s you have a way to programmatically discard it after X failed attempts.
- In your docs, give sample, "real world" payloads developers can test against. It saves times over creating a RequestBin, pushing there, copying, cURLing, etc. (Remember, you can't set up a webhook to localhost.)
- A nice to have is some sort of retry capability with an exponential back-off. Servers go offline and if they get pushed webhook data then, those messages are lost. You could say, "tough, it's the consumer's responsibility," but if having all the data is important, most people will resort to polling. (Somewhat related, you'd be surprised how often the APIs of some larger SaaS companies are "offline" -- e.g. returning 503 --, so these things do happen.)
It sounds trivial, but you'd be surprised how many APIs don't support one or both of those features. When you're writing an API it might seem unnecessary to start (after all, who could ever have 1000s of <object>?), but if someone ends up polling your API frequently, having those two features can reduce a lot of unnecessary load for both you and the poller. And, of course, make sure you have an index on the created and/or updated dimensions.
That said, webhooks are terrific. Few things to consider when implementing them:
- Think carefully about the payload you send to the webhook. It's usually a good idea to send some related objects/data because many times when someone gets a webhook payload, that'll trigger calls to your API to get related information you could've reasonably sent them in the initial payload.
- You'll likely want to some way to keep track of errors so if an endpoint starts returning 404s or 500s you have a way to programmatically discard it after X failed attempts.
- In your docs, give sample, "real world" payloads developers can test against. It saves times over creating a RequestBin, pushing there, copying, cURLing, etc. (Remember, you can't set up a webhook to localhost.)
- A nice to have is some sort of retry capability with an exponential back-off. Servers go offline and if they get pushed webhook data then, those messages are lost. You could say, "tough, it's the consumer's responsibility," but if having all the data is important, most people will resort to polling. (Somewhat related, you'd be surprised how often the APIs of some larger SaaS companies are "offline" -- e.g. returning 503 --, so these things do happen.)
You can send a webhook to localhost, just in a roundabout way using one of the local tunneling solutions: http://john-sheehan.com/blog/a-survey-of-the-localhost-proxy...
Great points though.
Great points though.
It is awfully difficult to get a technical TL;DR version from a website, whose target audience is programmers. I think explaining what this does concisely and technically, preferably on the front page with code examples etc would be really useful.
We also wrote a quick drop-in Django application [0] (that plugs right into TastyPie if you have an API built on it) . We also have a sample CRM with a live API running [1]. Our friends at NetEngine wrote a sample Rails app [2], and our very own James wrote an awesome Node.js Sails app [3].
[0] https://github.com/zapier/django-rest-hooks
[1] http://demo.resthooks.org/
[2] https://github.com/net-engine/resthooks
[3] https://github.com/zapier/node-resthooksdemo
[0] https://github.com/zapier/django-rest-hooks
[1] http://demo.resthooks.org/
[2] https://github.com/net-engine/resthooks
[3] https://github.com/zapier/node-resthooksdemo
Still scratching my head on the Ruby "reference" implementation. When I subscribe resource_subscriptions, who is receiving the update notification? Is the client side js code getting a push? Is the resource_subscriptions a long-polling endpoint? Not very clear at all.
The server itself will just POST a payload to a defined URL. I'm not wildly familiar with the internals of that app or Rails in general, otherwise I'd give more detail!
So all this is just about the following pattern?
A wants to get updates of B. Instead of polling B for changes, it sends B or C an URL of A and says: POST to this URL the moment something changes on B.
If that's the case, this really gets lost on all the hooks/subscriptions/load-mumbojumbo on this site.
A wants to get updates of B. Instead of polling B for changes, it sends B or C an URL of A and says: POST to this URL the moment something changes on B.
If that's the case, this really gets lost on all the hooks/subscriptions/load-mumbojumbo on this site.
That is the case! It's actually a hard concept to get across in words (we've found). If you have or see an example that really resonates with you, do let us know (and we're open to pull requests https://github.com/zapier/resthooks :-).
I think what is missing is the explanation of a subscription. On the github-page, there is only one sentence in that regard:
> REST Hooks are a lightweight subscription layer on top of your existing REST API.
That is too short. There the explanation is missing, the same way on the website. maybe in better english if mine is broken, but that addition could fix it: "...,thus that instead of having to poll that API regularly, the subscriber is notified with a POST the moment a change occurs."
I'm probably biased, because that is kind of the same way I once tried to explain the concept, but I think it would be clearer that way.
> REST Hooks are a lightweight subscription layer on top of your existing REST API.
That is too short. There the explanation is missing, the same way on the website. maybe in better english if mine is broken, but that addition could fix it: "...,thus that instead of having to poll that API regularly, the subscriber is notified with a POST the moment a change occurs."
I'm probably biased, because that is kind of the same way I once tried to explain the concept, but I think it would be clearer that way.
Thanks! We'll try to work that into the next round of edits!
Oh neat! We're launching resthooks.org today with several partners to promote an existing but underused real-time API pattern.
Here are more resources:
Intro: https://zapier.com/engineering/introducing-resthooksorg/
Why: http://resthooks.org/#why
Demo: http://demo.resthooks.org/
Here are more resources:
Intro: https://zapier.com/engineering/introducing-resthooksorg/
Why: http://resthooks.org/#why
Demo: http://demo.resthooks.org/
I'm looking for a quick response to "How", w/o going through code. Care to enlighten me?
http://resthooks.org/docs/ has a section titled "Minimum Implementation Walkthrough" that looks to be what you're looking for.
This is neat. Similar in spirit to http://jsonapi.org .
Has any thought been given to the concept of supporting the same framework over something useful in a browser? http://resthooks.org/docs/alternatives/ lists some problems with the common methods, and rightfully so, but I don't see a different recommendation.
Has any thought been given to the concept of supporting the same framework over something useful in a browser? http://resthooks.org/docs/alternatives/ lists some problems with the common methods, and rightfully so, but I don't see a different recommendation.
I hate to be THAT guy but someone was going to say it so it may as well be me :)
This is a collection of patterns, right? Well why not make this really REST and rather than list a bunch of URL templates provide REL types for each of these like so:
REL subscriptions-list -> GET subscription-create -> POST subscription-read -> GET subscription-update -> PUT subscription-destroy -> DELETE
Now it doesn't matter what the URL structure is, I can pull the <link> elements from the page and be TOLD the URL, rather than follow a URL template. That way, this doesnt rely on out of band knowledge i.e this web page and its (poor) description.
This is a collection of patterns, right? Well why not make this really REST and rather than list a bunch of URL templates provide REL types for each of these like so:
REL subscriptions-list -> GET subscription-create -> POST subscription-read -> GET subscription-update -> PUT subscription-destroy -> DELETE
Now it doesn't matter what the URL structure is, I can pull the <link> elements from the page and be TOLD the URL, rather than follow a URL template. That way, this doesnt rely on out of band knowledge i.e this web page and its (poor) description.
[deleted]
Has anyone use PubSubHubbub ( http://en.wikipedia.org/wiki/PubSubHubbub ) for this pattern in the past? For straight subscriptions, it's always been compelling to think of the use cases being covered by PSHB and RSS.
It is pretty fantastic.
So, PubSubHubbub is not linked with RSS anymore and works with any type of data format, including JSON obviously :)
The quick win of PubSubHubbub compared to Resthook is that it has a 'security' mechanism in it, which means that the subscriber can be 100% sure that the service sending the notification is not someone impersonating it.
Aside from this, I feel like PubSubHubbub and Reshook solve the same problem: programatically setting up webhooks.
The quick win of PubSubHubbub compared to Resthook is that it has a 'security' mechanism in it, which means that the subscriber can be 100% sure that the service sending the notification is not someone impersonating it.
Aside from this, I feel like PubSubHubbub and Reshook solve the same problem: programatically setting up webhooks.
This sounds like webhooks, but they have a set payload, so that the server that is sent the webhook just calls the rest endpoint it is given in the payload. Interesting idea, but it does sound like a regular webhook to me. That flow is the same way I use the Stripe webhooks for instance. I use the event id and make a call to Stripe directly, so that I can make sure the event is real.
I don't really understand what this is. Is this an attempt at a formalization of webhooks + JSON?
Same here - I keep coming back to "are they just talking about webhooks? or are they also talking about some kind of persistent http streaming api?"
I'm thinking these are just Webhooks, but the REST part is throwing me off because I think of it more as a consumption concept (consuming resources, etc).
Update: Found this, which I think explained it best: "REST Hooks itself is not a specification, it is a collection of patterns that treat webhooks like subscriptions. These subscriptions are manipulated via a REST API just like any other resource. That's it. Really." - but it was in /docs, not the main page.
I'm thinking these are just Webhooks, but the REST part is throwing me off because I think of it more as a consumption concept (consuming resources, etc).
Update: Found this, which I think explained it best: "REST Hooks itself is not a specification, it is a collection of patterns that treat webhooks like subscriptions. These subscriptions are manipulated via a REST API just like any other resource. That's it. Really." - but it was in /docs, not the main page.
Its not. Its promoting two things specifically:
1. Webhooks
2. A subscription layer via REST
Several major players already are doing this but it doesnt have a name.
REST Hooks are a way to consolidate that momentum and push it to a broarder audience.
1. Webhooks
2. A subscription layer via REST
Several major players already are doing this but it doesnt have a name.
REST Hooks are a way to consolidate that momentum and push it to a broarder audience.
It is just a pattern around making traditional webhooks another resource on your existing REST API so you can programmatically control subscriptions/callbacks/webhooks/etc. instead of asking users to copy-paste URLs.
Thats what it appears to be, but on closer inspection it doesn't go that far. They give an example rest path of /subscriptions but i cant tell if its an example or meant to be used as a standard endpoint name.
The way the info is presented bugs me. The first "case study" is Zapier. Later, in the "Learn how..." set of links, Zapier is listed as one of many. Look to the footer, and the Github link, and it's of course evident it's a Zapier project.
This project (er hmm, "initiative") is core to Zapier's business. If every service out there had a hook atop their service, it'd make things a lot easier for Zapier. That's cool. What bugs me is the feeling that Zapier's branding in the whole thing is less than transparent.
It seems that open source projects are getting more and more marketing driven, and the way this "initiative" is packaged is a sign of things to come.
This project (er hmm, "initiative") is core to Zapier's business. If every service out there had a hook atop their service, it'd make things a lot easier for Zapier. That's cool. What bugs me is the feeling that Zapier's branding in the whole thing is less than transparent.
It seems that open source projects are getting more and more marketing driven, and the way this "initiative" is packaged is a sign of things to come.
I don't see how a company being involved with this "initiative" affects the core concept - it's a net benefit for both API consumer and producer in moving to a subscribe/push model. Less requests, less load, closer to real time updates. Swap Zapier with any other consumer and the result is the same
Disclosure - had their name been at the top somewhere, and said "we" instead of "Zapier" it would boost the credibility of the project.
Curious if anybody can answer a question for me. I've recently started building some apps with MeteorJS, and am having loads of fun with it. At first I'd been a bit suspicious of its magic pixie dust -- seemed too good to be true -- but so far it's done everything it promised, and has allowed me to focus on developing the parts of the app I care about. After checking briefly to see that my data syncing was genuinely low-latency and low-bandwith (which under every test I've subjected it to, it appears to be), I've pretty much forgotten about this level of engineering and have focused on bigger-picture stuff.
Which brings me to the question: should I care about something like REST Hooks? Is it a mistake to become complacent and assume that the lower-level infrastructure will Just Work? Or can MeteorJS (and presumably other high-level frameworks) be trusted to handle this kind of stuff in a way that makes it safe for me to forget about it?
Just curious what the esteemed HN denizens think about this, as I'm sure there are some strong and reasonably well-informed opinions out there...
Which brings me to the question: should I care about something like REST Hooks? Is it a mistake to become complacent and assume that the lower-level infrastructure will Just Work? Or can MeteorJS (and presumably other high-level frameworks) be trusted to handle this kind of stuff in a way that makes it safe for me to forget about it?
Just curious what the esteemed HN denizens think about this, as I'm sure there are some strong and reasonably well-informed opinions out there...
I think, if anything, Meteor proves it's possible that it can be the responsibility of the framework, not the developer.
The docs talk about "event types" like contact.update which do not exist in HTTP and require knowing the semantics of a particular group of resources. To truly be RESTful, I would want "please notify me of any PUT, PATCH, or DELETE to this list of URLs, or POST creating a URL that matches this pattern." And if you're going to include a copy of the resource (I don't know what else a payload would be) you need the media type the subscriber expects as if you were doing content negotiation for a GET request from them. Or maybe the payload could just be the current result of a HEAD request so the subscriber can invalidate their caches, and they can send the appropriate GET request(s) when and if they care to.
There should be some standard link relation like <link rel="subscription"> so clients have some hint that this is available and where to request it.
I'd also want some way to manage the freshness/load tradeoff, like "please notify me within one hour but no more than every five minutes".
There should be some standard link relation like <link rel="subscription"> so clients have some hint that this is available and where to request it.
I'd also want some way to manage the freshness/load tradeoff, like "please notify me within one hour but no more than every five minutes".
Doesn't this assume that the API will only be used from a module living under a URL?
Most REST apis these days seem to be consumed form client side.
Most REST apis these days seem to be consumed form client side.
The focus is on server-server communication. A client like a web browser would need to use a different approach to get notifications, but there are options in that space besides polling.
Give a man a hammer, everything looks like a nail. REST has become a bit of a hammer. The reason a lot of RESTful designed software I've seen has such poor performance is because REST is used like a hammer... everywhere! And whole bunch of places it's not needed and it's not ideal.
I hate to be THAT guy but someone was going to say it so it may as well be me :)
This is a collection of patterns, right? Well why not make this really REST and rather than list a bunch of URL templates provide REL types for each of these like so:
REL subscriptions-list
This is a collection of patterns, right? Well why not make this really REST and rather than list a bunch of URL templates provide REL types for each of these like so:
REL subscriptions-list
Are there patterns for client side JavaScript to be notified of changes to REST resources? I don't believe this particular pattern applies if I understand correctly as it needs a URL to notify.
I think you'd have to have your server receive the hook, and then notify the client side using websockets/long polling etc. Pusher makes that really easy.
Websockets are still your best bet here.
I don't agree with this part:
> In other words, if everyone implemented REST Hooks, server load for both the sender and receiver could be reduced by 66x.
No, the number of requests could be reduced by a factor of 66. I'm not saying that's not impressive, I'm saying that the polling requests that ended up resulting in no action are cheaper than actionable requests, so, server load will go down by much less than a factor of 66x. The amount of work is the same, just busywork is less.
> In other words, if everyone implemented REST Hooks, server load for both the sender and receiver could be reduced by 66x.
No, the number of requests could be reduced by a factor of 66. I'm not saying that's not impressive, I'm saying that the polling requests that ended up resulting in no action are cheaper than actionable requests, so, server load will go down by much less than a factor of 66x. The amount of work is the same, just busywork is less.
That is a good point. What kills us the most is the latency on network IO when polling.
While the paradigm of having a webhook makes more sense, I personally hate working with them. I've yet to see a graceful way of handling local development that doesn't involve localtunnel (or some alternative) and constantly having to un-register and re-register endpoints every time localtunnel gives you a new hostname. I understand the point and would prefer working with them if there was just a good way to handle local development.
This is a great point, is there a better way to do local dev with webhooks? I always used localtunnel which is great it not perfect.
Some other options here: http://john-sheehan.com/blog/a-survey-of-the-localhost-proxy...
I personally think you should be using one that gives you permanent URLs (ngrok or Passageway) but that also keeps a log when you're client is disconnected. I'm pretty sure only our Passageway does that.
I personally think you should be using one that gives you permanent URLs (ngrok or Passageway) but that also keeps a log when you're client is disconnected. I'm pretty sure only our Passageway does that.
Powerful idea, should be made a specification. Simple enough too - what they describe is almost trivial to implement in /* insert your web framework of choice */. You wouldn't even need a library. Handle your HTTP verbs, store subscriber state, and call them out on events. It's nice.
Here's another (work in progress) spec for managing webhook/HTTP callback subscriptions: https://github.com/progrium/http-subscriptions/blob/master/S...
Why is this a thing? Your API interfaces with a message queue; your end client uses all the libraries and patterns built up over the years to consume from the queue. That doesn't sound all startup-y like "REST Hooks", but the pattern has been around for a long time.
I just want to point out that having read that entire page, I still have absolutely no idea what this is, how it works, etc.
I have a solution for polling. Its called websockets. Even more, there's a well-supported library called socket.io that transparently handles it for all browsers.
I have a solution for polling. Its called websockets. Even more, there's a well-supported library called socket.io that transparently handles it for all browsers.
After reading through all the comments here I still can't figure out what this is. Sorry.
This blog post should help, it helped me:
https://zapier.com/engineering/introducing-resthooksorg/
https://zapier.com/engineering/introducing-resthooksorg/
Isn't this like SUP?
http://blog.friendfeed.com/2008/08/simple-update-protocol-fe...
http://blog.friendfeed.com/2008/08/simple-update-protocol-fe...
FWIW http://barelyenough.org/blog/2009/05/push-or-pull/
case closed
> On average, 98.5% of polls are wasted
Erm. Is there a source for that?
Erm. Is there a source for that?
83% of statistics are made up on the spot.
A bit further down the page, they cite Zapier
"Over a representative time period, Zapier polled for changes 30 million times but only took action on 460,000 of those polls (1.5% efficient)."
"Over a representative time period, Zapier polled for changes 30 million times but only took action on 460,000 of those polls (1.5% efficient)."
It's a Zapier project, so they're citing themselves.
https://github.com/zapier/resthooks
https://github.com/zapier/resthooks
Ah, I must've missed that. Thanks!
[deleted]
TL;DR - REST Hooks is an API pattern where the API provides an endpoint from which webhooks can be programmatically setup and removed.
This sounds pretty much like the polyfill Socket.IO/SockJS do to create a socket atop polling-based transports.
I have no idea what this is.
Someone should show this to Facebook.... Just saying...
If someone has a better understanding of how this works, I would appreciate a TL;DR version for programmers (which are probably the target audience?). Thanks!