NATS - An application connectivity technology (L7). It was originally designed for low-latency M:N messaging, and that is still true today. In 2018, native multi-tenancy, clustering options, etc. got introduced. The persistence subsystem (JetStream) was introduced in 2021. It has a completely different design than Kafka, but with overlapping use cases. For better or worse, we get compared to Kafka a lot and virtually everyone who engages realizes the advantages and opportunities. NATS is much more flexible for application use cases, for example, it provides KeyValue and ObjectStore abstractions on top of the core persistent stream. There are a plethora of other details, but that is the basic gist. Overall, it has a lot of batteries included for building everything from embedded applications to large scale cloud-to-edge systems.
Synadia - The founder (Derek) created NATS. We are the primary maintainers of the project. We build products and services on top of NATS including a global SaaS where you can sign up and use "hosted NATS" with additional features. We offer a BYOC model, one of which we manage for you, or a Kubernetes-based self-service one that you deploy yourself. We also support fully self-hosted for customers that need to run in their own data centers or at the edge.
Regarding the comment re: the website, there are improvements we have in the works. Happy to engage and help clarify anything that is confusing.
I don't know Simon personally, I do know he has been blogging for a very long time and many of his posts are for his own benefit of recall, not necessarily intended for an external audience. Also He did not post this, a different person did.
(Disclaimer: I am a NATS maintainer and work for Synadia)
The parent comment may have been referring to the fact that NATS has support for durable (and replicated) work queue streams, so those could be used directly for queuing tasks and having a set of workers dequeuing concurrently. And this is regardless if you would want to use Nex or not. Nex is indeed fairly new, but the team on is iterating on it quickly and we are dog-fooding it internally to keep stabilizing it.
The other benefits of NATS is the built-in multi-tenancy which would allow for distinct applications/teams/contexts to have an isolated set of streams and messaging. It acts as a secure namespace.
NATS supports clustering within a region or across regions. For example, Synadia hosts a supercluster in many different regions across the globe and across the three major cloud providers. As it applies to distributed work queues, you can place work queue streams in a cluster within a region/provider closest to the users/apps enqueuing the work, and then deploy workers in the same region for optimizing latency of dequeuing and processing.
Could be worth a deeper look on how much you could leverage for this use case.
If a learning resource is lacking or confusing, Slack or GitHub issues/discussions is a way to engage and provide feedback so they can be improved. If the docs were confusing, there are other channels to get help to unblock folks. The outcome of that interaction would lead to improvements in the docs.
> You're in your own little bubble and expect people to be mind readers.
Out of curiosity, have you asked questions in Slack or Github? If so, and you had a bad experience with the interaction, I get the sentiment and would offer help. But this comment is not constructive without context.
> I'm guessing that the documentation is bad so one has to pay for support.
The NATS project has been open source for around 12 years, and part of the CNCF since 2018. This is an incorrect statement and very poor assumption to make because the documentation doesn't make sense for you.
I 100% agree it can be improved and we are working on a new docs site, but it is not quite ready.
Yes that reads correct. The `sub` would a NATS user public nkey, the `iss` would be the NATS account public nkey (either the issuer nkey in config-mode or existing nkey in decentralized auth).
As long as it can verify the chain of trust for the user JWT that is returned, it should work.
Auth callout is post-NATS client authentication, so it would not solve the "auth web flow" for authentication. Instead, the resulting token from that would be set as a cookie that then would be passed into the nats.ws client connection. The auth callout service would use that token to map to the concrete NATS user. The mechanism of doing that is up to the implementation. One option is to manage NATS claims into the OIDC provider (for the user authenticating) and then the auth service would decode that source JWT and extract the NATS claims and generate the NATS user JWT in the response.
> Seems like I need to write a service utilizing NATS which talks to the OIDC server.
Auth callout was designed to be a generic extension point to delegate authentication and generate dynamic a user JWT that NATS understands (permissions, limits, etc). It enables an arbitrary backend to be integrated with, not tied specifically to OIDC. But indeed, this requires implementing a service that does this integration.
> NATS auth story is a complicated one, and now with auth callout it's even more complicated.
There is a spectrum of auth options, starting with simple config-based, token or user/pass leading up to decentralized auth for use cases that need it. Auth callout is an opt-in thing, so it should only be adopted if it is truly necessary.
> you have to have dedicated functions which you would call in the request handler
[...] instead of being able to define a chain of middleware functions
I don't quite understand this statement. Wrapping a NATS handler is the same approach as wrapping an HTTP handler (within the same client app). The function would take a handler and return a handler. There can be inspection of the message within that function and the choice of calling the next handler, responding early, doing some external call, or doing nothing.
If you want to distribute this middleware, then you would need to have separate, explicit subjects that each middleware would subscribe to and then publish to for the next element in the chain (for a choreography approach).
There is also the "message slip pattern" where the ingest component sets the path as metadata (e.g. headers) that each middleware component uses that to know (at request time) which subject to publish the result to next.
> If messages were frequent enough, and/or snapshots took a while to compute/transfer, you might never have a roll-up succeed.
Yes, good point. The "snapshotting up to a lagging sequence" could be achieved with two separate subjects to reduce contention, but is a bit more work.
It sounds like, in Drift's case, the snapshot effectively brings up the tail (snapshot), but the head can still be appended to with new events.
#1: This isn't a hard requirement to achieve the desired permissions. For the DriftDB use case, my understanding is that all members in the room have full pub/sub k/v permissions, so that could be achieved by declaring a new permission pinned to the room when the room is created or joined (this can be done dynamically without a config file reload).
#2 Publishes do support optimistic concurrency control using the `Nats-Last-Expected-Sequence` (stream level) or `Nats-Last-Expected-Subject-Sequence` for the subject-level. This ensures to concurrent publishes will be serialized and all but one is rejected with "conflict wrong sequence" error. For example headers in Rust[0] and WS[1]
It is a top priority. I do appreciate the constructive criticism and conversation. Feel free to reach out to me (byron at synadia dot com) or on the NATS Slack (https://slack.nats.io).
> Would appreciate it if you disclosed your affiliation with Synadia and NATS before advertising it.
Fair point, but I did not mention Synadia. FWIW, I have been a NATS user for seven+ years prior to joining Synadia so I was speaking on behalf of myself and experience with the tech.
Also fair point that the nats.io website does not highlight this strongly. The NATS maintainers are aware (nearly all employed by Synadia) and we are working on it.
I disagree that simply because it is not advertised as a "edge" technology that it is not one of the best-in-class techs for edge. It simply means, we are doing a poor job at awareness.
> Why don't you start comparing yourself to products and technology that serve the edge
The vast majority of people and customers compare NATS to Kafka and the variety of variants out there. Once the push on edge occurs, I suspect comparison to these other tech will occur.
To be clear, I am not looking for a "winner" in this discussion, rather my original comment was to correct a gap in understanding of what NATS is capable of.
> Even with NATS jetstream, NATS has a focus on service communication.
It indeed excels at service communication as well. However, a core use case for NATS is the edge, be it your definition (browsers and mobile), but also in cars, factories, tractors, low-orbit satellites, etc, whether it is running on Kubernetes, k3s, or bare metal.
The issue you called out is a Firefox-specific issue, but it will be addressed and not indicative of an inherit limitation of NATS.
Was not aware of Plane, nice! Regarding the two points:
- A unique room ID/subject is a form of authentication. Essentially anyone having that unique identifier can join, akin to a token. This is straightforward to setup in NATS avoiding the ">" for all problem (which I may now need to write a blog post about ;-)
- Rollups are supported on a per-subject basis. Each room could be modeled as a subject and individually rolled up.
In fact it does all of these things now properly! STAN (NATS Streaming) was deprecated two years ago in favor of a new embedded subsystem called JetStream: https://docs.nats.io/nats-concepts/jetstream released in March 2021.
Nice, have you come across NATS? https://nats.io. The server natively supports WebSockets. There are many clients including Deno, Node, WebSockets, Rust, Go, C, Python, etc.
In addition to stateless messaging, it supports durable streams, and optimized API layers on top like key-value, and object storage.