DeepstreamHub: a faster Firebase with an open core(deepstreamhub.com)
deepstreamhub.com
DeepstreamHub: a faster Firebase with an open core
https://deepstreamhub.com/#main
9 comments
also: https://deepstream.io/info/performance/four-billion-messages...
direct benchmarks are something that Firebase' Terms and Conditions make hard to do
direct benchmarks are something that Firebase' Terms and Conditions make hard to do
Benches just aren't meant for this type of analysis.
Two different engines doing two different things.
Benches are for recording "personal best" stats, not "laps at the public pool". I.e. Testing against self and not others.
Two different engines doing two different things.
Benches are for recording "personal best" stats, not "laps at the public pool". I.e. Testing against self and not others.
Not exactly Firebase in that it doesn't provide an arbitrarily nested JSON structure as a datastore but more of a document-based approach. Overall a pretty good alternative to Firebase but there is no security rules, no default data store or auth so you need a lot more work. Lists are not very good being just a record with an array of record keys and not the actual data. I use the self-hosted version (deepstream.io) at work where we can't use Firebase because of company policy and regulations.
We switched from firebase to deepstream, and are hosting it ourselves.
With firebase since we were not running a node or java app, we ended up relying heavily on the rest interface. Which was sort of the not well taken care of. We would see crazy spikes and we could not get support to fix it. Multi tenancy would occasionally cause other performance issues.
We eneded up running a bit of a monkey patched system for awhile using kineses to pipe updates to a java/node app which then would write updates via the web socket interface.
Firebase also lacked support for pub/sub events, which can be sort of faked, but not entirely reliably.
Deepstream model is a bit different as you said, and it does have some weaknesses. I have submitted a number of patches already to fix some of the edge cases i've run into and will likely submit more.
Unlike firebase, we can actually directly improve the deepstream application core as well as build options into the platform that better support our use cases.
With firebase since we were not running a node or java app, we ended up relying heavily on the rest interface. Which was sort of the not well taken care of. We would see crazy spikes and we could not get support to fix it. Multi tenancy would occasionally cause other performance issues.
We eneded up running a bit of a monkey patched system for awhile using kineses to pipe updates to a java/node app which then would write updates via the web socket interface.
Firebase also lacked support for pub/sub events, which can be sort of faked, but not entirely reliably.
Deepstream model is a bit different as you said, and it does have some weaknesses. I have submitted a number of patches already to fix some of the edge cases i've run into and will likely submit more.
Unlike firebase, we can actually directly improve the deepstream application core as well as build options into the platform that better support our use cases.
Yes. Firebase needs your apps to written in JS or Java.
The lack of support for pub/sub was never a problem for me though since the channelRef.push(msg).remove() works 100% reliably.
Deepstreamhub member here: The platform, similar to Firebase supports multiple Auth strategies and has a fine grained permission language called Valve, similar to Firebase's Bolt. Please find an overview of security concepts here https://deepstreamhub.com/tutorials/guides/security-overview...
Dsh allows you to store arbitrary data, but breaks it down into smaller chunks a large Mongo or Couch db.
As these smaller chunks have their own life cycle lists make it easy to use them within iterators, e.g. For each in Vue etc, where every component manages bits own granular document
Dsh allows you to store arbitrary data, but breaks it down into smaller chunks a large Mongo or Couch db.
As these smaller chunks have their own life cycle lists make it easy to use them within iterators, e.g. For each in Vue etc, where every component manages bits own granular document
[deleted]
I like deepstream.io.
We use it in production. The use case is actually Server <-> Server communication and so far it works good.
We use it in production. The use case is actually Server <-> Server communication and so far it works good.
Very interesting, I like that it's open. PubNub was on my mind, but I would need to look into this. If anyone has any experiences with PubNub vs DeepstreamHub, I'd love to know.
So, is this equivalent of Realm Mobile Platform[1]?
I found realm to be convenient to use in mobile apps. Haven't tried the mobile platform yet. Anyone here can give a quick comparison of RMP and DeepstreamHub?
[1]: https://realm.io/products/realm-mobile-platform/
[1]: https://realm.io/products/realm-mobile-platform/
They're quite different. In particular, the Realm Mobile Platform is built on a full-blown database running on the user's device. One of Realm's biggest selling points is offline-first applications. As far as I can tell, this is not the case with Deepstream.
Just to build on what mwcampbell said (and in full disclosure, I'm on the team at Realm).
Architecturally, Realm and DeepstreamHub are quite different. Realm at its core is a distributed object database that updates changes to objects in realtime. DeepstreamHub, as far as I can tell, relies on a pub/sub and request/response architecture. They are two fundamentally different approaches to the challenge of realtime.
One of the biggest advantages with Realm is that you work with your objects in your native programming language, there's no JSON or ORMs to deal with, just your live objects that update automatically in the local datastore as changes are made on another sync'd client or on the server.
Architecturally, Realm and DeepstreamHub are quite different. Realm at its core is a distributed object database that updates changes to objects in realtime. DeepstreamHub, as far as I can tell, relies on a pub/sub and request/response architecture. They are two fundamentally different approaches to the challenge of realtime.
One of the biggest advantages with Realm is that you work with your objects in your native programming language, there's no JSON or ORMs to deal with, just your live objects that update automatically in the local datastore as changes are made on another sync'd client or on the server.
Thanks for the explanation. I should try out both with this understanding to see which approach is more suitable for different situations.
Actually, deepstream has a very similar datasync approach with its records, pub sub and rpcs are mainly a supporting feature.
Competitor here, please don't become an "open core" company - "Open Core" = "Crippleware".
Here is my thoughts/arguments around it, dealing with the RethinkDB and Parse shut down as well: https://hackernoon.com/the-implications-of-rethinkdb-and-par...
Plus, was invited to speak about it on the Changelog Podcast: https://changelog.com/podcast/236
Please stay truly open :) that way we can still compete. ;) Or else we'll be the only company that is fully Open Source (replication, auto failover, etc.)!
Here is my thoughts/arguments around it, dealing with the RethinkDB and Parse shut down as well: https://hackernoon.com/the-implications-of-rethinkdb-and-par...
Plus, was invited to speak about it on the Changelog Podcast: https://changelog.com/podcast/236
Please stay truly open :) that way we can still compete. ;) Or else we'll be the only company that is fully Open Source (replication, auto failover, etc.)!
Labeling deepstream.io as "crippleware" feels a bit odd to say the least. The relationship is more similar to Kafka/Confluence - a strong, standalone open source project that existed first and continues unrestrained and a solid service company based on it
I think open core is ok. If the community is not happy with the core that is open they can fork and add their own implementation for excluded features.
There was a big uproar over InfluxDB doing this "open core" (aka "Crippleware") approach: https://news.ycombinator.com/item?id=11262318 .
It isn't the right approach, and we ( http://gun.js.org ) aren't going to take it. More than happy to be the only one left standing, devoted to true open values - but I hope others join us.
It isn't the right approach, and we ( http://gun.js.org ) aren't going to take it. More than happy to be the only one left standing, devoted to true open values - but I hope others join us.
Leaving aside how odd it is to see a "competitor" give business model advice, open core is a perfectly fine open-source business model. Pure support based models, with very few exceptions, leave too much % of value on the table to make a strong business. Purely closed models take away too many freedoms from the customer (but can be very successful nonetheless.) Open core is a very reasonable compromise. The problem with InfluxDB is that they pulled a bait and switch - they launched looking like they were going to be fully open, and then went back on that later.
But at the end of the day any compromising on open-source ideals is going to piss someone off - heck even just choosing one OS license over another is going to piss people off. It doesn't matter. In business you need to have a thick skin to stick to your conviction about what is best for both customers and the company - whatever that may be.
But at the end of the day any compromising on open-source ideals is going to piss someone off - heck even just choosing one OS license over another is going to piss people off. It doesn't matter. In business you need to have a thick skin to stick to your conviction about what is best for both customers and the company - whatever that may be.
I tend to agree with the OP and think very similarly along these lines:
https://blogs.gnome.org/bolsh/2010/07/19/rotten-to-the-open-...
A purely services based model is difficult, but so is open core. With open core you compete against yourself. Look no further than Nginx as an example and how lousy it is to do things like say a http healthcheck of HEAD /status. In haproxy it is trivial, it can be done in nginx plus, but not the oss version (at least it wasn't supported in any of the versions I tried a year ago sans the commercial version). When you compete against yourself you get put in a difficult situation "Does this valuable user contribution to my OSS project compete with my commercial fork?" and that is just all around a bad situation to be in for the community and for the company.
https://blogs.gnome.org/bolsh/2010/07/19/rotten-to-the-open-...
A purely services based model is difficult, but so is open core. With open core you compete against yourself. Look no further than Nginx as an example and how lousy it is to do things like say a http healthcheck of HEAD /status. In haproxy it is trivial, it can be done in nginx plus, but not the oss version (at least it wasn't supported in any of the versions I tried a year ago sans the commercial version). When you compete against yourself you get put in a difficult situation "Does this valuable user contribution to my OSS project compete with my commercial fork?" and that is just all around a bad situation to be in for the community and for the company.
The economics are inevitable, any proprietary or crippleware system will be beaten by fully open alternatives. And over time, those fully open systems will have more collaborators and more contributors and more users.
And @yasserf's reply was fantastic, too. Sounds like they are more providing premium hosted services than anything else, which I think is definitely a good way to go.
And @yasserf's reply was fantastic, too. Sounds like they are more providing premium hosted services than anything else, which I think is definitely a good way to go.
co-founder/techie from deepstreamHub here
deepstreamHub literally runs the deepstream.io servers to provide all of the fundamental functionality. All of the client sdks are also Apache 2 based which means that they can be changed in whatever way suits people most, and written in any language.
We also have some really interesting forks by community members that have diverged a bit from how we do things, and has resulted in collaborative forums and in person meetings as well as even talks at meetups about the pros and cons which resulted in the divergence.
The non open aspects are all around the extra services ontop. These all work by hooking into deepstreams plugin architecture, meaning that as stated in following comments, open source plugins are encouraged and are documented in as much detail to help people get started. Offering all the non-core/multi-tenancy/usage monitoring/gui components aspects as open source is in that sense extremely counter intuitive, since we will instead be providing very specific functionality for the purpose of creating another PaaS system. That won't be stopping us from releasing detailed articles/give talks about how things work and run under the hood on AWS and the challenges we met hosting at large scale and across regions, it just means we won't be providing the actual code.
In terms of auto-failover, replication, etc, it would be great if you look at the current open-source aspects on deepstream.io that we have in place for that. deepstream is built with availability and clustering at its core, and its truly because of the involvement of the open-source community that we have managed to make it run in multiple production environments to date.
deepstreamHub literally runs the deepstream.io servers to provide all of the fundamental functionality. All of the client sdks are also Apache 2 based which means that they can be changed in whatever way suits people most, and written in any language.
We also have some really interesting forks by community members that have diverged a bit from how we do things, and has resulted in collaborative forums and in person meetings as well as even talks at meetups about the pros and cons which resulted in the divergence.
The non open aspects are all around the extra services ontop. These all work by hooking into deepstreams plugin architecture, meaning that as stated in following comments, open source plugins are encouraged and are documented in as much detail to help people get started. Offering all the non-core/multi-tenancy/usage monitoring/gui components aspects as open source is in that sense extremely counter intuitive, since we will instead be providing very specific functionality for the purpose of creating another PaaS system. That won't be stopping us from releasing detailed articles/give talks about how things work and run under the hood on AWS and the challenges we met hosting at large scale and across regions, it just means we won't be providing the actual code.
In terms of auto-failover, replication, etc, it would be great if you look at the current open-source aspects on deepstream.io that we have in place for that. deepstream is built with availability and clustering at its core, and its truly because of the involvement of the open-source community that we have managed to make it run in multiple production environments to date.
Great reply, thanks! I love the honesty, integrity, and openness.
Would you be willing to call it "open source with premium hosted services" or something rather than "open core"? I feel like that is more accurate and would be better for marketing, and avoid the whole "open core / crippleware" debacle (see the other comments about InfluxDB). Just my 2 cents!
Cheers! (Thanks for the reply)
Would you be willing to call it "open source with premium hosted services" or something rather than "open core"? I feel like that is more accurate and would be better for marketing, and avoid the whole "open core / crippleware" debacle (see the other comments about InfluxDB). Just my 2 cents!
Cheers! (Thanks for the reply)
> Or else we'll be the only company that is fully Open Source (replication, auto failover, etc.)
major humble-bragging. In bad taste to do that in an announcement for a "competitor"?
major humble-bragging. In bad taste to do that in an announcement for a "competitor"?
That is a sad way to spin it :( but being strongly values-focused is a good thing not a bad thing, and inviting open discussion with competitors improves both systems, not diminishes it. Silencing discussion hinders progress.
What exactly can be replaced with DeepstreamHub if your are using Firebase right now?
deepstreamHub's records can be used instead of Firebase's realtime db/data-sync part, deepstream's events are a replacement for their lately added PubSub capabilities. deepstreamHub does request response as well while Firebase offers integration with numerous Google services.
The core differences are in the way deepstreamHub structures its data (small, related documents with independent lifecycles rather than one big data tree), its speed and open core
The core differences are in the way deepstreamHub structures its data (small, related documents with independent lifecycles rather than one big data tree), its speed and open core
Ok, so this is really a replacement for Firebase realtime database, which is 1 of >25 services that Firebase offer nowadays. Right?
that's true - Firebase is pretty much a Gateway to google's cloud and add-platform. deepstreamHub is an alternative to Firebase core (or what Firebase used to be pre Google acquisition)
Thanks.
(Still looks good!)
(Still looks good!)
I've recently talked with Wolfram about this, but currently I use Firebase mostly with the Firepad library, Deepstream is missing:
* a better presence story * a better story around adding data to a list and subscribing to changes to specific parts of it
* a better presence story * a better story around adding data to a list and subscribing to changes to specific parts of it
+1 interested in replacing Firebase, would like to know more details
It's great to see so many choices for IoT infrastructure.
co-founder/techie @ deepstreamHub
I completely agree! One of the reasons we started is because we believe we have an interesting way of getting devices of all types ( IoT / browsers / phones.. etc ) talking together to generate new ways of displaying and moving data.
At the end of the day the selection of choices such as the eclipse iot project https://projects.eclipse.org/projects/iot, https://relayr.io/, aws IoT gateways, (shameless plug) deepstream.io and many others allow developers to stop having to think about how to get devices to communicate, but instead think about what they want it to do. I'm slightly overly passionate about this because my bachelor project a decade ago was around smart homes and I spent the majority of the time trying to get protocols to collaborate happily, where now that's just considered part of boiler plate setup! Really excited to see the potential outcome due to this general movement.
I completely agree! One of the reasons we started is because we believe we have an interesting way of getting devices of all types ( IoT / browsers / phones.. etc ) talking together to generate new ways of displaying and moving data.
At the end of the day the selection of choices such as the eclipse iot project https://projects.eclipse.org/projects/iot, https://relayr.io/, aws IoT gateways, (shameless plug) deepstream.io and many others allow developers to stop having to think about how to get devices to communicate, but instead think about what they want it to do. I'm slightly overly passionate about this because my bachelor project a decade ago was around smart homes and I spent the majority of the time trying to get protocols to collaborate happily, where now that's just considered part of boiler plate setup! Really excited to see the potential outcome due to this general movement.
How does this compare with https://horizon.io/?
Interesting, I though horizon is dead (because Rethink went down). Last news I heard was them going with LF but haven't seen much updates after that
Give it some time.
Can you provide more information regarding this reply? My opinion is if a service goes dark, I probably shouldn't consider using it.
Recently we had a meeting about the next phase of the project now that Horizon and RethinkDB are owned by the linux foundation: https://www.youtube.com/watch?v=EG0aucGzJE8&t=279s
From the Firebase JavaScript SDK, I can do one thousand individual gets with a 3.4k payload at 1-2ms each. http://jsfiddle.net/katowulf/8wkon62c/
Mileage varies, of course, but I'd love to see a benchmark I can run side by side that can beat those numbers. I can't even get close with a simple GET, which returns in 86ms.