Getting Meteor to 1.0(meteor.com)
meteor.com
Getting Meteor to 1.0
http://www.meteor.com/blog/2013/10/01/geoff-schmidt-at-devshop-8-getting-meteor-to-10
13 comments
Meteor is a lot more than an npm package. For example, if you type these four commands..
$ curl https://install.meteor.com | /bin/sh; meteor create myapp; cd myapp; meteor
you're up and running with a complete stack including MongoDB, node, the dozen-ish core packages that make up the Meteor pubsub and realtime templating stack, and the 'meteor' build tool (which can do things like compile coffeescript and less, generate source maps, minify your production code, and provide a realtime development environment where whenever you save a file your app updates in your browser.)
As for versioning, it's actually got a great way of doing that. Each of your Meteor projects is locked to a specific Meteor release version (similar to an Ubuntu release, it's a release-engineered snapshot of the Meteor core packages), which you can set with the `meteor update` command. And this has automatic `npm shrinkwrap` integration so that if you use npm packages alongside Meteor packages, everyone on your team is always running exactly the same versions of all of the code in your app. The Meteor tools automatically handle all the work of downloading any needed Meteor versions and keeping them installed side-by-side on your laptop, using the correct one for a given project, and notifying you of available updates.
$ curl https://install.meteor.com | /bin/sh; meteor create myapp; cd myapp; meteor
you're up and running with a complete stack including MongoDB, node, the dozen-ish core packages that make up the Meteor pubsub and realtime templating stack, and the 'meteor' build tool (which can do things like compile coffeescript and less, generate source maps, minify your production code, and provide a realtime development environment where whenever you save a file your app updates in your browser.)
As for versioning, it's actually got a great way of doing that. Each of your Meteor projects is locked to a specific Meteor release version (similar to an Ubuntu release, it's a release-engineered snapshot of the Meteor core packages), which you can set with the `meteor update` command. And this has automatic `npm shrinkwrap` integration so that if you use npm packages alongside Meteor packages, everyone on your team is always running exactly the same versions of all of the code in your app. The Meteor tools automatically handle all the work of downloading any needed Meteor versions and keeping them installed side-by-side on your laptop, using the correct one for a given project, and notifying you of available updates.
I want to like meteor.
It sounds like the installation script, aside from handling the MongoDB and node setup, could be treated like a typical node package. Based on the fact that meteor has no package.json, seemingly none of those core packages are installed via npm either.
I may be alone in this, but I guess I would just prefer if my web framework expected me to have the interpreter and database installed, and used the package manager already provided for this environment.
It sounds like the installation script, aside from handling the MongoDB and node setup, could be treated like a typical node package. Based on the fact that meteor has no package.json, seemingly none of those core packages are installed via npm either.
I may be alone in this, but I guess I would just prefer if my web framework expected me to have the interpreter and database installed, and used the package manager already provided for this environment.
Meteor does feel a little heavy (especially when you get around to deploying it) but take a look at demeteorizer, at https://github.com/onmodulus/demeteorizer
It converts Meteor apps into more traditional node.js with a package.json
It converts Meteor apps into more traditional node.js with a package.json
Demeteorizer is a offensively named script that automatically builds a package.json for you. It doesn't make Meteor apps lighter weight or anything like that. It simply makes it easier for you to install the requisite npm dependencies.
While it would be trivially easy to port the Meteor installer to npm, since Meteor doesn't require node or npm to be installed, it would be a step backward to do so.
Meteor is built against specific node and npm versions and the install script installs the necessary node binaries.
Considering what Meteor brings a developer, getting hung up on the install method is missing the big picture.
Meteor is built against specific node and npm versions and the install script installs the necessary node binaries.
Considering what Meteor brings a developer, getting hung up on the install method is missing the big picture.
Thank you.
Saying that Node has a package manager is a huge understatement. Node has one hell of a package manager, but you have to hack[1] around Meteor to use npm modules, even in the latest version.
My impression of Meteor, and the impression a number of node people seem to share, and which I've seen reinforced on HN, is:
Meteor is in an almost adversarial relationship with the node.js open-source community, because while they're good hackers building something cool, they took millions of dollars in funding[2] and want to keep their options open for monetizing that codebase.
This makes every decision they make differently than other open-source real-time frameworks (to build their own package manager, require a contributor agreement on an MIT licensed project, and use a nonstandard install process[3,4]) considerably more worrisome.
And it's frustrating, because there's nothing going on in Meteor that necessitates going outside of the "node.js ecosystem" -- aka NPM [4,5,6].
That impression might be unfair in some way, and obviously the meteor people spend a lot of time responding to just these sort of concerns from the node.js community.
But "Meteor" means "worry" to me.
---
[1] http://meteorhacks.com/complete-npm-integration-for-meteor.h... [2] http://venturebeat.com/2012/07/25/meteor-funding/ [3] https://atmosphere.meteor.com/ [4] https://github.com/meteor/meteor/pull/516#issuecomment-12919... [5] http://derbyjs.com/ [6] https://github.com/substack/node-browserify
Saying that Node has a package manager is a huge understatement. Node has one hell of a package manager, but you have to hack[1] around Meteor to use npm modules, even in the latest version.
My impression of Meteor, and the impression a number of node people seem to share, and which I've seen reinforced on HN, is:
Meteor is in an almost adversarial relationship with the node.js open-source community, because while they're good hackers building something cool, they took millions of dollars in funding[2] and want to keep their options open for monetizing that codebase.
This makes every decision they make differently than other open-source real-time frameworks (to build their own package manager, require a contributor agreement on an MIT licensed project, and use a nonstandard install process[3,4]) considerably more worrisome.
And it's frustrating, because there's nothing going on in Meteor that necessitates going outside of the "node.js ecosystem" -- aka NPM [4,5,6].
That impression might be unfair in some way, and obviously the meteor people spend a lot of time responding to just these sort of concerns from the node.js community.
But "Meteor" means "worry" to me.
---
[1] http://meteorhacks.com/complete-npm-integration-for-meteor.h... [2] http://venturebeat.com/2012/07/25/meteor-funding/ [3] https://atmosphere.meteor.com/ [4] https://github.com/meteor/meteor/pull/516#issuecomment-12919... [5] http://derbyjs.com/ [6] https://github.com/substack/node-browserify
I'm not sure what your problem with contributor agreements are; basically every sizable open source project has one. You know, like... Node: http://nodejs.org/cla.html
Meteor is awesome. It considerably simplifies web-development. I just want to let the core team know that many many people really appreciate your efforts and I sincerely hope that negative comments such as those by Mr. Luc don't discourage your team and do not distract you from the really awesome work you are doing.
Well, all of Meteor is under a MIT license, so if we think we're going to "monetize that codebase" we're pretty stupid :) I'm not a lawyer but I think that under the MIT license there's basically nothing that we can do with the code that you can't do too. Our actual plan is to sell operational tools for larger companies that have mission-critical Meteor apps in production.
The incredible amount of work that we've put into JavaScript build tools over the last two years has all been with the aim of creating a radically easier, faster developer experience, because we know that that's incredibly powerful marketing for the rest of our crazy ideas. In other words, we did it because we thought that the UX of the existing JavaScript tooling was just too janky. Seriously, try Meteor for yourself and see what you think -- maybe you think our work sucks and that we wasted our time, but if so, I wish you would just tell me that (I would genuinely love to know) instead of reading ulterior motives into what was a labor of love and a ton of hard work.
Of course there are many rough edges and it's not done (that's why it's not 1.0 yet) but from people that have actually used it for a while, we actually get the opposite feedback, which is that they want us to go much further down this path, and that's why we continue to slave away at what is by far the least fun part of building a framework.
The incredible amount of work that we've put into JavaScript build tools over the last two years has all been with the aim of creating a radically easier, faster developer experience, because we know that that's incredibly powerful marketing for the rest of our crazy ideas. In other words, we did it because we thought that the UX of the existing JavaScript tooling was just too janky. Seriously, try Meteor for yourself and see what you think -- maybe you think our work sucks and that we wasted our time, but if so, I wish you would just tell me that (I would genuinely love to know) instead of reading ulterior motives into what was a labor of love and a ton of hard work.
Of course there are many rough edges and it's not done (that's why it's not 1.0 yet) but from people that have actually used it for a while, we actually get the opposite feedback, which is that they want us to go much further down this path, and that's why we continue to slave away at what is by far the least fun part of building a framework.
Hmm. I've actually changed my mind, (though not about their rejection of npm) -- after digging around, I see that there's been more communication about how Meteor plans to make money, and hearing that they have feasible plans that don't sound like a conflict of interest with open source is better than any technical argument for assuaging concerns.
But about me 'imputing bad motives', or whatever -- I said "worry", and "almost adversarial", and I stand by that.
When a company pops up and says "Hi, we're from Silicon Valley, and we have a plan to monetize a framework made from your language of choice; we have VC funding and we leap-frogged similar community frameworks -- now, if you want to write apps with the new hotness, please use our new package manager instead of the excellent community one, and don't install our framework with the community standard method either" ...
You worry. You just do. I think that meteor/derby/etc style app dev is the future, so I'm worried by anything that might threaten to lock it down.
I'm glad to see that upon investigation that worry is diminished, though. And obviously I applaud your efforts in advancing that future (I just wish that it were done in an 'npm install meteor' way).
But about me 'imputing bad motives', or whatever -- I said "worry", and "almost adversarial", and I stand by that.
When a company pops up and says "Hi, we're from Silicon Valley, and we have a plan to monetize a framework made from your language of choice; we have VC funding and we leap-frogged similar community frameworks -- now, if you want to write apps with the new hotness, please use our new package manager instead of the excellent community one, and don't install our framework with the community standard method either" ...
You worry. You just do. I think that meteor/derby/etc style app dev is the future, so I'm worried by anything that might threaten to lock it down.
I'm glad to see that upon investigation that worry is diminished, though. And obviously I applaud your efforts in advancing that future (I just wish that it were done in an 'npm install meteor' way).
Out of curiosity, I tried doing an 'npm install meteor' and found that it actually exists right now (but it perhaps isn't official, it hasn't been kept up to date (meteor 0.5.2 compared to the current 0.6.6), and I don't know if it is equivalent to installing meteor the official way).
Don't get too hung up on the installation. I think their point was to make setup/installation/getting excessively simple.
With regard to why not "npm install -g meteor", I think the short answer is that the Meteor installation setups up/installs a lot more than just one node package.
With regard to why not "npm install -g meteor", I think the short answer is that the Meteor installation setups up/installs a lot more than just one node package.
Seems to me like you're throwing the baby out with the bath water. I personally think that's a mistake, but in any case if you don't want to run that command on your own machine, you can always try out Meteor with something like https://www.nitrous.io/
It is not about the running of the command - I read the installation script, and it looks fine.
It is about my desire to work on a node application in what I consider to be a standard way. I want to be able to treat my entire application as a node package, leveraging npm for dependency resolution.
Think of it like a rails application. For local development, you probably use rbenv (or rvm), bundler, and have the version of rails you need listed in your Gemfile. You allow the package manager provided with the language and maybe a layer to keep environments separated (already provided by npm) to take care of dependency resolution and updates.
I want a node framework where I can use the package.json file as one does the Gemfile for ruby applications.
It is about my desire to work on a node application in what I consider to be a standard way. I want to be able to treat my entire application as a node package, leveraging npm for dependency resolution.
Think of it like a rails application. For local development, you probably use rbenv (or rvm), bundler, and have the version of rails you need listed in your Gemfile. You allow the package manager provided with the language and maybe a layer to keep environments separated (already provided by npm) to take care of dependency resolution and updates.
I want a node framework where I can use the package.json file as one does the Gemfile for ruby applications.
Well, Meteor isn't really just a Node framework. Why not evaluate it on its own merits?
it's because your getting a full mongo database installed and configured with it
npm install -g meteorite
Then create a site with it and specify whatever local version of meteor you want.
The deets: https://github.com/oortcloud/meteorite/
Then create a site with it and specify whatever local version of meteor you want.
The deets: https://github.com/oortcloud/meteorite/
Meteor version locking is in core now (as of 0.6.0), and the rest of Meteorite (fetching packages from Atmosphere) will be folded into core as part of 1.0 :)
Congrats to the MDG team on having 1.0 within reach, and thanks for releasing an amazing platform.
Meteor is one of those platforms that you should try. I could give you a menu of features, but the productivity boost from Meteor is one of those things you really should experience. The productivity boost is very real and its quantifiable.
An experienced developer can get a very good feel for Meteor over a single weekend: 1. Buy the book: http://www.discovermeteor.com (the book is $39...I did not purchase the other packages) 2. Spend a weekend doing nothing but coding
Come Monday morning you'll know what Meteor is about.
PS: I am not part of MDG, and I have nothing to do with the book (other than having bought a copy).
Meteor is one of those platforms that you should try. I could give you a menu of features, but the productivity boost from Meteor is one of those things you really should experience. The productivity boost is very real and its quantifiable.
An experienced developer can get a very good feel for Meteor over a single weekend: 1. Buy the book: http://www.discovermeteor.com (the book is $39...I did not purchase the other packages) 2. Spend a weekend doing nothing but coding
Come Monday morning you'll know what Meteor is about.
PS: I am not part of MDG, and I have nothing to do with the book (other than having bought a copy).
Co-author of Discover Meteor here, glad you liked the book!
Curious to learn about meteor but probably will end up needing an up to date book when I actually get around to programming with it. So if I purchase the $40 package, will that include updates up to maybe a year from now?
Yes, we'll update the book for free at least until Meteor reaches 1.0.
I insta-bought the book after hearing the Changelog episode on it. Thank you for writing it.
When you say it is quantifiable - can you actually quantify it for us, or is this just hyperbole?
TL;DR: It's great for realtime apps, but not so much for the traditional web app as compared to something like RoR.
My perspective as someone who has been using it since its release. It depends on what kind of app you are building.
Coming from RoR, Meteor actually is more of a pain given that it still has edge cases that can eat developer time, especially once you dive past the initial levels of tutorials and documentation. RoR also has well established best practices and norms, unlike Meteor and perhaps Node.js as a whole. This means you will be writing a lot more boilerplate code. Meteor has some memory issues as well at times. You have to be aware of memory leaks and will frequent the profiler often, especially with large client-side applications. I've had some projects that I've worked on in Meteor hit 1GB of RAM client-side. Recent releases have been working on this, though. Live reload and hot code pushes are kind of a moot point with something like the Live Reload plugin for Sublime Text or equivalent.
For realtime applications, Meteor is a whole different beast. Realtime applications are much easier to code. I'm comparing this to making a realtime Node.js application. Meteor takes care of the boilerplate for you, pushing changes automatically to the client. It's not difficult to do something like socket.io and node.js but it is nice to have that all handled for you.
It's great for getting an application off the floor - it has a lot of that magic RoR had when it was released. Once applications get more advanced, the structure and maturity of RoR make it easier to work with. These issues are things Meteor is working on, especially in terms of stabilizing the spec/API and working on fixing the edge cases. With 1.0, it seems they are bringing some more stability to it by making it production ready. Its strengths are much more apparent in realtime applications as compared to a traditional web framework.
Edit: As compared to other node.js frameworks like Express, Meteor is much easier to work with - I probably would not go back to raw Express after Meteor. It's been moving fast to 1.0 where many of the issues I've had with it should be resolved with stability and as edge cases are resolved.
My perspective as someone who has been using it since its release. It depends on what kind of app you are building.
Coming from RoR, Meteor actually is more of a pain given that it still has edge cases that can eat developer time, especially once you dive past the initial levels of tutorials and documentation. RoR also has well established best practices and norms, unlike Meteor and perhaps Node.js as a whole. This means you will be writing a lot more boilerplate code. Meteor has some memory issues as well at times. You have to be aware of memory leaks and will frequent the profiler often, especially with large client-side applications. I've had some projects that I've worked on in Meteor hit 1GB of RAM client-side. Recent releases have been working on this, though. Live reload and hot code pushes are kind of a moot point with something like the Live Reload plugin for Sublime Text or equivalent.
For realtime applications, Meteor is a whole different beast. Realtime applications are much easier to code. I'm comparing this to making a realtime Node.js application. Meteor takes care of the boilerplate for you, pushing changes automatically to the client. It's not difficult to do something like socket.io and node.js but it is nice to have that all handled for you.
It's great for getting an application off the floor - it has a lot of that magic RoR had when it was released. Once applications get more advanced, the structure and maturity of RoR make it easier to work with. These issues are things Meteor is working on, especially in terms of stabilizing the spec/API and working on fixing the edge cases. With 1.0, it seems they are bringing some more stability to it by making it production ready. Its strengths are much more apparent in realtime applications as compared to a traditional web framework.
Edit: As compared to other node.js frameworks like Express, Meteor is much easier to work with - I probably would not go back to raw Express after Meteor. It's been moving fast to 1.0 where many of the issues I've had with it should be resolved with stability and as edge cases are resolved.
Yes, we measured it. I am rounding these numbers for simplicity, but essentially, our split of developer time and KLOC follows:
# Labor and spend allocation
- 75% of engineers were on our server-side (Java) (1)
- 25% of engineers were on front-end (JavaScript/HTML/CSS)
# KLOC (2)
- 14% Java
- 54% JavaScript
- Everything else is in other (Scala, CSS, HTML, shell scripts, SQL, etc.)
From speaking with other companies, we found these ratios are within the normal range. We've seen 60% server-side to 80% server-side, but the directional finding appear to be actionable. As mentioned in Note 1 below, we excluded backend work from server-side work.
Also, and this is important, these metrics hold for our single page applications (SPA) projects. I do not have metrics for projects with a more traditional server-side only rendering...but I would guess that these finding do not hold true in such a situation.
If anyone else is willing to share their metrics I would be very interested. I'm always open to seeing the bigger picture in case there are some important facts that I have not factored into my decision making process.
# Notes
(1) This 75% is after segmenting server-side work (i.e. Java application server) from back-end processing (i.e. Kafka, Storm, Hadoop). Obviously, data processing will remain in Java, so we excluded it from our comparison.
(2) KLOC is a poor metric, but it was quick and dirty for us and provided good directional information. Also, all developers (front and back end) are good. So we did not need to adjust for quality of code (i.e. good developers have lower KLOC on average).
# Labor and spend allocation
- 75% of engineers were on our server-side (Java) (1)
- 25% of engineers were on front-end (JavaScript/HTML/CSS)
# KLOC (2)
- 14% Java
- 54% JavaScript
- Everything else is in other (Scala, CSS, HTML, shell scripts, SQL, etc.)
From speaking with other companies, we found these ratios are within the normal range. We've seen 60% server-side to 80% server-side, but the directional finding appear to be actionable. As mentioned in Note 1 below, we excluded backend work from server-side work.
Also, and this is important, these metrics hold for our single page applications (SPA) projects. I do not have metrics for projects with a more traditional server-side only rendering...but I would guess that these finding do not hold true in such a situation.
If anyone else is willing to share their metrics I would be very interested. I'm always open to seeing the bigger picture in case there are some important facts that I have not factored into my decision making process.
# Notes
(1) This 75% is after segmenting server-side work (i.e. Java application server) from back-end processing (i.e. Kafka, Storm, Hadoop). Obviously, data processing will remain in Java, so we excluded it from our comparison.
(2) KLOC is a poor metric, but it was quick and dirty for us and provided good directional information. Also, all developers (front and back end) are good. So we did not need to adjust for quality of code (i.e. good developers have lower KLOC on average).
Earlier this year, I did an experiment: I implemented the design for a reasonably simple rich web app using two different stacks: Clojure on the server Clojurescript for the client, and the other being Meteor.
I have lots of Clojure experience but little Javascript experience. Still, the Meteor implementation took me half as long and had more features. That was a surprising result for me.
I have lots of Clojure experience but little Javascript experience. Still, the Meteor implementation took me half as long and had more features. That was a surprising result for me.
If you're curious about Meteor, I recently wrote a short introduction to it from the point of view of front-end JS developers: https://www.discovermeteor.com/2013/09/17/meteor-for-front-e...
I made a meteor app and it was one of the best experiences I've ever had making a web application. Even though what I made was fairly basic, the ease of creating a real-time application was amazing.
Thanks for posting this. Great overview of what to look forward to. As someone writing an app in production based on 0.6.5, should I plan on any big "rewrites" as the framework approaches 1.0? A better question might be, what will be backwards compatible and what will I have to scrap? It sounds like packages are the one area I should be the most careful with...
There will be breaking changes as Meteor moves from 0.6.5.1 to 0.7 to 0.8 up until it reaches 1.0. This is part of the bargain of working with pre 1.0 software (APIs etc. will be stabilized once 1.0 is reached).
I have found these "breaks" are minor and usually require a few minutes to fix. That said, I also tend to lag behind by a few weeks as other in the community work through the details of the breaking changes.
Meteor Hacks (http://meteorhacks.com) and EventedMind (https://www.eventedmind.com) are two great resources.
I have found these "breaks" are minor and usually require a few minutes to fix. That said, I also tend to lag behind by a few weeks as other in the community work through the details of the breaking changes.
Meteor Hacks (http://meteorhacks.com) and EventedMind (https://www.eventedmind.com) are two great resources.
I'd love to see freebsd support for the install scripts. Last I looked support was in its infancy. Maybe I should revisit.
I've read through the documentation twice (once when it was released and again a week ago) and reviewed the basic demos. Meteor's front end structure seems like it's missing something. It has a data layer and a view / template layer but nothing else. No routing, controllers, etc.
Is there a good demo / tutorial that integrates these missing components, either from meteorite libs or well known projects like Angular or Backbone?
Is there a good demo / tutorial that integrates these missing components, either from meteorite libs or well known projects like Angular or Backbone?
I agree, it's missing a good routing system. iron-router is a 3rd party solution but I didn't have good luck with it. Seems this is functionality that is core to the framework (it's hard to build anything more than a tech demo without routing).
Once you understand iron-router, which takes an hour or two of going through the documentation and trying examples, it really does everything you could ask a router to do. I think its worth a shot of trying it.
I've been using Meteor since its release and its been great at making apps at maximum velocity.
You mentioned that some people are already scaling Meteor across multiple servers. Looking at it, round robin load balancing won't work given requests to a server seem to be stateful. How is that being done as of now? Do you guys have any guidance on how to do that?
You mentioned that some people are already scaling Meteor across multiple servers. Looking at it, round robin load balancing won't work given requests to a server seem to be stateful. How is that being done as of now? Do you guys have any guidance on how to do that?
Yes, if you're using DDP over sockjs instead of just over websockets (which Meteor currently does out of the box to support older browsers), you'll need to use a HTTP proxy with sticky session support.
This will be in the official docs at some point, but for now, Arunoda's blog post about this is a great starting point:
http://meteorhacks.com/load-balancing-your-meteor-app.html
This will be in the official docs at some point, but for now, Arunoda's blog post about this is a great starting point:
http://meteorhacks.com/load-balancing-your-meteor-app.html
I'm very impressed by the technology behind Meteor, I just wish it weren't quite so "opinionated". It relies on standard parts of the node/js web stack, it would be great to have a stand-alone library that supported DDP for front-ends and servers of the developer's choosing.
Any hope of writing/supporting meteor with python backend? :)
Sure! The Meteor client and the Meteor server speak a protocol called DDP, which runs over websockets or HTTP long polling. Meteor clients can connect to any DDP server and vice versa. So you just need to implement a DDP server for Python.
DDP is pretty simple (just a few messages) and the spec for the 'pre1' version of the protocol is in Git: https://github.com/meteor/meteor/blob/devel/packages/livedat...
This is pretty close to version '1' of the protocol that will be locked down in Meteor 1.0: https://trello.com/c/kMkw800Z/53-official-ddp-specification
DDP is pretty simple (just a few messages) and the spec for the 'pre1' version of the protocol is in Git: https://github.com/meteor/meteor/blob/devel/packages/livedat...
This is pretty close to version '1' of the protocol that will be locked down in Meteor 1.0: https://trello.com/c/kMkw800Z/53-official-ddp-specification
I love the DDP as a concept. It's akin to REST-on-steroids, over a websocket with pubsubhubbub sprinkled on top. I really wish more people knew about it, I hope drivers are adapted quickly (some have been). Interoperability & extensibility will make it a killer web feature.
I know I sound like a fundamentalist, but how is anything like REST? It's an RPC protocol! Nothing against RPC, but it doesn't fit any of the REST constraints, except for the client-server; it's not stateless, it can't have middle-man caching and it doesn't really follow an uniform interface. It's nothing like REST (and as such it doesn't get its benefits).
Speaking as the protocol designer, the eventual intention for DDP is to model reactive data set publishing in a sufficiently semantic way that it can be cached by proxies. We didn't get there in pre1 and decided to not let that block Meteor 1.0, but it is the goal for the future when we have some post-1.0 breathing room (and when the Meteor community needs it.)
The idea behind DDP is that HTTP got us three huge benefits: (1) shared tooling (I can write a caching proxy, and anyone with a website can use it); (2) interchangeable parts (your client and your server can be built with totally different languages and frameworks); and (3) easy APIs (I can describe my site's REST-y API to you in a page or two). These days, a lot of sites are moving away from REST/HTTP and toward ad-hoc, custom publishing schemes that run over websockets or a HTTP long-polling transport that is emulating websockets. Since they use these ad-hoc protocols to move around data rather than something like REST, they lose (1), (2), and (3). But all of these ad-hoc protocols are basically isomorphic to each other. DDP is an attempt to nail this kernel of data publishing functionality down semantically to the point that you can get those three benefits back.
As an aside, what's the pressure that's pushing these apps away from REST? The two big ones are:
- They cache data locally and need to keep the cache fresh (say they are apps that run in your browser and never reload the page, or they are native mobile apps), and they don't want to poll for updates. And/or,
- They need to do joins and they care about latency. (When you are loading the news feed, you want to fetch the feed stories, the comments, and the userpic URLs in one round trip, not first request the stories, and then only once you have them request the comments.)
Also, many modern apps have verbs that don't map well to REST (a RPC like transferBalance affects multiple objects and doesn't map well to the idea of updating the representation of an object identified by a URL), and many modern apps need to perform some form of latency compensation (predicting the outcome of a RPC and simulating it locally to update the client's display while waiting for the server's answer, but then reverting to the authoritative outcome chosen by the server.)
DDP attempts to address these needs and run over a stream transport like websockets, while recovering some of the benefits of standardization that made REST so successful.
The idea behind DDP is that HTTP got us three huge benefits: (1) shared tooling (I can write a caching proxy, and anyone with a website can use it); (2) interchangeable parts (your client and your server can be built with totally different languages and frameworks); and (3) easy APIs (I can describe my site's REST-y API to you in a page or two). These days, a lot of sites are moving away from REST/HTTP and toward ad-hoc, custom publishing schemes that run over websockets or a HTTP long-polling transport that is emulating websockets. Since they use these ad-hoc protocols to move around data rather than something like REST, they lose (1), (2), and (3). But all of these ad-hoc protocols are basically isomorphic to each other. DDP is an attempt to nail this kernel of data publishing functionality down semantically to the point that you can get those three benefits back.
As an aside, what's the pressure that's pushing these apps away from REST? The two big ones are:
- They cache data locally and need to keep the cache fresh (say they are apps that run in your browser and never reload the page, or they are native mobile apps), and they don't want to poll for updates. And/or,
- They need to do joins and they care about latency. (When you are loading the news feed, you want to fetch the feed stories, the comments, and the userpic URLs in one round trip, not first request the stories, and then only once you have them request the comments.)
Also, many modern apps have verbs that don't map well to REST (a RPC like transferBalance affects multiple objects and doesn't map well to the idea of updating the representation of an object identified by a URL), and many modern apps need to perform some form of latency compensation (predicting the outcome of a RPC and simulating it locally to update the client's display while waiting for the server's answer, but then reverting to the authoritative outcome chosen by the server.)
DDP attempts to address these needs and run over a stream transport like websockets, while recovering some of the benefits of standardization that made REST so successful.
Thanks; I can perfectly understand the advantages of having a standard protocol, and if DDP becomes an effective standard for applications outside of Meteor, it'll be a great achievement. I just wish people were more aware of what REST really means.
DDP seems like a good technology for webapps, but the issue is that I find webapps a really uninspiring trend in web development; my vision for the web is much more data-oriented than the code silos we're all making. And I believe the lack of understanding of what REST means is preventing people from seeing a bigger picture.
Also, many modern apps have verbs that don't map well to REST (a RPC like transferBalance affects multiple objects and doesn't map well to the idea of updating the representation of an object identified by a URL)
I've heard that a lot, and I don't doubt that it's true in many cases, but usually it seems more representative of a difficulty in the modeling process.
For example, a transferBalance call certainly doesn't map well to updating any object, but creating a new Transfer (and passing the URLs of both Accounts) would be perfectly RESTful, in my opinion.
DDP seems like a good technology for webapps, but the issue is that I find webapps a really uninspiring trend in web development; my vision for the web is much more data-oriented than the code silos we're all making. And I believe the lack of understanding of what REST means is preventing people from seeing a bigger picture.
Also, many modern apps have verbs that don't map well to REST (a RPC like transferBalance affects multiple objects and doesn't map well to the idea of updating the representation of an object identified by a URL)
I've heard that a lot, and I don't doubt that it's true in many cases, but usually it seems more representative of a difficulty in the modeling process.
For example, a transferBalance call certainly doesn't map well to updating any object, but creating a new Transfer (and passing the URLs of both Accounts) would be perfectly RESTful, in my opinion.
> but the issue is that I find webapps a really uninspiring trend in web development; my vision for the web is much more data-oriented than the code silos we're all making.
Can you tell me more about your vision?
Can you tell me more about your vision?
I'm sorry, I started a reply, but after two paragraphs of rambling I had to delete it all.
If you want, I can try to form a more cogent reply later, but in a short summary, it would be keeping the current trend of sending structured data, but restoring the principles that made the web great - decoupling code from data, connecting it across services and communities (turning Hypertext into Hyperdata) and aggressively adopting standard formats.
I see a web where I don't go to Hacker News; messages come to me from everywhere, to where I want them to. Where I don't open an issue on Github, but on my personal tracker, which federates with theirs. Where businesses work for consumers, responding to their demands, instead of pushing their crap on us.
This view is mostly based on a mix of the concepts such as the Semantic Web / Linked Data, VRM, the Intention Economy and other ideas.
If you want, I can try to form a more cogent reply later, but in a short summary, it would be keeping the current trend of sending structured data, but restoring the principles that made the web great - decoupling code from data, connecting it across services and communities (turning Hypertext into Hyperdata) and aggressively adopting standard formats.
I see a web where I don't go to Hacker News; messages come to me from everywhere, to where I want them to. Where I don't open an issue on Github, but on my personal tracker, which federates with theirs. Where businesses work for consumers, responding to their demands, instead of pushing their crap on us.
This view is mostly based on a mix of the concepts such as the Semantic Web / Linked Data, VRM, the Intention Economy and other ideas.
I'm totally there with you, and my hope with DDP 1 was to make some progress on that first step: decoupling code from data, so that a standard DDP client can connect to any DDP server and pull data out of it.
In the future, I'm super excited about investigating ideas like DDP discovery (asking a DDP server what data and methods it has) and description (having a DDP server publish metadata in some standard format that explains the semantics of the other data that it's publishing in terms of microformats). I think the main place we differ is that I think that these formats should use something more like a relational model rather than a hypermedia model, or a formal knowledge representation model like RDF. Isomorphisms exist between these models, and used properly they are all equally powerful, so really this just a design decision about what will be easiest to implement and adopt. And of course I believe that it has to be server push (websockets) rather than poll (HTTP), and that it has to support server-side joins (doing 1 query rather than N+1 queries, and taking 1 roundtrip rather than 2, to retrieve your newsfeed stories plus the comments on them.) I can't tell from your comment whether you agree with that or not, but like I said, these are two of the main things that motivate DDP.
All of that said, I think that by far the hardest part of the puzzle is the product design and federation issues. In your federated Github and HN, the technical work is the first 10%, the next 90% is finding a user experience for it that makes sense to and is ergonomic for a large number of real users, and the final 90% (really should be more like 9000%) is convincing the industry (read, creating incentives to force the industry) to cooperate with the federation and to resolve all of the political squabbles about advertising, control of the data, licensing, and so on. I mean, Google pulled out of XMPP federation -- all practical evidence suggests that keeping these federation together is wicked hard.
So that's why we're just trying to build the first 10% of that first 10%, which has a very clear use case for building rich browser apps and mobile apps -- and hoping that we will be a stepstool that others such as yourself can stand on to build the full solution that everyone dreams of.
Anyway, if you are sympathetic to this line of argument, I hope you'll check out DDP and see what interesting discovery and self-description features could be built on top of it, or suggest how its semantics could be strengthened!
In the future, I'm super excited about investigating ideas like DDP discovery (asking a DDP server what data and methods it has) and description (having a DDP server publish metadata in some standard format that explains the semantics of the other data that it's publishing in terms of microformats). I think the main place we differ is that I think that these formats should use something more like a relational model rather than a hypermedia model, or a formal knowledge representation model like RDF. Isomorphisms exist between these models, and used properly they are all equally powerful, so really this just a design decision about what will be easiest to implement and adopt. And of course I believe that it has to be server push (websockets) rather than poll (HTTP), and that it has to support server-side joins (doing 1 query rather than N+1 queries, and taking 1 roundtrip rather than 2, to retrieve your newsfeed stories plus the comments on them.) I can't tell from your comment whether you agree with that or not, but like I said, these are two of the main things that motivate DDP.
All of that said, I think that by far the hardest part of the puzzle is the product design and federation issues. In your federated Github and HN, the technical work is the first 10%, the next 90% is finding a user experience for it that makes sense to and is ergonomic for a large number of real users, and the final 90% (really should be more like 9000%) is convincing the industry (read, creating incentives to force the industry) to cooperate with the federation and to resolve all of the political squabbles about advertising, control of the data, licensing, and so on. I mean, Google pulled out of XMPP federation -- all practical evidence suggests that keeping these federation together is wicked hard.
So that's why we're just trying to build the first 10% of that first 10%, which has a very clear use case for building rich browser apps and mobile apps -- and hoping that we will be a stepstool that others such as yourself can stand on to build the full solution that everyone dreams of.
Anyway, if you are sympathetic to this line of argument, I hope you'll check out DDP and see what interesting discovery and self-description features could be built on top of it, or suggest how its semantics could be strengthened!
Here's an interesting project which uses DDP to talk with a Raspberry Pi: http://pijs.io/
Hope you finish soon.
I am excited than for an review of meteor and derby, i think that this are the two most exciting projects at the moment.
Metheor got about something like 16M $ in funding, if I remember good, and they can't afford a proper microphone for the speaker? :)
comeon, downvoters, don't you have a sense of humor? ... man...
Doing AV right is hard! You can devote your whole career to learning how to do AV well, and I have a ton of respect for the skills of the people that choose this path. It's a lot more than plugging in a microphone.
We do try to improve AV with every Devshop, and we have recently started to bring in AV professionals to help. But honestly, pre-1.0, I think it's way more appropriate for us to be spending our time and money on advancing the framework than on super slick video presentations. In many ways it's more like the Meteor community's time and money.
We do try to improve AV with every Devshop, and we have recently started to bring in AV professionals to help. But honestly, pre-1.0, I think it's way more appropriate for us to be spending our time and money on advancing the framework than on super slick video presentations. In many ways it's more like the Meteor community's time and money.
I really wanted to like it. I was predisposed to do so.
But from the very beginning, I was confronted with something I consider a deal breaker. While I am wary and generally disinclined toward the increasingly popular pattern of curl an installation script and pipe into sh, in this case it is not only skeptical, it is downright ludicrous. Why would I, nay anyone, want to install an npm package this way? Should not the installation instructions simply be `npm install -g meteor`? And what if I need to work on multiple meteor projects with different versions? Surely there is support for adding it to your package.json file, but why is this not the primary means of installation and well documented?
Maybe I am being overly sensitive to these issues, but it really baffled me that the very foundation upon which a meteor project is predicated would be contradictory to the typical node workflow. I will probably try meteor to spike out a project at some point in time, but I do not foresee myself using it extensively in the near future.