Zed Shaw's Advanced Network Architectures With ZeroMQ at Pycon 2011 [video](vodpod.com)
vodpod.com
Zed Shaw's Advanced Network Architectures With ZeroMQ at Pycon 2011 [video]
http://vodpod.com/watch/5757197-pycon-2011-advanced-network-architectures-with-zeromq
5 comments
It is challenging to explain anything subtle, when your viewpoint depends so much on what you already know, your preconceptions, and assumptions.
Sorry for the comic book style. It's how my mind works. Feel free to send a patch. :)
The benefits of 0MQ to me, as a programmer, are:
* Really easy to write utterly solid multithreading code, in a language (C) that has zero support for this; * Really easy to take this same code and make it multiprocess, or multibox, with few changes. * Built-in handling of asynchronous I/O, which I need in any real application. * Speed, meaning I can be lazy and my code still runs very quicky. This is less obvious than it seems. 0MQ's critical path is extremely evil.
It's so hard to explain this with a diagram or a slideshow. The only way to understand is to use it, to write code with 0MQ. Certainly for me, when I started programming 0MQ apps, my understanding of what this library was about, and what it could do for me, totally changed.
Sorry for the comic book style. It's how my mind works. Feel free to send a patch. :)
The benefits of 0MQ to me, as a programmer, are:
* Really easy to write utterly solid multithreading code, in a language (C) that has zero support for this; * Really easy to take this same code and make it multiprocess, or multibox, with few changes. * Built-in handling of asynchronous I/O, which I need in any real application. * Speed, meaning I can be lazy and my code still runs very quicky. This is less obvious than it seems. 0MQ's critical path is extremely evil.
It's so hard to explain this with a diagram or a slideshow. The only way to understand is to use it, to write code with 0MQ. Certainly for me, when I started programming 0MQ apps, my understanding of what this library was about, and what it could do for me, totally changed.
I started using ZeroMQ at work a few weeks ago after reading the guide.
The part that I like about ZeroMQ is that it's a simplifying abstraction, e.g. it's a fairly simple wrapper about sockets that makes life easier for me. I guess it's mostly like a library of patterns for socket usage, so it makes a request-response loop and pub-sub scenarios really easy to implement. The other thing that's great is that it's pretty much programming language agnostic, meaning that C++ code and Python code look fairly similar and can trivially be used together.
As for the transparent queueing, there are simple options to control the size of the queue; it's even possible to spool the queue to disk as soon as the memory limit is reached (see setsockopt docs and look for HWM, for high water mark).
The part that I like about ZeroMQ is that it's a simplifying abstraction, e.g. it's a fairly simple wrapper about sockets that makes life easier for me. I guess it's mostly like a library of patterns for socket usage, so it makes a request-response loop and pub-sub scenarios really easy to implement. The other thing that's great is that it's pretty much programming language agnostic, meaning that C++ code and Python code look fairly similar and can trivially be used together.
As for the transparent queueing, there are simple options to control the size of the queue; it's even possible to spool the queue to disk as soon as the memory limit is reached (see setsockopt docs and look for HWM, for high water mark).
All of your reasons make sense to me, it's just that all the advocacy docs set ZMQ up to be this game-changing system. I can totally buy "a fairly simple wrapper about sockets that makes life easier for me." "world-saving superheros of the networking world" not so much.
I don't know, the simple building blocks it provides can be combined in a bunch of ways that are really pretty cool. I think apenwarr would might call it a "simplifying assumption". It's hardly ever the complex things that are game-changing, right?
"Topology-agnostic programming" has been the norm on Wall Street for over a decade now; Tibco calls it "subject based" networking, I think?
I also think it's kind of a misfeature, and that in the real world it tends to devolve to "hub and spoke client/server with lots more failure modes".
I also think it's kind of a misfeature, and that in the real world it tends to devolve to "hub and spoke client/server with lots more failure modes".
I don't think there's much to gain (at least right now) by using zmq in a simple single-client -> server setup.
I read the testimonials, which got me excited, so I then tried to make use of zmq in a system I was working on. Using plain old sockets, I could whip up a connection in a few lines (OK, python helps here), and deal with all failure modes associated with them. Mapping this robust setup to zmq, required multiple layers, async requests, and a polling mechanism. I don't recall now, but I may have had to throw a router in there as well. I'm not dissing zmq, as it's a great technology, and solves some complex problems well, but it's not for everyone, and people without complex problems will more likely be confused by it than helped.
The documentation is OK, but I think tries to abstract too much of what's going on, and just leaves you with a bunch of less-than-descriptive names for things that you have to take for granted. All the mamas, papas, and dealers analogies didn't help me at all.
I read the testimonials, which got me excited, so I then tried to make use of zmq in a system I was working on. Using plain old sockets, I could whip up a connection in a few lines (OK, python helps here), and deal with all failure modes associated with them. Mapping this robust setup to zmq, required multiple layers, async requests, and a polling mechanism. I don't recall now, but I may have had to throw a router in there as well. I'm not dissing zmq, as it's a great technology, and solves some complex problems well, but it's not for everyone, and people without complex problems will more likely be confused by it than helped.
The documentation is OK, but I think tries to abstract too much of what's going on, and just leaves you with a bunch of less-than-descriptive names for things that you have to take for granted. All the mamas, papas, and dealers analogies didn't help me at all.
To me the headline feature of ZMQ is that I don't need to run an independent server to pass messages around. It's basically a socket with queuing. That reduces my administrative complexity, which is a clear win.
Previously (with comments): http://news.ycombinator.com/item?id=2319875
Is there some text available somewhere (slides, perhaps)?
I guess it would've been better if the title were "Advanced Network Communication Architectures (or Patterns)" with ZeroMQ. Network Architectures sounds more hardware-ish and (at least to me) sounds like it's talking about the actual network topology... :-)
Today I gave it another shot, and after half an hour of trying to glean technical detail and skipping any parts that read like a like a comic book ("zap-pow-kaboom satori paradigm-shift moment"), I think I've learned the following:
The main features seem to be topology-agnostic programming (since you don't have to know what a socket is connected to to send/receive over it) and transparent queuing. In my opinion transparent queues can be problematic because they transparently use up memory that can be hard to account for. Topology-agnostic programming certainly seems interesting, but in my experience of distributed systems programming I never seem to need or want complex messaging topologies. I guess I don't really get what all the hype is about. Maybe I should watch the video.