I use Petri nets. They allow me to diagram communicating systems, making explicit how active components (like functions or services) and passive components (like storage or i/o-devices) interact.
Petri nets have formal semantics easy enough to grasp (firing rule) and provide enough optical cues to make them readable at a glance.
With a little practice, nets can be visually appealing, which helps in presentations and publications. I use Inkscape as a drawing tool.
> The Java/JVM bias is a deficiency of open source Big Data, not bioinformatics!!!
I agree.
> Bioinformatics is built on Unix tools and Unix composition AFAICT
That is true for most bioinformatics tools with a few important exceptions. CummeRbund, e.g., is a must for RNA-Seq and you need to drive it from R. It's hard to tell whether supporting only command line would actually be enough to conveniently describe workflows. For Cuneiform, we assume that it isn't.
> what's wrong with what they're doing?
It's not that moving to Big Data stacks is the ultimate answer to everything. If your workflow runs fast enough on one machine, probably it's best to change nothing. Nevertheless, we're observing a dramatic drop in sequencing costs right now. And the more this trend continues, the bigger will be the demand for scalable infrastructures in Next Generation Sequencing.
The question, why many scientific communities haven't been racing to adopt Big Data technology is a tough one. Whether or not utilization is a major factor I can't really say. For Cuneiform, we have assumed that the major obstacle for using platforms like Hadoop or Spark directly is that these systems require a library to have a Java or Scala interface. If your software is written in, say, Python you need to either
- write a wrapper (an approach taken by Crossbow [1], SeqPig [2], or BioPig [3])
- reimplement (an approach taken by CloudBurst [4] or ADAM [5])
Reimplementing a whole pipeline represents a huge upfront cost. Writing wrappers is more feasible and, accordingly, a huge amount of custom tailored pipelines and ad-hoc worklfow systems have emerged recently. So if it boils down to wrapping libraries and running them on Hadoop, then we need to make this as easy as somehow possible. Cuneiform is a step in this direction.
Automatic parallelization is usually not part of general purpose languages.
Programs written in a general purpose programming language (like Python) are neither automatically parallelizable nor do they run on more than one machine. Of course, most general purpose languages offer facilities for parallelization and/or distribution. Python has the multiprocessing library. In R there is the parallelMap library. But responsibility over how parallelization is used in such a program still resides with the programmer.
This is different in Cuneiform. Everything is automatically parallelized here. Most task applications are relatively large computational pieces of work that run for seconds or minutes (in contrast to, e.g., an addition). In this scenario it makes sense to parallelize wherever possible.
Cuneiform allows you to drive any API.
Another thing that is difficult in most general purpose programming languages is the incorporation of libraries not written in that particular language. E.g., it is non-trivial to use a Python library in Java. Even calling command line applications requires a small wrapper (os.system() in Python). In Cuneiform this is the normal mode of operation. Writing a wrapper around a Bash script is just as hard as defining a function.
I enjoyed reading this post. What especially surprised me is, how easy it is to get going with Glow.
One caveat with the standalone example though is: You need to install Glow first (even though this is a one-liner and appears later in the tutorial)
You addressed the static nature of Glow data flows but also mentioned, that it is possible to do fix-point iteration (by the example of a linear regression). I would love to see an example for such a data flow. I've looked up the examples on [1] and [2] but didn't see one.
As I get it, it is also possible to branch a data flow depending on the result of a condition (if-then-else). Is there an example for this?
Wouldn't it be possible to do even general recursion with this programming model? (I can't come up with a reason why not if the aforementioned things work.)
It is possible to use distributed file systems (like HDFS) as well as distributed databases (like MongoDB) as data sources and intermediate data, as I get it, is sent via channels between processes. If such a channel breaks (e.g., because of a network partition) is there a some kind of failover (rescheduling, making sure a recovering process doesn't break the data flow, ...)? Any plans incorporating Hystrix?
This is great stuff. I'm looking forward to read more about Glow.
Petri nets have formal semantics easy enough to grasp (firing rule) and provide enough optical cues to make them readable at a glance.
With a little practice, nets can be visually appealing, which helps in presentations and publications. I use Inkscape as a drawing tool.