Hey - I'm Eugene, cofounder of the parent organization of this project (Work On Climate).
We left Google to work on climate.
One of the first things we've discovered after talking to lots of people in climate tech is that almost all startups want good engineers, often CTO-level.
One of the first things we're doing is building a community to help people make the jump into the space; this is part of it.
Hi, I'm also an engineer on the Dataflow team.
This is a very exciting project! However it is still constrained by Spark's limitations outlined in this post in terms of what Dataflow pipelines it can run. For example, currently it doesn't support session windows, it windows based on processing time instead of event time, and it doesn't seem to support triggers (the word "trigger" doesn't appear in the code base). AFAIK Spark has plans to add support for event time, and then it will be possible to make this runner run more pipelines consistently with the semantics of the Dataflow model. It is also likely that the capabilities of this runner will expand greatly as https://wiki.apache.org/incubator/BeamProposal progresses.
Hi! I'm an engineer on the Dataflow team. I'd love to understand the issue you mentioned about size of intermediate files, and help make Dataflow work well for you. Could you send a note to [email protected] and elaborate?
On maturity: Dataflow is built by the people who over the past 12 years created MapReduce, FlumeJava and FlumeC++, Pregel and Millwheel, based on the sum of experience obtained from all of these. It shares most of the back-end stack (work scheduling, pipeline optimization, fault tolerance etc.) with FlumeJava and FlumeC++ for batch jobs and Millwheel for streaming jobs, all of which are extensively used inside Google for data processing, and it shares a lot of the Java SDK with FlumeJava.
Happy to answer additional questions. If I forget to check on this thread, feel free to ask on [email protected] or on StackOverflow with tag google-cloud-dataflow - we constantly monitor these and usually answer everybody.
We left Google to work on climate.
One of the first things we've discovered after talking to lots of people in climate tech is that almost all startups want good engineers, often CTO-level.
One of the first things we're doing is building a community to help people make the jump into the space; this is part of it.
Ask us anything!