Research HN: Is there a data science DevOps?
I'm looking for people who are working in data science, big data analytics, business intelligence, and related areas, who would be willing to be interviewed (confidentially if you like) about how this type of work is best done in teams. I'm particularly interested in the infrastructure or the DevOps, so to speak. We know a lot about the operations management that supports regular software development teams, but I haven't found any good guidelines for which (if any) tools and processes play a similar role in supporting analytics work. Does anyone here have any thoughts or experience with this problem?
2 comments
I was a data engineer for a couple years at one of the largest websites in Europe. My job was to develop applications to handle our data pipline and deploying the infrastructure on Amazon EC2, Redshift, EMR, etc.
What is it that you need and what are the problems you're facing?
What is it that you need and what are the problems you're facing?
I'm coming at this as an academic, and the problem I'm hearing from companies about the whole idea of a "data scientist" is that it describes a superstar -- someone who knows math, statistics, programming, the business domain, etc, and is self-sufficient. The glowing articles about data scientists focus on the talents and skills they have, and don't tell us much about the operational "work" they do. I can't tell from the literature, for example, what kind of workflow I'd see on a kanban or scrum taskboard for a data science team.
Closer to home, I am also very interested in how a university can train students to do this kind of work... so I want to understand the operational processes and not just a list of technologies the students should learn.
I'm currently teaching a course on software development which touches on workflows and DevOps, and I'd like to do the same for our new analytics major.
Closer to home, I am also very interested in how a university can train students to do this kind of work... so I want to understand the operational processes and not just a list of technologies the students should learn.
I'm currently teaching a course on software development which touches on workflows and DevOps, and I'd like to do the same for our new analytics major.
You mentioned that you know about general software development devops already, why and what do you think is different for data science workflow?
Yup. Loads. reach out at wtpayne-at-gmail-dot-com if you want to talk.