> When he died at the age of sixty-two, in 1896, his doctor commented on the cause of his death: “I consider the case is this: the disease is simply being William Morris, and having done more work than most ten men”.
I had one of these! Very cool piece of technology, but a slew of papercuts (buggy app, connectivity issues, poorly designed charger) made me give up on it after a couple weeks.
HubSpot | Senior+ Engineer(s), Infrastructure Orchestration | Boston, MA or Remote (US-based) | Full-time
Hi! I'm an infrastructure engineering leader at HubSpot.
HubSpot is a marketing, sales, and services platform for growing businesses, and we're looking for senior engineer(s) for our lowest-level infrastructure provisioning / orchestration / tooling teams. One of our big projects revolves around building an end-to-end declarative representation of our infrastructure based entirely around Kubernetes and composable custom resource definitions (CRDs).
For our standard applications, a deploy is done with the push of a button and app developers do not need to think about Kubernetes manifests or even the existence of Kubernetes at all. So that portion of the problem is already completed at scale. What remains is handling all of the lower-level pieces of infrastructure and stateful services which are not easy to generalize for. We do this by extending Kubernetes with additional building blocks, rather than attempting to make an all-encompassing framework for doing so since such a framework would approach the complexity of Kubernetes itself. It can almost be thought of as building a lower-level PaaS for infrastructure teams.
With this model, each infra team builds CustomResourceDefinitions and operators to represent their services. For example:
- The Kafka team allows for creating a Kafka cluster via a KafkaCluster CRD
- The Kafka operator creates the ZooKeeper cluster (an underlying dependency) for that with a ZooKeeperCluster CRD, which is handled by the ZooKeeper team's operators
- Those operators get all of the secrets they need via our SecretTemplate CRD, handled by Infra Orch's kube secret operator (which injects Vault secrets into appropriate Kubernetes secrets)
The vision is to use CRDs for every un-automated corner of infrastructure, all the way down to the EC2 VPCs themselves. We're striving to reach a point where creating a new data center is done by applying the CRDs that define the top-level clusters, and everything else automatically is configured end-to-end with no manual human effort. For any case where there is a common problem that all teams running on Kubernetes must solve, our job is to find an abstraction for that problem.
HubSpot | Senior+ SWE - Infrastructure | Full-time REMOTE (US) or in-person (Cambridge, MA)
We're hiring for many roles across the board (FE / BE / PM / UX), but I'm specifically hiring for our Infrastructure Orchestration team, which is responsible for the provisioning, management, and scaling of our cloud infrastructure.
If some or most of these describe you, we should chat:
- You’re an expert in Python or Go.
- You have extensive experience with the core offerings of AWS or GCP (e.g. VPCs, EC2, EBS, and S3 for AWS).
- You have experience with the Linux kernel and networking stack.
- You’ve written custom eBPF programs to monitor or secure your infrastructure.
- You have experience with writing Kubernetes operators and/or interacting with the Kubernetes APIs.
- You’ve contributed to a CNCF project.
Thanks for the kind words! Singularity is still a core piece of infrastructure for running stateless apps at HubSpot (our status page currently reads ~13k Requests, ~22k Active Tasks, ~700 Agents). We're also heavily investing in Kubernetes, but Singularity is solid enough that our focus has been more towards reliably running and scaling stateful things like MySQL, ZK, Kafka, HBase, and ES.
Awesome post -- it'd be cool to see Kyle test more Mesos frameworks. Singularity (http://github.com/HubSpot/Singularity) is another good option for running scheduled jobs in Mesos (along with web services, background workers, and one-off tasks). It fails fast like Chronos, but we made it clear to use monit, supervisor, or systemd in the docs. ;) We run our entire product on it (~1,600 deployable items) with no issues. Our PROD cluster has launched 8,000,000 tasks so far in 2015!