CDK outputs CloudFormation templates that you can pass to any other tool if you wish. CDK is technically just a glorified shim over CFN (although 1000x better but you do get caught in the awkwardness that is CFN many times).
If anyone notices a problem you'll likely need to write a COE, there's no way to get around that. Not updating the status page absolutely doesn't get you out of that task.
COE also doesn't lead to negative marks on anyone at AWS that I know of. It's a learning experience to know why it happened and action items so it doesn't happen again.
Companies usually like to downgrade when hiring. Amazon does it a lot. So even if you qualified as an L7 they may hire you as an L6 (sometimes with L7 pay) and you need to go through a promotion cycle to get back to L7.
Github has been a well known Rails monolith. Absolutely they have services that power all sorts of stuff (they have a few blog posts on it) but it's just impractical to even start a discussion on splitting it up for the reasons you initially argued for.
I've been through a number of large rewrites/reworks that took monoliths much like Github (with many many many services behind it) and split them up into modular pieces and it's an insane amount of work that can take years. You simply need very good reasons (including business reasons) to do that.
Moreover, companies at these sizes just have a LOT of code all over the place. Tooling, infra, supporting services, etc... Not to mention it's just not useful to have external contributors for a business product like Github. Doing code reviews, addressing bugs that were introduced, spending time discussing things with contributors takes an incredible amount of time.
Basically if the reason you want Github open sourced (and reworked into some weird architecture you described) is so that people can contribute to fix things and add features....Github could/will just hire more devs to work on that.
This just isn't possible. No company in their right mind would take their monolith and rewrite a bunch of stuff just to please a couple randoms on the internet.
I think you're drastically underestimating the amount of code Github is powered by and how freaking long any type of refactor/rewrite would take. We're talking about years.
when you have to manage a large number of services with a bunch of different devs touching things, Docker is almost required to get a consistent development state.
There's literally nothing wrong with reaching for Docker right away. It allows you to have the same, repeatable development environment. Maybe you deploy to production without Docker, but any project I start I'll reach for Docker right away.
As someone without a degree, getting an interview at the Big 5 was quite easy. Passing those interviews is another thing entirely but having the right experiences in your resume has been more important in my case.
L3 at other companies is an L4 at Amazon (I believe only contractors or part-time employees can be < 4). L5 also has a much wider surface area and the gap from L5 to L6 is quite big.
The biggest issue with PubSub and Amazon's alternative is the cost. Being capped at a per-message cost would be a no go.
If you can get away with using PubSub or the like it would be far easier than to manage your own Kafka deployment (correctly).
If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data (if done correctly of course). NSQ was great but lacked durability and replication. We can guarantee that two or more Kafka brokers persisted the message before moving on. With NSQ, if one of our instances died it was a big problem.
Managing Kafka in a cloud environment hasn't been easy and required a lot of investment and we have yet to move everything over to it.
We started deploying our Kafka cluster as a set of N EC2 instances but we started running into a bunch of issues (rolling the cluster, rolling an instance without moving partitions around, moving partitions around, etc...)
Now we run Kafka through ECS and wrote some tooling to manage rolling the cluster and replacing brokers. krollout(1) (currently private) basically prevents partitions from becoming unavailable while rolling.
Now that multiple teams are using Kakfa we started exploring how to scale up. Each team may have different requirements and isolation can become an issue. Likely more tooling will need to be built around this.
NSQ has been (IMO) far far easier to operate than Kafka has been. With Kafka you need a Zookeeper cluster in addition to your Kafka brokers. Not to mention developing against NSQ is pretty simple whereas Kafka you need to think about partitions and offsets.
If you're worried about data loss, Kakfa can be what you're looking for (but takes a lot to learn how to operate it correctly).
NSQ has served us pretty well but long term persistence has been a massive concern to us. If any of our NSQ nodes go down it's a big problem.
Kafka has been far more complicated to operate in production and developing against it requires more thought than NSQ (where you can just consume from a topic/channel, ack the message and be done). More to that, if you want more capacity you can just scale up your services and be done. With Kafka we had to plan how many partitions we needed and autoscaling has become a bit trickier.
We now have critical services running against Kafka and started moving our whole pipeline to it as well. It's a slow process but we're getting there.
We've had to build some tooling to operate Kafka and ramp up everyone else on how to use it. To be fair, we've also had to build tooling for NSQ, specifically nsq-lookup to allow us to scale up.
A bunch of people at Segment have looked into it. We manage our infrastructure with Terraform and operationally IMO Kubernetes introduces too much complexity compared with ECS and Terraform.
ECS is pretty dead simple (just run an agent on the host) and while it doesn't offer nearly the same feature set, it's really good operationally.
Some people here are tinkering with it for some non-core services.
One thing we experienced at Segment was the fact that we needed to quickly handle a surge in volume but couldn't overload partners. Essentially we wanted something that scaled up quickly at first but was pretty conservative after that.
We settled on constant increases/decreases using queue depth thresholds but ideally ECS would support feeding multiple metrics and doing some basic math to figure out how much we're currently process, how long it would take to flush the queue, and how many more containers we need to drain in a timely fashion.
CPU + memory autoscaling has worked pretty well for most services at Segment. We did have to build a custom queue-based autoscaler that feeds into ECS for our integration stack but the limitations of the ECS autoscaling rules makes it a bit clunky.
I'm talking about _my_ use case for using AWS. I'm sure other people have similar requirements. We manage hundreds of servers, process over 50 billion events/month and losing data is unacceptable.
In the HN echo chamber, you might think everyone just has a SPA and just needs a Digital Ocean droplet. Everyone has different requirements and AWS fits those for many people.