I've personally noticed and saved multiple $xx,xxx monthly cost billing spikes just by take a daily glance at our cost explorer. I'm in the AWS accounts every day doing investigative work anyway that an extra 30-60 seconds is trivial.
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.
I was thinking the exact same thing. There are multiple places to implement hooks (git hooks, Claude hooks, etc.).
One thing I've been wondering about is how to reliably protect specific portions of the system from unexpected/unnecessary change (for example, a failing test that Claude decides to comment out or rewrite to get it to pass). My only thought for this was to automatically revert test changes during specific portions of the implementation, but that feels overly rigid and potentially prevents things like refactoring code.
This is a good question, and it felt like nepotism. I do want to point out that this is all somewhat hazy memories from years ago when all of this happened, so take everything with a grain of salt (as usual). Also, a lot of this is going to sound like nepotism, which is most likely was, but this is hearsay from other people.
My understanding of how the "fixer" came into there position is a somewhat circuitous route. From my understanding (I didn't hear any of this directly from the "fixer" themselves, but other people who spent far more time with the "fixer" than myself), the "fixer" had spent about a decade out of the workforce prior to joining Tesla. My understanding is that they were raising kids while also dealing with aging parents. We'll just call this time the "fixer"'s work hiatus.
Prior to the hiatus, the "fixer" had moved into a small-team managerial role at a large, name-brand tech company during the late 90s/early 2000s. At the end of the hiatus, they leveraged some connections and somehow attained a director position at Tesla managing a team of about 30-40 people straight out of the hiatus.
From my understanding, the first team the "fixer" managed at Tesla didn't like working for them and after about 18 months, the team basically forced the "fixer" out. I'm not exactly sure what the team was doing to push the person out, but from what I heard, work basically ground to a halt for the entire team where they refused to work for the "fixer".
This was around the same time that the two projects went sideways that I mentioned, so the director I reported to was on the outs and the director's manager (a VP) was looking for someone who could step into the role. The VP somehow connected with the "fixer" and they worked out a deal where the "fixer" would lead the team on a 3-month probation period while the VP continued to look for someone to come into the position, while also giving the "fixer" a chance to earn the role.
(Side note: One other bit of context I want to provide is that the team I was on was about 50-60 or so people at this time right before the "fixer" came on. The "fixer" also did not have any sort of technical background and this team consisted of probably ~90% software professionals in some capacity. A lot of the conversations were very technical in nature, and the "fixer" did A LOT of delegating and "just tell me what decision you'd make and we'll do that" leadership.)
During this probation period, I thought the "fixer" actually did a good job getting a lay of the land, the social dynamics at play, and helped work out some inefficiencies. However, a lot of this improvement was done by bringing in consultants to do the deep dive, discover problems, and provide guidance to the "fixer" on how to address the problems.
Once the probation period was over, the consultants left and the "fixer" was in charge. Pretty quickly, the firings began and over the course of the next 5-6 months, more than 70% of the team under the "fixer" was replaced. At the same time, the team I was working for merged with another team, and the team size under the "fixer" shot up to about 100-120 people post-merge (I forget the exact number). The "fixer" also hired quite a few more people thinking more people get the same projects done faster.
To say the least, it was a pretty chaotic time because the entire team was under a lot of pressure with in-flight projects, not knowing if they were going to randomly be fired or not, new people to mentor/gel with, and lots of random projects being thrown at us.
About 6 months after I left, the "fixer" was fired and someone else who had extensive experience was brought in to right the ship. Per my understanding with people who were still working there about a year after the "fixer" left, the new person was very successful and had done a good job leading the team. Also, the person who I found to be my replacement stayed nearly 7 years at Tesla, so I guess I did a good job with that one.
IMO, this is a good question and deserves a solid answer, so I’ll do my best.
Setting aside the “fixer” for the time being, I really enjoyed the work I did at Tesla. Tesla was the first company that gave me very high levels of autonomy to just own projects and deliver. It also pushed me to take on projects that I had previously wanted to do that I hadn’t been given a chance to work on before.
(Side note: At that point in time in my career, my thinking was that I needed to earn opportunities to work on projects at work to build skills that would enhance my career. I didn’t see the value in working on projects outside of work to build skills because I didn’t think those side-project skills would be valued by other companies the same as “day job” experience. I’ve since learned this isn’t true when it’s done right.)
I spent a lot of time at Tesla delivering value for a bunch of people who desperately needed it at the time, and the thanks I received from them was genuine. It felt very good to help others at Tesla out in a meaningful way, so I kept chugging along to the best of my abilities. Life was throwing lemons at me in my personal dealings, and Tesla was helping me make lemonade from a career standpoint. Besides, all the long work hours were a good distraction from the home life stuff.
In a lot of ways, it was a very fulfilling environment to work in, but it wasn’t for the faint of heart. People often quit within a month or two because the environment was too fast paced with too many projects under tight deadlines and projects quickly followed one after another. An environment like Tesla just doesn’t let up, so one has to figure out how to manage the stress without much support from others. Oftentimes, if you do need to let up at Tesla (or introduce friction in any sort of seemingly non-constructive way), that’s the cue you aren’t working out for the company anymore and it’s time to find someone to replace you.
Coming back around to the original question of why I stuck it out until the end. Just before the “fixer” was brought in, I was “soft promoted” by a director (no title change, but was given direct reports and a pay bump, the title change was suppose to come a couple of months later as the soft-promotion happened just before an annual review cycle). The director who soft-promoted me was someone who I got along with well and it seemed like things were going in the right direction in my career at that point. The director was in charge of a couple of projects that went sideways in a very visible way, and Elon basically fired the director after the second project went south, which is why the “fixer” was brought in.
When the “fixer” first took over things, it seemed like I was going to continue on the path that the director had originally laid out for me. The “fixer” said I was going to get more headcount and work on bigger projects, but this never materialized.
I really didn’t like working for the “fixer” after a while. IMO, it was clear they didn’t know what they were doing, they weren’t willing to listen to feedback, and I spent a lot of time trying to provide guidance to the “fixer”, but it wasn’t seen as helpful and I felt like I was spinning gears. My mental health did start to suffer as I got more burned out towards the end of my tenure there.
Eventually, I was tasked with hiring someone to be my manager and I saw the writing on the wall (sort of). I started to look for a new job just in case. At one point, I thought bringing in someone between myself and the “fixer” would be a good thing. I didn’t realize I was actually finding my replacement. Two days after my replacement was hired, I was let go (this was the 1:1 meeting where I was going to turn in my notice, but HR served me papers instead).
To your original point, if I was in a similar situation now, I would be planning my exit immediately instead of trying to make the best of a bad situation, but I had to learn that lesson the hard way.
I’ve experienced it at other places as well, just not the frequency or indirectness as Tesla.
During the first 24 hours of the Model 3 pre-order launch, Elon tweeted that we would support another 3-4 currencies than we had built and tested for. The team literally found out because of his tweet and had not planned for those currencies. That wasn’t the first time that sort of deal happened where we found out about a feature because of one of his tweets.
Definitely one approach to the circumstances. I tried some variation of this and it blew up in my face (as I expected ).
Towards the end of my time there, a “fixer” was brought in to shore up the team that I was working on. The “fixer” also became my manager when they were brought on.
The “fixer” proceeded to fire 70+% of the team over the course of 6-8 months and install a bunch of yes people, in addition to wasting about $2,000,000 on a subscription to rebuild our core product with a framework product no one on the team knew. I was told to deploy said framework product on top of Kubernetes (which not a single person on my team had any experience with) while delivering on other in-flight projects. I ignored the whole thing.
I ended up deciding I was done with Tesla and went into a regularly scheduled 1:1 with my manager (the “fixer”) with a written two-weeks notice in hand, only to be fired (with 6-weeks severance, thankfully) before I was able to say anything about giving notice.
Agreed. Tesla taught me the hard way about work/life boundaries. I spent a lot of time working a full 8-9 hours during the day, then doing deployments during the nights, weekends, and on “vacations”. A 60-hour week was a “light” week at Tesla.
Didn’t have kids or friends at the time and was going through a breakup, so I was okay with throwing myself at the job for a while. Once my situation got better, all those hours didn’t make as much sense, so I started looking for another job. The very next job was an immediate pay bump of 20% for half the amount of work.
These days, I clearly restate what is being asked (per my understanding), what I’m currently working on, if the thing is being asked is more important or not, and if the requestor is willing to delay the original timeline by the amount of time the interrupt will take plus context switching time.
From my time at Tesla, this is 100% the case. When Elon asked for something, it was “drop what you are doing and deliver it”, then you got pressed to still deliver the thing you were already working on against the original timeline before the interrupt.
Are you doing this for non-library projects as well? I'm trying to wrap my head around the idea of packaging a microservice using this mechanism (I haven't heard of this before now).
My wife and I are currently borrowing a Snoo from a friend and have found it very helpful. Our daughter is 8+ weeks and regularly sleeping 7+ hours in a single stretch each night, probably heavily due to the Snoo. When we first got the Snoo, we saw an immediate change in our daughter's sleep where she went from having a unreliable 2-3 hour stretch between feedings to a solid 3+ hours between feedings with less fits during her sleep and also falling to sleep faster when we put her in the Snoo.
We understand the desire for the company to make money, but we feel there's a happy middle-ground where the Snoo could have the premium app subscription waived for the first child (6-12 months premium subscription free), but require a fee for the app for future children. That being said, the Snoo has been advertised for years around the core features that are now being locked behind a subscription.
We are very fortunate to be borrowing the Snoo from our friend, but it definitely makes us second guess buying a Snoo if the price goes up due to the "mandatory" subscription fee. Would we still use the Snoo even if we had to pay the subscription fee? Most likely, because one is ultimately buying sleep back by using a Snoo. At the same time, the Snoo does not work for every child and we've heard of multiple parents in our friend circles who bought the Snoo but didn't end up using it because it didn't work for their children. It's kind of an expensive, risky bet to make for the potential chance that it may not work out.
I personally think the Snoo is overpriced and think the true price is probably around $1,000, but it sounds like there are inefficiencies to be ironed out on Happiest Baby's side. The "mattresses" the Snoo comes with are simple foam and it's made up of a ton of plastic. Not being a physical product engineer myself, I think it could probably be re-engineered to bring the cost down while retaining the same feature set.
I've seen this a handful of times with libraries and other software. Typically, it's a year of updates, so 18 months is on the more generous side of things with this model.
He often doesn't do things "by the book" and will not wait for what he thinks is unnecessary red-tape. He will skirt around regulations and taunt the process the entire way. We've seen it many times before, and it's always purely in the benefit of whatever company he's helping at the time.
He has no problem throwing people/money/etc. at the problem to get the result he wants and if someone so much as says the opposite, they are removed from the equation (fired, publicly ridiculed, etc.).
I personally don't see what benefit he can bring to Twitter other than shaking up the culture. Who knows, he may even come in, take over, and claim to be a founder of Twitter, just like he did with Tesla.
I completely agree. That section was a gross misunderstanding of what Continuous Delivery actually is. It should have actually been called "Packaging". The "Continuous Delivery" section was more about versioned packaging (and vendoring dependencies into packages), while the automation section was about configuration management of servers (which included deploying versioned packages).
If someone wants to understand what Continuous Delivery actually is (which is pretty well understood), there's a great book on the topic called... oddly enough... "Continuous Delivery" by Jez Humble. It covers CD pipelines like you mentioned, while also covering a bunch of the other listed topics at a high level in a much better way. Granted it's an actual book, but it's worth it's place on an DevOps/Release/Software Engineer's desk when implemented appropriately.
Actually, it's not good advice. It's terrible advice. You'll never get a straight answer out of them. I've tried it myself multiple times and I never got an honest answer. The ACTUAL best way to get the low down on the worst parts of working at a company is to network with former employees and ask them, since they have much less of a reason to fabricate an answer.
This is also a problem with Simple Mobile (a MVNO that I'm trying out with a carrier and development unlocked AT&T Branded HTC Titan). Repeatedly, it will default to T-Mobile or AT&T APN and Cell settings, causing me to lose cell service and internet access. Oddly enough, when they've switched settings and cell/internet is supposedly down, I will get MMS messages. There is no way to set the settings correctly and lock them down from an end user perspective. It's pretty much a wash when it comes to service since there aren't enough T-Mobile towers where I'm at, and after the amount I've already paid for is up, I'm switching to Straight Talk to give them a shot.
Amazon wants you (to buy through them, because they are like Walmart, getting a sliver of each transaction, but making an enormous amount of transactions). You get liberality with content.
Google wants you (to view their ads, because every time they display an ad/make a conversion, they get a sliver). You get liberality with hardware/software.
Apple wants you (to buy an iDevice, because every time you buy an iDevice, you just gave them a 30%-50% of the product price as straight profit, in addition to being locked into their marketplace, which they get a hunk of each sale, in addition to ads being run through free apps and them getting a sliver for the view/conversion). You get a well-done, end-to-end experience.
This is one of the biggest problems to tackle when updating software: How can the user be uninterrupted? This is one of the biggest problems I have on my WP7. My apps stay out of date for a long time until I manually go update them. I don't think to manually update them very often (maybe once a month). I'd much rather the app be updated silently in the background, and when I re-open it after the updates have been downloaded/applied, I then notice that things have changed.
This is pretty much the same way websites operate and you don't have much control over it. You may be able to prolong the update while they are "beta" testing it (a la Google/Facebook), but eventually you get the update and have to deal with it.
Are people banned/ghosted here? I'm assuming so, because that's how most forums work, but I'm not sure since this seems to be custom built.
While I don't think that HN has hit this level yet, one solution that I've read about involved people who trolled or posted general nastiness. When people had reached a certain threshold or pissed off the wrong high-level mod, they got "ghosted". What this meant was that people who were "alive" or considered "living" could see "living" contributors' posts, but could not see "ghost" posts. Ghosts, however, could see other ghosts' posts, so they could, for what it was worth, writhe in their own debauchery and piss each other off, but not the pleasant and constructive members. I thought it was an interesting solution and apparently it worked very well.
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.