How I Fired Myself(edu.mkrecny.com)
edu.mkrecny.com
How I Fired Myself
http://edu.mkrecny.com/thoughts/how-i-fired-myself
411 comments
More than anything else, this describes an appalling failure at every level of the company's technical infrastructure to ensure even a basic degree of engineering rigor and fault tolerance. It's noble of the author to quit, but it's not his fault. I cannot believe they would have the gall to point the blame at a junior developer. You should expect humans to fail: humans are fallible. That's why you automate.
News flash,
If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"
If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?"
They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are.
As an aside: Way back in time I caused about ten thousand companies to have to refile some pretty important government documents because I was doubling xml decoding (& became &). My boss actually laughed and was like "we should have caught this a long time ago"... by we he actually meant himself and support.
If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"
If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?"
They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are.
As an aside: Way back in time I caused about ten thousand companies to have to refile some pretty important government documents because I was doubling xml decoding (& became &). My boss actually laughed and was like "we should have caught this a long time ago"... by we he actually meant himself and support.
This is certainly a monumental fuckup, but these things inevitably happen even with better development practices, this is why you need backups, preferably daily, and as much separation of concerns and responsibilities as humanly possible.
Anecdote:
I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important later) table with lots of imported data. One day, I made some trivial looking migration (added a flag column to that table). I tested it locally, rolled it out to staging server. Everything seemed A-OK until we started migration on the production server. Suddenly, everything broke. By everything, I mean EVERYTHING, our web application showed massive 500-s, total DEFCON1 across the whole company. It turned out we ran out of disk space, since apparently myisam tables are altered the following way: first the table is created with updated schema, then it is populated with data from the old table. MyISAM ran out of disk space and somehow corrupted the existing tables, mysql server would start with blank tables, with all data lost.
I can confirm this very feeling: "The implications of what I'd just done didn't immediately hit me. I first had a truly out-of-body experience, seeming to hover above the darkened room of hackers, each hunched over glowing terminals." Also, I distinctly remember how I shivered and my hands shook. It felt like my body temperature fell by several degrees.
Fortunately for me, there was a daily backup routine in place. Still, several hour long outage and lots of apologies to angry clients.
"There are two types of people in this world, those who have lost data, and those who are going to lose data"
Anecdote:
I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important later) table with lots of imported data. One day, I made some trivial looking migration (added a flag column to that table). I tested it locally, rolled it out to staging server. Everything seemed A-OK until we started migration on the production server. Suddenly, everything broke. By everything, I mean EVERYTHING, our web application showed massive 500-s, total DEFCON1 across the whole company. It turned out we ran out of disk space, since apparently myisam tables are altered the following way: first the table is created with updated schema, then it is populated with data from the old table. MyISAM ran out of disk space and somehow corrupted the existing tables, mysql server would start with blank tables, with all data lost.
I can confirm this very feeling: "The implications of what I'd just done didn't immediately hit me. I first had a truly out-of-body experience, seeming to hover above the darkened room of hackers, each hunched over glowing terminals." Also, I distinctly remember how I shivered and my hands shook. It felt like my body temperature fell by several degrees.
Fortunately for me, there was a daily backup routine in place. Still, several hour long outage and lots of apologies to angry clients.
"There are two types of people in this world, those who have lost data, and those who are going to lose data"
Tens of thousands of paying customers and no backups?
No staging environment (from which ad-hoc backups could have been restored)!?!?
No regular testing of backups to ensure they work?
No local backups on dev machines?!?
Using a GUI tool for db management on the live db?!?!?
No migrations!?!?!
Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?!
What an astonishing failure of process. The higher ups are definitely far more responsible than some junior developer for this, he shouldn't have been allowed near the live database in the first place until he was ready to take changes live, and then only on to a staging environment using migrations of some kind which could then be replayed on live.
They need one of these to start with, then some process:
http://www.bnj.com/cowboy-coding-pink-sombrero/
No staging environment (from which ad-hoc backups could have been restored)!?!?
No regular testing of backups to ensure they work?
No local backups on dev machines?!?
Using a GUI tool for db management on the live db?!?!?
No migrations!?!?!
Junior devs (or any devs) testing changes on the live db and wiping tables?!?!?!
What an astonishing failure of process. The higher ups are definitely far more responsible than some junior developer for this, he shouldn't have been allowed near the live database in the first place until he was ready to take changes live, and then only on to a staging environment using migrations of some kind which could then be replayed on live.
They need one of these to start with, then some process:
http://www.bnj.com/cowboy-coding-pink-sombrero/
When I was 18 I took out half my towns power for 30 minutes with a bad scada command. It was my summer job before college and I went from cleaning the warehouse to programming the main SCADA control system in a couple weeks.
Alarms went off, people came running in freaking out, trucks started rolling out to survey the damage, hospitals started calling about people on life support and how the backup generators were not operational, old people started calling about how they require AC to stay alive and should they take their loved ones on machines to the hospital soon.
My boss was pretty chill about it. "Now you know not to do that" were his words of wisdom, and I continued programming the system for the next 4 summers with no real mistakes.
Alarms went off, people came running in freaking out, trucks started rolling out to survey the damage, hospitals started calling about people on life support and how the backup generators were not operational, old people started calling about how they require AC to stay alive and should they take their loved ones on machines to the hospital soon.
My boss was pretty chill about it. "Now you know not to do that" were his words of wisdom, and I continued programming the system for the next 4 summers with no real mistakes.
Whoever was your boss should have taken responsibility. Someone gave you access to the production database instead of setting up a proper development and testing environment. For a company doing "millions" in revenues, it's odd that they wouldn't think of getting someone with a tiny bit of experience to manage the development team.
The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue".
No, the CEO was at fault, as was whoever let you develop against the production database.
If the CEO had any sense, he should have put you in charge of fixing the issue and then making sure it could never happen again. Taking things further, they could have asked you to find other worrying areas, and come up with fixes for those before something else bad happens.
I have no doubt that you would have taken the task extremely seriously, and the company would have ended up in a better place.
Instead, they're down an employee, and the remaining employees know that if they make a mistake, they'll be out of the door.
And they still have an empty users table.
No, the CEO was at fault, as was whoever let you develop against the production database.
If the CEO had any sense, he should have put you in charge of fixing the issue and then making sure it could never happen again. Taking things further, they could have asked you to find other worrying areas, and come up with fixes for those before something else bad happens.
I have no doubt that you would have taken the task extremely seriously, and the company would have ended up in a better place.
Instead, they're down an employee, and the remaining employees know that if they make a mistake, they'll be out of the door.
And they still have an empty users table.
I was in a situation very similar to yours. Also a game dev company, also lots of user data etc etc. We did have test/backup databases for testing, but some data was just on live and there was no way for me to build those reports other than to query the live database when the load was lower.
In any case, I did a few things to make sure I never ended up destroying any data. Creating temporary tables and then manipulating those.. reading over my scripts for hours.. dumping table backups before executing any scripts.. not executing scripts in the middle/end of the day, only mornings when I was fresh etc etc.
I didn't mess up, but I remember how incredibly nerve wracking that was, and I can relate to the massive amount of responsibility it places on a "junior" programmer. It just should never be done. Like others have said, you should never have been in that position. Yes, it was your fault, but this kind of responsibility should never have been placed on you (or anyone, really). Backing up all critical data (what kind of company doesn't backup its users table?! What if there had been hard disk corruption?), and being able to restore in minimum time should have been dealt with by someone above your pay grade.
In any case, I did a few things to make sure I never ended up destroying any data. Creating temporary tables and then manipulating those.. reading over my scripts for hours.. dumping table backups before executing any scripts.. not executing scripts in the middle/end of the day, only mornings when I was fresh etc etc.
I didn't mess up, but I remember how incredibly nerve wracking that was, and I can relate to the massive amount of responsibility it places on a "junior" programmer. It just should never be done. Like others have said, you should never have been in that position. Yes, it was your fault, but this kind of responsibility should never have been placed on you (or anyone, really). Backing up all critical data (what kind of company doesn't backup its users table?! What if there had been hard disk corruption?), and being able to restore in minimum time should have been dealt with by someone above your pay grade.
If it helps explain things, the only experience the CEO had before this social game shop was running a literal one-man yogurt shop.
This happened a week before I started as a Senior Software Engineer. I remember getting pulled into a meeting where several managers who knew nothing about technology were desperately trying to place blame, figure out how to avoid this in the future, and so on.
"There should have been automated backups. That's really the only thing inexcusable here.", I said.
The "producer" (no experience, is now a director of operations, I think?) running the meeting said that was all well and good, but what else could we do to ensure that nobody makes this mistake again? "People are going to make mistakes", I said, "what you need to focus on is how to prevent it from sinking the company. All you need for that is backups. It's not the engineer's fault.". I was largely ignored (which eventually proved to be a pattern) and so went on about my business.
And business was dumb. I had to fix an awful lot of technical things in my time there.
When I started, only half of the client code was in version control. And it wasn't even the most recent shipped version. Where was the most recent version? On a Mac Mini that floated around the office somewhere. People did their AS3 programming in notepad or directly on the timeline. There were no automated builds, and builds were pushed from peoples' local machines -often contaminated by other stuff they were working on. Art content live on our CDN may have had source (PSD/FLA) distributed among a dozen artist machines, or else the source for it was completely lost.
That was just the technical side. The business/management side was and is actually more hilarious. I have enough stories from that place to fill a hundred posts, but you can probably get a pretty good idea by imagining a yogurt-salesman-cum-CEO, his disbarred ebay art fraudster partner, and other friends directing the efforts of senior software engineers, artists, and other game developers. It was a god damn sitcom every day. Not to mention all of the labor law violations. Post-acquisition is a whole 'nother anthology of tales of hilarious incompetence. I should write a book.
I recall having lunch with the author when he asked me "What should I do?". I told him that he should leave. In hindsight, it might have been the best advice I ever gave.
This happened a week before I started as a Senior Software Engineer. I remember getting pulled into a meeting where several managers who knew nothing about technology were desperately trying to place blame, figure out how to avoid this in the future, and so on.
"There should have been automated backups. That's really the only thing inexcusable here.", I said.
The "producer" (no experience, is now a director of operations, I think?) running the meeting said that was all well and good, but what else could we do to ensure that nobody makes this mistake again? "People are going to make mistakes", I said, "what you need to focus on is how to prevent it from sinking the company. All you need for that is backups. It's not the engineer's fault.". I was largely ignored (which eventually proved to be a pattern) and so went on about my business.
And business was dumb. I had to fix an awful lot of technical things in my time there.
When I started, only half of the client code was in version control. And it wasn't even the most recent shipped version. Where was the most recent version? On a Mac Mini that floated around the office somewhere. People did their AS3 programming in notepad or directly on the timeline. There were no automated builds, and builds were pushed from peoples' local machines -often contaminated by other stuff they were working on. Art content live on our CDN may have had source (PSD/FLA) distributed among a dozen artist machines, or else the source for it was completely lost.
That was just the technical side. The business/management side was and is actually more hilarious. I have enough stories from that place to fill a hundred posts, but you can probably get a pretty good idea by imagining a yogurt-salesman-cum-CEO, his disbarred ebay art fraudster partner, and other friends directing the efforts of senior software engineers, artists, and other game developers. It was a god damn sitcom every day. Not to mention all of the labor law violations. Post-acquisition is a whole 'nother anthology of tales of hilarious incompetence. I should write a book.
I recall having lunch with the author when he asked me "What should I do?". I told him that he should leave. In hindsight, it might have been the best advice I ever gave.
So the person who made a split-second mistake while doing his all for the business was pressured into resigning - basically, got fired.
What I want to know is what happened to whoever decided that backups were a dispensable luxury? In 2010?
There's a rule that appears in Jerry Weinberg's writings - the person responsible for a X million dollar mistake (and who should be fired over such a mistake) is whoever has controlling authority over X million dollars' worth of the company's activities.
A company-killing mistake should result in the firing of the CEO, not in that of the low-level employee who committed the mistake. That's what C-level responsibility means.
(I had the same thing happen to me in the late 1990's, got fired over it. Sued my employer, who opted to settle out of court for a good sum of money to me. They knew full well they had no leg to stand on.)
What I want to know is what happened to whoever decided that backups were a dispensable luxury? In 2010?
There's a rule that appears in Jerry Weinberg's writings - the person responsible for a X million dollar mistake (and who should be fired over such a mistake) is whoever has controlling authority over X million dollars' worth of the company's activities.
A company-killing mistake should result in the firing of the CEO, not in that of the low-level employee who committed the mistake. That's what C-level responsibility means.
(I had the same thing happen to me in the late 1990's, got fired over it. Sued my employer, who opted to settle out of court for a good sum of money to me. They knew full well they had no leg to stand on.)
Klicknation is hiring. Of themselves, they say:
"We make astonishingly fun, ferociously addictive games that run on social networks. ...KlickNation boasts a team of extremely smart, interesting people who have, between them, built several startups (successful and otherwise); written a novel; directed music videos; run game fan sites; illustrated for Marvel Comics and Dynamite Entertainment with franchises like Xmen, Punisher, and Red Sonja; worked on hit games like Tony Hawk and X-Men games; performed in rock bands; worked for independent and major record lables; attended universities like Harvard, Stanford, Dartmouth, UC Berkeley; received a PhD and other fancy degrees; and built a fully-functional MAME arcade machine."
And this is hilarious: their "careers" page gives me a 404:
http://www.klicknation.com/careers/
That link to "careers" is from this page:
http://www.klicknation.com/contact/
I am tempted to apply simply to be able to ask them about this. It would be interesting to hear if they have a different version of this story, if it is all true.
"We make astonishingly fun, ferociously addictive games that run on social networks. ...KlickNation boasts a team of extremely smart, interesting people who have, between them, built several startups (successful and otherwise); written a novel; directed music videos; run game fan sites; illustrated for Marvel Comics and Dynamite Entertainment with franchises like Xmen, Punisher, and Red Sonja; worked on hit games like Tony Hawk and X-Men games; performed in rock bands; worked for independent and major record lables; attended universities like Harvard, Stanford, Dartmouth, UC Berkeley; received a PhD and other fancy degrees; and built a fully-functional MAME arcade machine."
And this is hilarious: their "careers" page gives me a 404:
http://www.klicknation.com/careers/
That link to "careers" is from this page:
http://www.klicknation.com/contact/
I am tempted to apply simply to be able to ask them about this. It would be interesting to hear if they have a different version of this story, if it is all true.
One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired."
Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about.
And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.
Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about.
And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.
"I found myself on the phone to Rackspace, leaning on a desk for support, listening to their engineer patiently explain that backups for this MySQL instance had been cancelled over 2 months ago."
Here's something I don't get: didn't Rackspace have their own daily backups of the production server, e.g. in case their primary facility was annihilated by a meteor (or some more mundane reason, like hard drive corruption)?
Regardless, here's a thought experiment: suppose that Rackspace did keep daily backups of every MySQL instance in their care, even if you're not paying for the backup service. Now suppose they get a frantic call from a client who's not paying for backups, asking if they have any. How much of a ridiculous markup would Rackspace need to charge to give the client access this unpaid-for backup, in order to make the back-up-every-database policy profitable? I'm guessing this depends on 1) the frequency of frantic phone calls, 2) the average size of a database that they aren't being paid to back up, and 3) the importance and irreplacebility of the data that they're handling (and 4) the irresponsibility of their major clients).
Here's something I don't get: didn't Rackspace have their own daily backups of the production server, e.g. in case their primary facility was annihilated by a meteor (or some more mundane reason, like hard drive corruption)?
Regardless, here's a thought experiment: suppose that Rackspace did keep daily backups of every MySQL instance in their care, even if you're not paying for the backup service. Now suppose they get a frantic call from a client who's not paying for backups, asking if they have any. How much of a ridiculous markup would Rackspace need to charge to give the client access this unpaid-for backup, in order to make the back-up-every-database policy profitable? I'm guessing this depends on 1) the frequency of frantic phone calls, 2) the average size of a database that they aren't being paid to back up, and 3) the importance and irreplacebility of the data that they're handling (and 4) the irresponsibility of their major clients).
I really feel sorry for this guy. Accidents happen, which is why development happens in a sandboxed copy of the live system and why back ups are essential. It simple shouldn't be possible (or at least, that easy) for human error to put an entire company in jeopardy.
Take my own company, I've accidentally deleted /dev on development servers (not that major of an issue thanks to udev, but the timing of the mistake was lousy), a co-worker recently dropped a critical table on dev database and we've had other engineers break Solaris by carelessly punching in chmod -R / as root (we've since revised engineers permissions so this is no longer possible). As much as those errors are stupid and as much as engineers of our calibre should know better, it can only takes a minor lack of concentration at the wrong moment to make a major fsck up. Which is doubly scary when you consider how many interruptions the average engineer gets a day.
So I think the real guilt belongs to the entire technical staff as this is a cascade of minor fcsk ups that lead to something catastrophic.
Take my own company, I've accidentally deleted /dev on development servers (not that major of an issue thanks to udev, but the timing of the mistake was lousy), a co-worker recently dropped a critical table on dev database and we've had other engineers break Solaris by carelessly punching in chmod -R / as root (we've since revised engineers permissions so this is no longer possible). As much as those errors are stupid and as much as engineers of our calibre should know better, it can only takes a minor lack of concentration at the wrong moment to make a major fsck up. Which is doubly scary when you consider how many interruptions the average engineer gets a day.
So I think the real guilt belongs to the entire technical staff as this is a cascade of minor fcsk ups that lead to something catastrophic.
Last year I worked at a start-up that had manually created accounts for a few celebrities when they launched, in a gutsy and legally grey bid to improve their proposition†. While refactoring the code that handled email opt-out lists I missed a && at the end of a long conditional and failed to notice a second, otherwise unused opt-out system that dealt specifically with these users. It was there to ensure they really, really never got emailed. The result?
http://krugman.blogs.nytimes.com/2011/08/11/academia-nuts/
What a screw up!
These mistakes are almost without fail a healthy mix of individual incompetence and organisational failure. Many things - mostly my paying better attention to functionality I rewrite, but also the company not having multiple undocumented systems for one task, or code review, or automated testing - might have saved the day.
[†] They've long been removed.
http://krugman.blogs.nytimes.com/2011/08/11/academia-nuts/
What a screw up!
These mistakes are almost without fail a healthy mix of individual incompetence and organisational failure. Many things - mostly my paying better attention to functionality I rewrite, but also the company not having multiple undocumented systems for one task, or code review, or automated testing - might have saved the day.
[†] They've long been removed.
Once, a long time ago, I spent the best part of a night writing a report for college, on an Amstrad PPC640 (http://en.wikipedia.org/wiki/PPC_512).
Once I was finished, I saved the document -- "Save" took around two minutes (which is why I rarely saved).
I had an external monitor that was sitting next to the PC; while the saving operation was under way, I decided I should move the monitor.
The power switch was on top of the machine (unusual design). While moving the monitor I inadvertently touched this switch and turned the PC of... while it was writing the file.
The file was gone, there was no backup, no previous version, nothing.
I had moved the monitor in order to go to bed, but I didn't go to bed that night. I moved the monitor back to where it was, and spent the rest of the night recreating the report, doing frequent backups on floppy disks, with incremental version names.
This was in 1989. I've never lost a file since.
Once I was finished, I saved the document -- "Save" took around two minutes (which is why I rarely saved).
I had an external monitor that was sitting next to the PC; while the saving operation was under way, I decided I should move the monitor.
The power switch was on top of the machine (unusual design). While moving the monitor I inadvertently touched this switch and turned the PC of... while it was writing the file.
The file was gone, there was no backup, no previous version, nothing.
I had moved the monitor in order to go to bed, but I didn't go to bed that night. I moved the monitor back to where it was, and spent the rest of the night recreating the report, doing frequent backups on floppy disks, with incremental version names.
This was in 1989. I've never lost a file since.
This happened to me once on a much smaller scale. Forgot the "where" clause on a DELETE statement. My screwup, obviously.
We actually had a continuous internal backup plan, but when I requested a restore, the IT guy told me they were backing up everything but the databases, since "they were always in use."
(Let that sink in for a second. The IT team actually thought that was an acceptable state of affairs: "Uh, yeah! We're backing up! Yeah! Well, some things. Most things. The files that don't get like, used and stuff.")
That day was one of the lowest feelings I ever had, and that screwup "only" cost us a few thousand dollars as opposed to the millions of dollars the blog post author's mistake cost the company. I literally can't imagine how he felt.
We actually had a continuous internal backup plan, but when I requested a restore, the IT guy told me they were backing up everything but the databases, since "they were always in use."
(Let that sink in for a second. The IT team actually thought that was an acceptable state of affairs: "Uh, yeah! We're backing up! Yeah! Well, some things. Most things. The files that don't get like, used and stuff.")
That day was one of the lowest feelings I ever had, and that screwup "only" cost us a few thousand dollars as opposed to the millions of dollars the blog post author's mistake cost the company. I literally can't imagine how he felt.
I know how you felt. Many years ago when I was a junior working in a casual game company, I were to add a bunch of credits to a poker player (fake money). I forget the where in the SQL clause and added credits to every player in our database. Lucky me it was an add and not a set and I could revert it. Another time I was going to shutdown my pc (a debian box) using "shutdown -h now" and totally forgot that I was in a ssh session to our main game server. I had to call the tech support overseas and tell him to physically turn on the server...
You did them more good than harm.
1) Not having backups is an excuse-less monumental fuckup.
2) Giving anyone delete access to your production db, especially a junior dev through a GUI tool, is an excuse-less monumental fuckup.
Hopefully they rectified these two problems and are now a stronger company for it.
1) Not having backups is an excuse-less monumental fuckup.
2) Giving anyone delete access to your production db, especially a junior dev through a GUI tool, is an excuse-less monumental fuckup.
Hopefully they rectified these two problems and are now a stronger company for it.
If you ever notice that your employer or client isn't backing up important data, take a tip from me: do a backup, today, in your free time, and if possible, and again in your free time, create the most basic regular backup system you can.
When the time comes, and someone screws up, you will seem like a god when you deliver your backup, whether it's a 3-month-old one-off, or from your crappy daily backup system.
When the time comes, and someone screws up, you will seem like a god when you deliver your backup, whether it's a 3-month-old one-off, or from your crappy daily backup system.
> The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue".
Yes, it is a monumental fuck-up. You put a button in front of a junior developer that can cost the company millions if he accidentally clicks it and doesn't even have undo.
Yes, it is a monumental fuck-up. You put a button in front of a junior developer that can cost the company millions if he accidentally clicks it and doesn't even have undo.
Mistakes happen, and there should have been better safeguards -- backups, locking down production, management oversight.
But, I actually applaud how he tried to take responsibility for his actions and apologized. Both "junior" AND "senior" people have a hard time doing this. I've seen experienced people shrug and unapologetically go home at 6pm after doing something equivalent to this.
The unfortunate thing here seems to be that he took his own actions so personally. He made an honest mistake, and certainly there were devastating consequences, but it's important to separate the behavior from the person. I hope he realizes this in time and forgives himself.
But, I actually applaud how he tried to take responsibility for his actions and apologized. Both "junior" AND "senior" people have a hard time doing this. I've seen experienced people shrug and unapologetically go home at 6pm after doing something equivalent to this.
The unfortunate thing here seems to be that he took his own actions so personally. He made an honest mistake, and certainly there were devastating consequences, but it's important to separate the behavior from the person. I hope he realizes this in time and forgives himself.
There are several reasons why you should not feel guilty. The company was asking for trouble and you just happen to be the trigger. These are the three top things that could prevented that incident.
1) A cron job for the manually task you were doing.
2) Not working directly on production.
3) Having daily backups
And this could happened to anybody. After midnight any of us are at junior level and very prone to do this kind of mistakes.
1) A cron job for the manually task you were doing.
2) Not working directly on production.
3) Having daily backups
And this could happened to anybody. After midnight any of us are at junior level and very prone to do this kind of mistakes.
I cannot believe that people still don't have reliable back up in place.
My feeling is this: If you are in any way responsible for data that is not backed up, you should be fired or resign right now. You should never work in IT, in anyway, ever again. If you are the CEO of a company in a similar state, again, fire your self right now. Vow to never ever run a business again. This is 2013. And guess what? You still can't buy your unique data back from PCWorld. Your data is "the precious".
As for the treatment of this guy, IMHO, his employers were the worst kind of spineless cowards. This was 100% the fault of the management, and you know what? They know it. To not have backups is negligent, and should result in a high up firings. Yet these limp cowards sought to blame this kid. Pure corporate filth of the lowest order. Even the fact he was junior is irrelevant, any one could have done that, more likely a cocky senior taking some short cut. Let me tell you now, I have made a similar cock up, and I think I know it all. But I had backups, and lucky for me, it was out of business hours. Quick restore, and the users never knew. I did fess up to my team since I thought it had direct value as a cautionary tail.
Frankly, I am utterly amazed and gutted that such a thing can still happen. The corporate cowardice is sadly expected, but to not have backups is literally unforgivable negligence.
Yeah, Im quite fundamentalist about data and backups. I'd almost refer to my self as a backup jihadist.
My feeling is this: If you are in any way responsible for data that is not backed up, you should be fired or resign right now. You should never work in IT, in anyway, ever again. If you are the CEO of a company in a similar state, again, fire your self right now. Vow to never ever run a business again. This is 2013. And guess what? You still can't buy your unique data back from PCWorld. Your data is "the precious".
As for the treatment of this guy, IMHO, his employers were the worst kind of spineless cowards. This was 100% the fault of the management, and you know what? They know it. To not have backups is negligent, and should result in a high up firings. Yet these limp cowards sought to blame this kid. Pure corporate filth of the lowest order. Even the fact he was junior is irrelevant, any one could have done that, more likely a cocky senior taking some short cut. Let me tell you now, I have made a similar cock up, and I think I know it all. But I had backups, and lucky for me, it was out of business hours. Quick restore, and the users never knew. I did fess up to my team since I thought it had direct value as a cautionary tail.
Frankly, I am utterly amazed and gutted that such a thing can still happen. The corporate cowardice is sadly expected, but to not have backups is literally unforgivable negligence.
Yeah, Im quite fundamentalist about data and backups. I'd almost refer to my self as a backup jihadist.
Just wondering, when consulting I usually take care that there are appropriate clauses in the contract to make me not liable. But what is the rule for employees, are they automatically insured?
In Germany there is the concept of "Fahrlässig" (negligence) and "severe negligence". Per law you are already liable if you are just negligent, but it is possible to lower it to severe negligence in the contract. That is my understanding anyway (not a lawyer). Usually I also try to kind of weasel out of it by saying the client is responsible for appropriate testing and stuff like that... Overall it is a huge problem, though, especially if the client has a law department. Getting insurance is quite expensive because it's easy to create millions of dollars in damages in IT.
Before court "standard best practices" can become an issue, too. This worries me because I don't agree with all the latest fads in software development. It seems possible that in the future x% test coverage could be required by law, for example. Or even today a client could argue that I didn't adhere to standard best practices if I don't have at least 80% test coverage (or whatever, not sure what a reasonable number would be).
In Germany there is the concept of "Fahrlässig" (negligence) and "severe negligence". Per law you are already liable if you are just negligent, but it is possible to lower it to severe negligence in the contract. That is my understanding anyway (not a lawyer). Usually I also try to kind of weasel out of it by saying the client is responsible for appropriate testing and stuff like that... Overall it is a huge problem, though, especially if the client has a law department. Getting insurance is quite expensive because it's easy to create millions of dollars in damages in IT.
Before court "standard best practices" can become an issue, too. This worries me because I don't agree with all the latest fads in software development. It seems possible that in the future x% test coverage could be required by law, for example. Or even today a client could argue that I didn't adhere to standard best practices if I don't have at least 80% test coverage (or whatever, not sure what a reasonable number would be).
Whoever cancelled the backups was equally responsible
Sounds like no one else at that company had any more of a clue what they were doing than you did. The whole scenario is horrifying.
I've did a very similar thing after one year working at my company instead of clearing the whole user table I replaced every user information with my account information.
I forgot to copy the WHERE part of the query .....
The only difference is that it was policy to manually do a backup before doing anything on production and the problem was restored in less than 10 minutes. Even if I had forgotten to make a backup manually we had a daily complete backup and an incremental one every couple of hours.
I forgot to copy the WHERE part of the query .....
The only difference is that it was policy to manually do a backup before doing anything on production and the problem was restored in less than 10 minutes. Even if I had forgotten to make a backup manually we had a daily complete backup and an incremental one every couple of hours.
If I were the author, I would rewrite this and reflect on what was actually wrong here. At the end of the day, you resigned out of shame for a serious incident that you triggered.
But the fact that the organization allowed you to get to that point is the issue. Forget about the engineering issues and general organizational incompetence... the human side is the most incredibly, amazingly ridiculous.
I respect your restraint. If I was singled out with physical intimidation by some asshat boss while getting browbeaten by some other asshat via Skype, I probably would have taken a swing at the guy.
Competent leadership would start with a 5-why's exercise. Find out why it happened, why even the simplest controls were not implemented. I've worked in places running on a shoestring, but the people in charge were well aware of what that meant.
But the fact that the organization allowed you to get to that point is the issue. Forget about the engineering issues and general organizational incompetence... the human side is the most incredibly, amazingly ridiculous.
I respect your restraint. If I was singled out with physical intimidation by some asshat boss while getting browbeaten by some other asshat via Skype, I probably would have taken a swing at the guy.
Competent leadership would start with a 5-why's exercise. Find out why it happened, why even the simplest controls were not implemented. I've worked in places running on a shoestring, but the people in charge were well aware of what that meant.
The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue". His co-founder (remotely present via Skype) chimed in "you're lucky to still be here".
This is when you should have left. That's no way to manage a crisis.
This is when you should have left. That's no way to manage a crisis.