Textmate 2 and why you should never rewrite your code(kerr.io)
kerr.io
Textmate 2 and why you should never rewrite your code
http://kerr.io/textmate-2-and-why-you-shouldnt-rewrite-your-code/
17 comments
I read the Textmate announcement, saw some great comments here and then decided to write the article instead of composing a comment. I didn't think the article would get any traction.
Agreed about the context being key. It always is. My point was more about adding incremental value to your customers with solutions to small problems rather than going after all their problems with one big release.
Agreed about the context being key. It always is. My point was more about adding incremental value to your customers with solutions to small problems rather than going after all their problems with one big release.
Do you have any examples of Microsoft throwing away their codebase and starting fresh? I believe that is what Apple did with the OS9 to OSX transition. I can't think of too many examples where starting from scratch worked out though.
But Apple didn't really "start fresh" internally with OSX, they acquired NeXT which was already a pretty robust product when they made that decision. The abandoned the OS9 codebase, but they didn't really start over from a blank slate.
Well there was also the many-years development of Copland that ultimately failed and precipitated the purchase of NeXT just to get a decent OS.
It's hard to say whether Copland was a rewrite or a refactoring, but what is indisputable is that the root architectural decisions of Mac OS were a dead-end.
They actually had to go beyond a rewrite to build something completely new and different, and then retrofit an emulator onto it to support the old system.
It's hard to say whether Copland was a rewrite or a refactoring, but what is indisputable is that the root architectural decisions of Mac OS were a dead-end.
They actually had to go beyond a rewrite to build something completely new and different, and then retrofit an emulator onto it to support the old system.
From Joel's classic article:
"Microsoft almost made the same mistake, trying to rewrite Word for Windows from scratch in a doomed project called Pyramid which was shut down, thrown away, and swept under the rug. Lucky for Microsoft, they had never stopped working on the old code base, so they had something to ship, making it merely a financial disaster, not a strategic one."
"Microsoft almost made the same mistake, trying to rewrite Word for Windows from scratch in a doomed project called Pyramid which was shut down, thrown away, and swept under the rug. Lucky for Microsoft, they had never stopped working on the old code base, so they had something to ship, making it merely a financial disaster, not a strategic one."
> Do you have any examples of Microsoft throwing away their codebase and starting fresh? I believe that is what Apple did with the OS9 to OSX transition.
Even that wasn't "starting fresh". The starting point was a complete operating system: OPENSTEP. It was just a different operating system than the one they were using before.
Even that wasn't "starting fresh". The starting point was a complete operating system: OPENSTEP. It was just a different operating system than the one they were using before.
Windows NT? The Windows 95-98-Me series' kernel was thrown in the bin, replaced by the NT kernel when they released Windows XP.
That's a successful version of the Pyramid story. NT had been developed alongside OT (or whatever the old tech kernel was called).
They wrote a new system in tandem with maintaining the old one and when it was mature enough, swapped out old for new. This pattern is very commonplace.
They wrote a new system in tandem with maintaining the old one and when it was mature enough, swapped out old for new. This pattern is very commonplace.
NT was created in parallel, and run in parallel for a long time. Eventually the benefits of NT outweighed the 9x kernel, and computers were fast enough that the extra overhead was acceptable. 9x was EOLed and XP went with the NT kernel.
They were two different products that ended up converging enough that sharing code made sense.
They were two different products that ended up converging enough that sharing code made sense.
If there is any lesson from OS X and Windows, I think you've hit on it, exactly. The way to replace isn't to stop one and start another – instead, you need to build in parallel and have a smart migration plan.
Yes, but they had been developing the NT product line for years, so it wasn't a full start over. Then they had the Windows 2000 release which was their first stab at a "merged" kernel for desktop and pro/server. XP was the full cutover after Windows 2000 validated that it'd work.
[deleted]
Gotta call you out for a sensationalist headline and over-the-top one-sided article. People rewrite their codebases all the time, and quite often it's a very good idea.
In other words, this is a prototypical HN submission. Progblogging is so bad.
I've submitted like 3 articles to hacker news, I'm clearly not a pro.
I think mattacular wrote about the run-of-the-mill programming blog that gets submitted to HN. They feel more substantial than one of those spammy "125 free icon sets that'll turn you into a Design god", but they're mostly half-baked regurgitations of Software Engineering folk-wisdom, rehashed from a dozen other blogs which quote some testimony from a famous book.
No code, no maths, no algorithms, no statistics, no nothing.
No code, no maths, no algorithms, no statistics, no nothing.
Lessons I took away from the Mythical Man Month:
Version 1: Get to market, prove your product, make a big mess
Version 2: Make a ton of mistakes, screw everything up trying to clean up the mess but get sidetracked with all the great ideas you don't have time to implement
Version 3: Scale back, get smart, and build what version 2 should have been in less time
In my experience, most companies either get completely discouraged or worse yet just wreck themselves on version 2. The nice thing about iterative development is that the V1 - V3 spectrum happens fast and your failures are smaller.
Version 1: Get to market, prove your product, make a big mess
Version 2: Make a ton of mistakes, screw everything up trying to clean up the mess but get sidetracked with all the great ideas you don't have time to implement
Version 3: Scale back, get smart, and build what version 2 should have been in less time
In my experience, most companies either get completely discouraged or worse yet just wreck themselves on version 2. The nice thing about iterative development is that the V1 - V3 spectrum happens fast and your failures are smaller.
And yet Firefox rose from the ashes of Netscape. Who would have thought Mozilla would become relevant again five years after Netscape died? I'm starting to think open source projects are a lot more hardy than startups because they can't run out of money.
It's possible that Firefox would be more mature or popular than it already is if they never did the rewrite.
1. There's no way to confirm or falsify that counterfactual.
2. Even if we stipulate that Firefox would be more mature/popular without the rewrite, I think there are many decisions that might have had an equal or larger impact. For example, the design choices they made for their plugin architecture made it harder to compete when Chrome came along, and have had deep influence on their ability to do frequent unobtrusive releases. If you admit one counterfactual you must admit them all. Was the rewrite really the biggest turning point in their trajectory?
3. If you'd offered the firefox creators the level of success it has today, they would take it in a heartbeat. Quibbling about levels of success is a luxury you can only afford after you have managed to not die. (http://paulgraham.com/die.html) That was what my comment was concerned with.
Conventional wisdom (http://www.joelonsoftware.com/articles/fog0000000069.html) makes the far stronger claim that rewrites kill; if we retreat to arguing levels of success I will happily declare victory and go home :)
2. Even if we stipulate that Firefox would be more mature/popular without the rewrite, I think there are many decisions that might have had an equal or larger impact. For example, the design choices they made for their plugin architecture made it harder to compete when Chrome came along, and have had deep influence on their ability to do frequent unobtrusive releases. If you admit one counterfactual you must admit them all. Was the rewrite really the biggest turning point in their trajectory?
3. If you'd offered the firefox creators the level of success it has today, they would take it in a heartbeat. Quibbling about levels of success is a luxury you can only afford after you have managed to not die. (http://paulgraham.com/die.html) That was what my comment was concerned with.
Conventional wisdom (http://www.joelonsoftware.com/articles/fog0000000069.html) makes the far stronger claim that rewrites kill; if we retreat to arguing levels of success I will happily declare victory and go home :)
Doubtful. The old rendering engine was a dead end and everyone knew it. Netscape 5 still would have been miles behind IE 6, and it would have taken substantial engineering effort to get it there. Instead, they focused on Gecko and Mozilla.
Firefox would not exist without the rewrite. We'd still be stuck using Mozilla.
I prefer an alternative interpretation: if rewriting your code is an existential risk, you weren't writing it right in the first place.
Startups need to value flexibility. The longer they stay flexible -- even in the presence of growth -- the higher their valuations get. So the conclusion seems inescapable: find ways to make rewrites utterly banal. This may seem hard, but I think it's possible. Part of the answer might be replacing compatibility concerns with automated tests to a far more comprehensive level: http://news.ycombinator.com/item?id=4361596.
One benefit of Google's NIH syndrome that I think people don't focus on enough: because they built the entire stack from scratch they had no backwards compatibility overheads in the early days. (Of course this is no longer true.)
Startups need to value flexibility. The longer they stay flexible -- even in the presence of growth -- the higher their valuations get. So the conclusion seems inescapable: find ways to make rewrites utterly banal. This may seem hard, but I think it's possible. Part of the answer might be replacing compatibility concerns with automated tests to a far more comprehensive level: http://news.ycombinator.com/item?id=4361596.
One benefit of Google's NIH syndrome that I think people don't focus on enough: because they built the entire stack from scratch they had no backwards compatibility overheads in the early days. (Of course this is no longer true.)
Agree. I work for a highly successful startup that has gone through several major rewrites on subcomponents and we're fine. Rewrites are not in and of themselves a universal evil - but writing shitty code is.
Sounds more like a large refactoring than a rewrite. Or at the least you already has a service-oriented architecture with a well-defined interface that allowed the rewrite to be sane.
My rule of thumb has always been to break things down into the smallest discrete tasks possible. If you stay on top of your architecture this almost never turns into a total rewrite, but sometimes there is no choice but to get your hands really dirty in a major refactoring. In these cases it's critical that each escalation is well-justified and not just trying to capture more "low-hanging fruit" that the dev team is brainstorming in high-level discussions.
My rule of thumb has always been to break things down into the smallest discrete tasks possible. If you stay on top of your architecture this almost never turns into a total rewrite, but sometimes there is no choice but to get your hands really dirty in a major refactoring. In these cases it's critical that each escalation is well-justified and not just trying to capture more "low-hanging fruit" that the dev team is brainstorming in high-level discussions.
[deleted]
I think the idea he's trying to convey is that rewrites are a bad idea IF you already have a nice clean, simple product that works, with a large code base. I've been on projects where the entire code base was essentially trashed, except for some specific reference materials, but the code was terrible, and that was actually a pleasant experience.
Except that TM2 is vastly superior in almost every way to the original, it might be 'too late' but it is a great editor.
Perhaps TM2 was possible as a series of small changes, maybe it wasn't. But I think we should be grateful for the time spent on TM2, and that it is now open source so we can learn from it, and maybe even make it even better.
Perhaps TM2 was possible as a series of small changes, maybe it wasn't. But I think we should be grateful for the time spent on TM2, and that it is now open source so we can learn from it, and maybe even make it even better.
The problem with the approach is that rewriting from scratch means you have a bunch of great features you can't possibly implement until you've re-coded your original feature-set.
It worked fine the first time, why chuck it out? Refactor it as you go along, for sure, but to toss out good work for some silly, non-existent ideal is fucking stupid. You should only resort to that if it's quicker and cheaper than trying to maintain the existing codebase.
It worked fine the first time, why chuck it out? Refactor it as you go along, for sure, but to toss out good work for some silly, non-existent ideal is fucking stupid. You should only resort to that if it's quicker and cheaper than trying to maintain the existing codebase.
Maybe the real lesson here is, don't get in technical debt. Do it right from the start.
If only it were that simple. The problem is that you don't know what the 'right way' is until you've done it once and discovered all of your the mistakes and pitfalls.
The temptation then becomes to start over and do it right after having learned your lesson. The problem with that is that often, in addition to taking to long too start over, you just end up making a whole new set of mistakes.
The temptation then becomes to start over and do it right after having learned your lesson. The problem with that is that often, in addition to taking to long too start over, you just end up making a whole new set of mistakes.
I read the Mythical Man Month years ago and for a long time I've thought of this as the "second-system effect": You can create a much better architecture the second time around. (Hence the advantage of quick, throwaway prototypes.)
The article (and wikipedia) tell me I'm wrong on Brooks' meaning.
The article (and wikipedia) tell me I'm wrong on Brooks' meaning.
> The problem is that you don't know what the 'right way' is until you've done it once and discovered all of your the mistakes and pitfalls.
Isn't that what experience is? (As opposed to talent.)
Isn't that what experience is? (As opposed to talent.)
But experience can't always prevent mistakes and pitfalls because you can't always predict the future. This is especially prevalent in the land of startups where you can think the most important thing is X, you build a bunch of stuff for it, then 3 months later the most important thing is Y, then a 3 months later the most important thing is Z, and all of the sudden you have a bunch of technical debt.
Even if you could build X, Y, and Z "right the first time" those 3 month transitions may become 6 month transitions, and now you're out of business. (to be clear, those numbers are made up as an example)
There are obviously best practices to be followed so you don't have big screwups or an unneededly bloated system, but sometimes these problems can't be prevented because you can't always predict the future.
Even if you could build X, Y, and Z "right the first time" those 3 month transitions may become 6 month transitions, and now you're out of business. (to be clear, those numbers are made up as an example)
There are obviously best practices to be followed so you don't have big screwups or an unneededly bloated system, but sometimes these problems can't be prevented because you can't always predict the future.
Also most companies don't live in happy land, where you are allowed infinite time to do this things.
It can be "I have one day, to a 4 day job", better hack it sort of thing.
It can be "I have one day, to a 4 day job", better hack it sort of thing.
Are there really no instances of rewrites succeeding? Are we sure that projects that failed doing rewrites wouldn't have failed more painfully if they'd continued dealing with the cruft of the old codebase?
It's possible the problem is more in how these projects went about their rewrites, than just the fact that they did. What was the scope of the rewrite - did they simultaneously re-architect, redesign the UI, and try to implement all their hoped-for features (second-system effect)? Did they try to re-use old modules (temporarily or permanently - did they not reuse enough, or did they invest too much time making old code compatible)? Did the entire team immediately switch over to working on the rewrite, or did it get too few resources?
It might be worth investigating how a rewrite could possibly be done successfully, instead of assuming that the only option is to slowly refactor.
It's possible the problem is more in how these projects went about their rewrites, than just the fact that they did. What was the scope of the rewrite - did they simultaneously re-architect, redesign the UI, and try to implement all their hoped-for features (second-system effect)? Did they try to re-use old modules (temporarily or permanently - did they not reuse enough, or did they invest too much time making old code compatible)? Did the entire team immediately switch over to working on the rewrite, or did it get too few resources?
It might be worth investigating how a rewrite could possibly be done successfully, instead of assuming that the only option is to slowly refactor.
> Are there really no instances of rewrites succeeding?
The most famous instance of a successful rewrite was IBM's transition to OS/360 [1] in the 1960s, which was not just a complete rewrite but a completely new system to replace all the previous ones. (The "360" comes from the number of degrees in a full turn.)
Who spearheaded this huge effort? A certain Frederick P. Brooks, who used his experience with the project to write _The Mythical Man Month_, which coined the term "second-system effect" [3].
Of course, while the 360 project was _ultimately_ successful, they made all the now-familiar mistakes that Brooks documented in his book -- it was an extremely expensive and time-consuming project that was only possible thanks to IBM's enormous resources and strong leadership.
---
[1] http://www-01.ibm.com/software/os/systemz/pdf/360Revolution_...
[2] http://en.wikipedia.org/wiki/The_Mythical_Man-Month
[3] http://en.wikipedia.org/wiki/Second-system_effect
The most famous instance of a successful rewrite was IBM's transition to OS/360 [1] in the 1960s, which was not just a complete rewrite but a completely new system to replace all the previous ones. (The "360" comes from the number of degrees in a full turn.)
Who spearheaded this huge effort? A certain Frederick P. Brooks, who used his experience with the project to write _The Mythical Man Month_, which coined the term "second-system effect" [3].
Of course, while the 360 project was _ultimately_ successful, they made all the now-familiar mistakes that Brooks documented in his book -- it was an extremely expensive and time-consuming project that was only possible thanks to IBM's enormous resources and strong leadership.
---
[1] http://www-01.ibm.com/software/os/systemz/pdf/360Revolution_...
[2] http://en.wikipedia.org/wiki/The_Mythical_Man-Month
[3] http://en.wikipedia.org/wiki/Second-system_effect
It depends on the individual case. It's not always possible to isolate subsystems in order to refactor incrementally, but on the other hand it's also almost certainly a foregone conclusion that you will underestimate the complexity of a rewrite.
The Netscape rewrite became Firefox.
I was involved in a complete project rewrite, it was mostly a success but it took a long, long time. The biggest issue is you have a clean slate but a legacy code base that you still need to support, bug fix and frequently improve. Big time sink right there.
> Look at it as a living organism that can perhaps be healed, and can evolve.
What life we can see now is only survivors and most of gene line actually failed.
What life we can see now is only survivors and most of gene line actually failed.
Grub2 is a complete re-write
[deleted]
It's scary to me that someone thinks it's ok to clone a website so precisely. This is an identical clone of the svbtle style.
It's a wordpress plugin: https://github.com/gravityonmars/wp-svbtle
On the subject of rewrites, with the benefit of hindsight it's easy to point out when it was a bad idea. But clearly it doesn't always have to be that way. After all, it never hurt Microsoft on occasion, and Apple.
As almost always, context is key.