Google's ‘Gopher Team’(wired.com)
wired.com
Google's ‘Gopher Team’
http://www.wired.com/wiredenterprise/2013/07/gopher/
39 comments
I think this was initially reported in 2012 [1], because I remember reading about it a while ago. While I was Googling about for the link, I came across the "dl.google.com: Powered by Go" slide deck [2].
[1] http://grokbase.com/t/gg/golang-nuts/12asyfnbea/go-nuts-dl-g...
[2] http://talks.golang.org/2013/oscon-dl.slide#1
[1] http://grokbase.com/t/gg/golang-nuts/12asyfnbea/go-nuts-dl-g...
[2] http://talks.golang.org/2013/oscon-dl.slide#1
"If code doesn't receive constant love, it turns to shit."
But it sounds like this code was receiving "love", only the "love" was coming from run-of-the-mill "just get it to work" C++ programmers.
I guess we need context to understand Fitzpatrick's statement. Perhaps he just means code at Google.
Are there any examples of code that has survived for many years without "constant love"? Netcat has not received "constant love" over the years. It hasn't turned to shit. Neither has the original awk. I can think of many other examples. These programs have proven to need very little maintenance.
I posit that simple programs that are well written do not need "constant love". They only need love when there's a bug. And there are plenty of programs that are in constant use where no bug has been discovered for many years. The bugs were vetted and fixed early on, decades ago.
Hence I disagree with Fitzpatrick.
But it sounds like this code was receiving "love", only the "love" was coming from run-of-the-mill "just get it to work" C++ programmers.
I guess we need context to understand Fitzpatrick's statement. Perhaps he just means code at Google.
Are there any examples of code that has survived for many years without "constant love"? Netcat has not received "constant love" over the years. It hasn't turned to shit. Neither has the original awk. I can think of many other examples. These programs have proven to need very little maintenance.
I posit that simple programs that are well written do not need "constant love". They only need love when there's a bug. And there are plenty of programs that are in constant use where no bug has been discovered for many years. The bugs were vetted and fixed early on, decades ago.
Hence I disagree with Fitzpatrick.
[deleted]
Ah, the joys of language PR. Alan Kay was right when he bemoaned the eternal pop culture of programming.
No they don't. One guy started a major rewrite of a broken piece of infrastructure, and "the Go team [which he is part of] now regularly volunteers to help other teams with small projects", so as to learn more about how Go can replace and augment existing systems.
This headline is so far beyond journalism as to be a joke. It's just making shit up for the sake of having a "story".
I can imagine it now. They heard "gopher", thought of another animal, then thought about how awesome some of the coders on the Go team are (look at the blinking lights!), and BAM! invented an elite squadron of coders who fix whatever problem you have based on a combination of the navy Seals and the A-team.
This headline is so far beyond journalism as to be a joke. It's just making shit up for the sake of having a "story".
I can imagine it now. They heard "gopher", thought of another animal, then thought about how awesome some of the coders on the Go team are (look at the blinking lights!), and BAM! invented an elite squadron of coders who fix whatever problem you have based on a combination of the navy Seals and the A-team.
Being married to a Journalist and also hanging out with her journalist friends has made me realize that, VERY OFTEN, the person writing the article isn't the one that writes the headline. And they sometimes hate the headline as much as the readers do, but that's how it goes in that industry, I suppose.
Yep. I wouldn't surprised to see this article come from some other outlet; I did not think it's already time to add Wired to the "yellow media" bucket (though last week's article about LLVM being the last thread holding Google and Apple together just about did it)
Wired really depends on the author, IMO. I still enjoy Threat Level
Discussion from 2 days ago: https://news.ycombinator.com/item?id=6110398
These slides from Brad Fitzpatrick himself on the subject were a lot more informative:
http://talks.golang.org/2013/oscon-dl.slide
http://talks.golang.org/2013/oscon-dl.slide
Thanks. This form of internet journalism really irritates me, trying desperately to create a compelling narrative out of fairly dry technical parts, only to end abruptly to some upbeat testimonial - it reads more like a poorly written press release.
If code doesn’t receive constant love it turns to shit,
I would prefer "If bad code doesn't receive constant love it turns to shit"
I am aware of thousands of examples of heavily used commercial software that hasn't been touched in 5, 10, 15, even 20 years because it just works and always has. Properly designed and written scalable software can last indefinitely with little or no modification, even through geometric changes.
But I like OP's quote. I think it should become part of every code reviewer's checklist:
I would prefer "If bad code doesn't receive constant love it turns to shit"
I am aware of thousands of examples of heavily used commercial software that hasn't been touched in 5, 10, 15, even 20 years because it just works and always has. Properly designed and written scalable software can last indefinitely with little or no modification, even through geometric changes.
But I like OP's quote. I think it should become part of every code reviewer's checklist:
Will this turn to shit without constant love?
No: Pass.
Yes: Then fix it now.It doesn't work that way. The underlying assumptions change. The program can be perfect but still not matching what becomes necessary in a few years. The example they write about is exactly such a kind. Sometime in 2007 it was apparently slow to access files from the distributed storage on Google. Therefore, a Python script was written to fetch these files to some local disks. Python was obviously enough since that process was slow. Then, another program, written in C++ was to deliver the files from the local disks to the clients. The C++ program was then "sub-optimally" maintained, simply because it wasn't on the critical path for the company. There weren't even the good test cases to verify correctness of the changes.
Still, until 2013 a lot of assumptions based on which the program was written changed completely: Accessing files from the distributed storage became faster. The typical computer which is to deliver the data seems to have much more free RAM, allowing using RAM for caching purposes instead of the local hard disk. The nature of the load changed so much that accessing the local hard disk became the bottleneck anyway. Which is not so surprising: one disk seek is still around 10 ms, which means as soon as you can't deliver something from RAM you can deliver only up to 100 items from the disk per second, when they are not in the same block of data.
Don't tell me you have to write the program for "all possible assumptions." Unless you have unlimited time, you write it to solve the problem under existing assumptions. Once they change, you change the implementation.
Moreover, to achieve the changes related to the underlying assumptions, Go wasn't needed in the current example, but it was the most convenient approach for Brad, who is by the way obviously an extraordinary programmer.
What I agree with you is that if the program is bad even under the initial assumptions, it's certainly something that should be observed and tracked by the good manager: he must know that he can't rely on such program and that whenever the program gets any heavier load, the problems will happen.
Still, until 2013 a lot of assumptions based on which the program was written changed completely: Accessing files from the distributed storage became faster. The typical computer which is to deliver the data seems to have much more free RAM, allowing using RAM for caching purposes instead of the local hard disk. The nature of the load changed so much that accessing the local hard disk became the bottleneck anyway. Which is not so surprising: one disk seek is still around 10 ms, which means as soon as you can't deliver something from RAM you can deliver only up to 100 items from the disk per second, when they are not in the same block of data.
Don't tell me you have to write the program for "all possible assumptions." Unless you have unlimited time, you write it to solve the problem under existing assumptions. Once they change, you change the implementation.
Moreover, to achieve the changes related to the underlying assumptions, Go wasn't needed in the current example, but it was the most convenient approach for Brad, who is by the way obviously an extraordinary programmer.
What I agree with you is that if the program is bad even under the initial assumptions, it's certainly something that should be observed and tracked by the good manager: he must know that he can't rely on such program and that whenever the program gets any heavier load, the problems will happen.
Not all software operates at scale. The backwards compatibility fostered by posix and wintel has meant many such applications have been silently doing their jobs for many years without (m)any changes.
Yes, when the underlying assumptions don't change and the program is already good it will remain good. The point is that the changes of the underlying assumptions can make even the once perfect program one that's suddenly unusable and it's impossible to cover all future possibilities. Targeting realistically is the only way to make the good program at the start.
Properly designed and written scalable software can last indefinitely with little or no modification
The problem is that what may be "proper" today (thinking, say, 5-10 years out) may fail the test in 10-15 years. One has to design software under some constraints, usually defined by the problem boundary and thus it by definition cannot be infinitely scalable and flexible.
Also, the problem is the code ends up being maintained by programmers of varying levels of competency. Even if competency is not a problem, the varying styles inadvertently lead to spaghetti and inconsistent code.
The problem is that what may be "proper" today (thinking, say, 5-10 years out) may fail the test in 10-15 years. One has to design software under some constraints, usually defined by the problem boundary and thus it by definition cannot be infinitely scalable and flexible.
Also, the problem is the code ends up being maintained by programmers of varying levels of competency. Even if competency is not a problem, the varying styles inadvertently lead to spaghetti and inconsistent code.
Properly designed and written scalable software can last indefinitely with little or no modification, even through geometric changes.
Daniel Bricklin wrote an interesting article[1] related to long lasting software (PDF)
[1] http://changethis.com/manifesto/download/6.200YearSoftware
Daniel Bricklin wrote an interesting article[1] related to long lasting software (PDF)
[1] http://changethis.com/manifesto/download/6.200YearSoftware
The analogy between bridge building and software is very old and very wrong.
Software can indeed be designed like a bridge, proven and then built for one unchanging use case and set in stone. And unless you are writing for NASA, soon enough that code will be junked as an obsolete millstone because use cases change. No engineer could design a bridge that might have to be rebuilt into an aircraft runway, or a 20-floor apartment complex, at a whim.
Software can indeed be designed like a bridge, proven and then built for one unchanging use case and set in stone. And unless you are writing for NASA, soon enough that code will be junked as an obsolete millstone because use cases change. No engineer could design a bridge that might have to be rebuilt into an aircraft runway, or a 20-floor apartment complex, at a whim.
> The analogy between bridge building and software is very old and very wrong.
I disagree. I think that the assumptions and use cases just change much more quickly in software than in the physical world.
The covered bridges from 150 years ago were built with the assumption that they'd be used for horses, carts and people. When cars became the primary mode of transportation then those assumptions no longer held. No one would say that the original bridge was poorly built, just that the use case changed.
Similarly, software that was built with the assumption that network access is slow has had its typical use case change. Software that was built to keep things on disk because RAM is expensive needed to change. Pretty soon we'll all demand that software assume that disk access is fast again because we'll move to SSDs.
I disagree. I think that the assumptions and use cases just change much more quickly in software than in the physical world.
The covered bridges from 150 years ago were built with the assumption that they'd be used for horses, carts and people. When cars became the primary mode of transportation then those assumptions no longer held. No one would say that the original bridge was poorly built, just that the use case changed.
Similarly, software that was built with the assumption that network access is slow has had its typical use case change. Software that was built to keep things on disk because RAM is expensive needed to change. Pretty soon we'll all demand that software assume that disk access is fast again because we'll move to SSDs.
I would argue that bad code is already shit. There's no doubt that good code can be turned to shit too if it's being modified all the time with bug fixes, etc, without paying the required technical debts.
Not sure what your example of untouched code represents. The OP says that what makes code turn bad is accumulated local changes and bug fixes without thinking about the big picture. Of course if no one ever touches the code, its quality doesn't change.
Not sure what your example of untouched code represents. The OP says that what makes code turn bad is accumulated local changes and bug fixes without thinking about the big picture. Of course if no one ever touches the code, its quality doesn't change.
Or just deploy the half-assed solution, which will get the job done and allow your limited engineering staff to focus on higher priorities. Then, a decade later, when the company is worth hundreds of billions, you can rewrite it from scratch the proper way.
See also: A Tale of Two Bridges http://hintjens.com/blog:16
See also: A Tale of Two Bridges http://hintjens.com/blog:16
It's a good sentiment, but it doesn't usually work that way.
A decade later you find that you don't just have one half-assed solution to rewrite, you have a decade's worth. And because you're spending every second of your working life maintaining that teetering pile of kludges the idea of setting aside enough time to do a proper rewrite is sheer fantasy.
Certainly there is a good case to be made for accepting technical debt as a necessity in order to get products into the market and revenue coming out of the market, but on the other hand the typical software development organization tends to be drowning in technical debt. Pay down your technical debt regularly and quickly, not after a decade. For every example of the half-assed temporary solution that made the company money there is a story of development hampered by the burden of maintaining more technical debt than is wise.
A decade later you find that you don't just have one half-assed solution to rewrite, you have a decade's worth. And because you're spending every second of your working life maintaining that teetering pile of kludges the idea of setting aside enough time to do a proper rewrite is sheer fantasy.
Certainly there is a good case to be made for accepting technical debt as a necessity in order to get products into the market and revenue coming out of the market, but on the other hand the typical software development organization tends to be drowning in technical debt. Pay down your technical debt regularly and quickly, not after a decade. For every example of the half-assed temporary solution that made the company money there is a story of development hampered by the burden of maintaining more technical debt than is wise.
> A decade later you find that you don't just have one half-assed solution to rewrite, you have a decade's worth.
Whereas if you don't have a good sense of priorities, and you don't get your product out the door, a decade later your company's been dead for nine years.
Whereas if you don't have a good sense of priorities, and you don't get your product out the door, a decade later your company's been dead for nine years.
Of course, but these are ends on the spectrum, it's important to find the happy medium.
One of the biggest downsides of taking on a lot of technical debt is that the cost tends to be indirect and diffuse. And that's the sort of thing that organizations find it easiest to ignore. It's only when the pain becomes acute and obvious that it tends to be addressed, but by then the cost of addressing it is often several orders of magnitude higher than it would have been if it was addressed at a more appropriate time. This can cause a severe stoppage of development velocity and has doomed a sizable number of major projects and products over the years.
One of the biggest downsides of taking on a lot of technical debt is that the cost tends to be indirect and diffuse. And that's the sort of thing that organizations find it easiest to ignore. It's only when the pain becomes acute and obvious that it tends to be addressed, but by then the cost of addressing it is often several orders of magnitude higher than it would have been if it was addressed at a more appropriate time. This can cause a severe stoppage of development velocity and has doomed a sizable number of major projects and products over the years.
What causes code to turn to shit is taking a well designed, well-factored solution, and hanging a patch off the side to make it do something new, or to fix a use case it wasn't designed to handle.
The "constant love" needed is that after a change you must redesign or at least refactor. Preferably every time, or you build up a debt of work.
The "constant love" needed is that after a change you must redesign or at least refactor. Preferably every time, or you build up a debt of work.
Have any heavy C++ devs switched to go? What's the major advantage/compelling reason to switch? It seems to me that Go is more comparable to Python/Ruby and Java on the Web, but maybe I'm wrong.
* No circular dependencies (big win here)
* Package importing is easy and a first-class language structure
* Fast compiles (related to no circular dependencies)
One of the more controversial features is no inheritance -- only composition. It's an interesting break from vertical/horizontal inheritance schemes.
* Package importing is easy and a first-class language structure
* Fast compiles (related to no circular dependencies)
One of the more controversial features is no inheritance -- only composition. It's an interesting break from vertical/horizontal inheritance schemes.
I'm not a heavy C++ developer and only use it to augment other languages when performance demands it, but not having to deal with memory management and the problems it can create when binding to another language is a nice benefit. The only drawback with Go though for my usage is when using on Android, since it has to be compiled statically when used with the NDK.
I really don't understand why dl.google.com isn't just running Nginx or some web server. It's just serving files. Why does it need software written in house?
Nginx was built with 'single server' architecture in mind while Google generally operates on a 'distributed everything' architecture. This involves stuff like distributed file systems/data stores. My guess is it's not compatible with the single file system interfaces of Nginx. Also note that dl.google.com serves stuff on a massively bigger scale than the majority of the rest of the internet. On that scale I'd guess they face some rather uncommon challenges and bottlenecks that can only be addressed by custom software.
True, but a static-file server should be trivially scalable. That is, scalable by placing it behind a load balancer(or using any number of alternative methods that don't acutally touch the server itself).
Even if the server implements Access Control, it should still be trivially scalable.
Also, nginx doesn't have such...interesting ideas as the custom server: http://talks.golang.org/2013/oscon-dl.slide#19
I mean, if by "rather uncommon challenges" you're referring to the "what are threads?" design philosophy of the original... then yeah, it does require "custom software".
Also, nginx doesn't have such...interesting ideas as the custom server: http://talks.golang.org/2013/oscon-dl.slide#19
I mean, if by "rather uncommon challenges" you're referring to the "what are threads?" design philosophy of the original... then yeah, it does require "custom software".
dl.google.com isn't just a place to dump files to the public web, it's also got complicated ACL schemes that define restrictions on data access. dl.google.com is used internally.
Google is the only company on the planet with anything approaching its scale requirements. It's entirely possible that even nginx breaks down at their level of speed/concurrency/distribution.
True, but it isn't the only company serving apt packages, and it isn't even the biggest one. Considering that their apt server was as slow as it was(http://talks.golang.org/2013/oscon-dl.slide#9) - they obviously did something wrong that everyone else was doing right.
Perhaps the problem was that dl.google.com was being used for too many things at once, but this doesn't excuse building custom software that offers worse performance than free off-the-shelf products that literally everyone else has been using for years.
Perhaps the problem was that dl.google.com was being used for too many things at once, but this doesn't excuse building custom software that offers worse performance than free off-the-shelf products that literally everyone else has been using for years.
Maybe they should take a look at feedburner.
I hope all journalism majors are sitting up and reading this carefully. This is how you make a nice imaginative story out of a small interview. A good story/link bait needs something in desperate need ('lousy code'). Then it needs a hero ('gopher team'). Then it needs a title which will catch people's attention (people can't help reading about 'google'). Creative journalism at it's best.
In case it isn't clear, the only store here is that one guy wrote Google's download server in Go because it was not working well. That's it, that's the only story. Google doesn't send some task force or anything like that.
In case it isn't clear, the only store here is that one guy wrote Google's download server in Go because it was not working well. That's it, that's the only story. Google doesn't send some task force or anything like that.
They should really dispatched to fix the Google Drive Client program for Mac. It consistently maxes out a CPU even when there is no sync going on. I can consistently reproduce this issue on all Macs I own.
"The team was able to make many improvements to the way the language handles clustering and file transfers."
Either the journalist had no business writing this sentence, or the people who designed Go put things in the language that should have been in the libraries. Or both.
Either the journalist had no business writing this sentence, or the people who designed Go put things in the language that should have been in the libraries. Or both.
You can see this in the response to "we can't serve files correctly" in that there was to rush in write new code in a Google language (Go), as if there weren't thousands of existing ways to solve this with an open source component already.