Stone knives and bear skins – there is no money in tools(queue.acm.org)
queue.acm.org
Stone knives and bear skins – there is no money in tools
https://queue.acm.org/detail.cfm?id=3606027
15 comments
Well said. I know there's money in tools because I've been making money making and selling tools for a long time.
But not every kind of tool. A gdb replacement would be a hard sell unless it managed to actually perform miracles. But specialty tools of the sort that can save a relatively small group of devs major pain and/or time? There's good money in that for a small business. That the market is too small to be attractive to large companies is just gravy.
But not every kind of tool. A gdb replacement would be a hard sell unless it managed to actually perform miracles. But specialty tools of the sort that can save a relatively small group of devs major pain and/or time? There's good money in that for a small business. That the market is too small to be attractive to large companies is just gravy.
Time-travel seems miraculous to me, but uptake seems very slow to me.
I hope https://www.replay.io/ succeeds.
I hope https://www.replay.io/ succeeds.
Commercially available time travel debugging is over 20 years old at this point [1]. So yeah, the uptake is pretty slow.
[1] http://jakob.engbloms.se/archives/1564
[1] http://jakob.engbloms.se/archives/1564
>Well said. I know there's money in tools because I've been making money making and selling tools for a long time
Couldn't a succesful actor say the same? "I know there's money in acting, because I've been making money acting for a long time".
And yet we'd instantly understand that doesn't refute the sense the statement was meant, and that acting isn't really the field you wanna pursue if you want to make money".
Same for any other niche endeavor, where there are thousands more lucrative fields and niches, and much fewer succesful examples compared to other areas.
Couldn't a succesful actor say the same? "I know there's money in acting, because I've been making money acting for a long time".
And yet we'd instantly understand that doesn't refute the sense the statement was meant, and that acting isn't really the field you wanna pursue if you want to make money".
Same for any other niche endeavor, where there are thousands more lucrative fields and niches, and much fewer succesful examples compared to other areas.
> acting isn't really the field you wanna pursue if you want to make money
That depends on the person. By the same token, software engineering isn’t the right field for the average person (because most people don’t have the knack for coding), and yet some people find they do have that talent.
A career is like dating, it’s mostly about finding a good fit, which means you also need to know yourself. Once you do, you may find a ‘bad’ niche is actually the right one for you.
That depends on the person. By the same token, software engineering isn’t the right field for the average person (because most people don’t have the knack for coding), and yet some people find they do have that talent.
A career is like dating, it’s mostly about finding a good fit, which means you also need to know yourself. Once you do, you may find a ‘bad’ niche is actually the right one for you.
>That depends on the person
Yes, that's the fallacy/issue I was addressing.
It's a market, and we try to evaluate it as a market, and talk about its characteristics.
Not about whether "if you're the next Brad Pitt you'd make millions".
Yes, that's the fallacy/issue I was addressing.
It's a market, and we try to evaluate it as a market, and talk about its characteristics.
Not about whether "if you're the next Brad Pitt you'd make millions".
I don't understand what fallacy you were addressing, honestly. I was not asserting that making tools is a large market that can support a larger number of players. In fact, I think I acknowledged that it was not. But I don't understand the relevance of that to my original point.
>I don't understand what fallacy you were addressing, honestly
That a general statement about the statistical properties of something (in this case, the IT tooling market) is refuted by pointing out individual counter-examples. Those can only refuse absolute statements.
Though I'm afraid this is not even a logical error. Instead it's a reading comprehension error, people read "there's no money in tooling", and think the author means "nobody made or makes money from selling tooling".
The latter (absolute statement) is not what TFA means, nobody claimed is the case, and is totally irrelevant to what's discussed. As are counter-examples to it (everybody already knows there are counter-examples).
That a general statement about the statistical properties of something (in this case, the IT tooling market) is refuted by pointing out individual counter-examples. Those can only refuse absolute statements.
Though I'm afraid this is not even a logical error. Instead it's a reading comprehension error, people read "there's no money in tooling", and think the author means "nobody made or makes money from selling tooling".
The latter (absolute statement) is not what TFA means, nobody claimed is the case, and is totally irrelevant to what's discussed. As are counter-examples to it (everybody already knows there are counter-examples).
No, it depends on supply and demand. Lots of people want to act. Lots of people want to make tools.
And some are successful.
Which is irrelevant when evaluating a job market.
Some are succesful in any crap market, where the chances are that most will go under or utterly fail, and where the conditions and average salaries and profit margins and such are attrocious.
"Some are succesful" as some kind of retort basically translate to "I don't care about market conditions, I have confidence in myself and I know I'll be one of the succesful ones".
Which is all well and good, but it doesn't say anything about one's chances of success, or that job market's behavior in general. Besides, historically it's what every fresh-eyed kid fresh off the bus from Nebraska, still waiting tables 20 years later, believed too.
Some are succesful in any crap market, where the chances are that most will go under or utterly fail, and where the conditions and average salaries and profit margins and such are attrocious.
"Some are succesful" as some kind of retort basically translate to "I don't care about market conditions, I have confidence in myself and I know I'll be one of the succesful ones".
Which is all well and good, but it doesn't say anything about one's chances of success, or that job market's behavior in general. Besides, historically it's what every fresh-eyed kid fresh off the bus from Nebraska, still waiting tables 20 years later, believed too.
What I reject is evaluating the a job maket and completely ignoring the market participant, treating them as interchanagable.
Sure, many people suffer from self delusion, but that doesnt mean that their individual characteristics are meaningless.
If Jeff Bezos wanted to make tools, with all the power and money he has, he would have a much better chance than the average market participant.
Sure, many people suffer from self delusion, but that doesnt mean that their individual characteristics are meaningless.
If Jeff Bezos wanted to make tools, with all the power and money he has, he would have a much better chance than the average market participant.
> Couldn't a succesful actor say the same? "I know there's money in acting, because I've been making money acting for a long time".
Yes. That even one person makes money doing a thing is strong evidence that it's possible to make money doing that thing. Whether or not doing that thing is something you should do is an entirely different question.
Yes. That even one person makes money doing a thing is strong evidence that it's possible to make money doing that thing. Whether or not doing that thing is something you should do is an entirely different question.
>Yes. That even one person makes money doing a thing is strong evidence that it's possible to make money doing that thing.
Which would be relevant if the question at stake was whether it's "possible or not".
As opposed to the question whether people should focus on that market as a good market, where relatively more opportunities and money are than other IT areas.
Or did you thought those writing the post didn't already know several examples of companies making money selling tooling, and they didn't change their point, because the possibility of it was never what they were talking about?
Which would be relevant if the question at stake was whether it's "possible or not".
As opposed to the question whether people should focus on that market as a good market, where relatively more opportunities and money are than other IT areas.
Or did you thought those writing the post didn't already know several examples of companies making money selling tooling, and they didn't change their point, because the possibility of it was never what they were talking about?
IDA Pro competes with gdb and Ghidra and seems profitable enough:
https://hex-rays.com/IDA-pro/
Then you have the people who sold shovels and pickaxes to prospectors during the Gold Rush
https://hex-rays.com/IDA-pro/
Then you have the people who sold shovels and pickaxes to prospectors during the Gold Rush
What kind of tools do you build? I'd be interested in your opinion on what are the preconditions for tools to profitable.
All a gdb replacement has to do to succeed is not require the binary to be codesigned.
ASML's crown jewels are not mere "tools", they are machines. It is bleeding edge technology and entirely irrelevant to a general discussion of economics of 'tool makers'.
https://www.youtube.com/watch?v=5Ge2RcvDlgw
https://www.youtube.com/watch?v=5Ge2RcvDlgw
In the industry they are very literally called "tools". No one calls them "machines". E.g., "the fab is ordering another six EUV tools".
Similarly a lathe or vertical mill is "a machine tool". They're ideally suited for a very specific task in the process of making a thing. It's possible for something to be both a machine and a tool.
If we're being pedantic, the ASML tools (and indeed any modern wafer fabrication equipment) are also robots.
More related to the business of tool making broadly, however, the entire industry of semiconductor wafer fabrication equipment (WFE) is quite profitable and margins tend to be generous. Much of the industry is very technical and does involve "bleeding edge technology", but not all of it. The publicly traded Japanese company Disco makes tools for wafer grinding[1], which are not really any more complex than your typical computer-controlled metal cutting machine tools, yet they are quite profitable[2]. Just look at those headline margin figures!
It seems to be a matter of niche and competition.
[1]: https://www.disco.co.jp/eg/products/grinder/dfg8560.html
[2]: https://www.disco.co.jp/eg/ir/library/doc/film/20230420.pdf
Similarly a lathe or vertical mill is "a machine tool". They're ideally suited for a very specific task in the process of making a thing. It's possible for something to be both a machine and a tool.
If we're being pedantic, the ASML tools (and indeed any modern wafer fabrication equipment) are also robots.
More related to the business of tool making broadly, however, the entire industry of semiconductor wafer fabrication equipment (WFE) is quite profitable and margins tend to be generous. Much of the industry is very technical and does involve "bleeding edge technology", but not all of it. The publicly traded Japanese company Disco makes tools for wafer grinding[1], which are not really any more complex than your typical computer-controlled metal cutting machine tools, yet they are quite profitable[2]. Just look at those headline margin figures!
It seems to be a matter of niche and competition.
[1]: https://www.disco.co.jp/eg/products/grinder/dfg8560.html
[2]: https://www.disco.co.jp/eg/ir/library/doc/film/20230420.pdf
>There no money in tools: The Dutch company ASML would like to talk to you.
"You made a general argument. Here's a single counter-example, on a very tiny niche, compared to thousands of examples in your favor, making much more money in other areas of IT".
"You made a general argument. Here's a single counter-example, on a very tiny niche, compared to thousands of examples in your favor, making much more money in other areas of IT".
The article specifically reaches into the whole "spanners are too cheap to be worth making" space. The article didn't say "except of course, those gold plated NASA spanners" or "ok: so this is a broad brush but there are notable exceptions"
I called out an exception. I like the article. I think the point is well made. I would say it a bit differently: commodities are very hard to make profitably once the exemplar is good enough, and cheap enough. If you seek higher utility then don't be surprised if the people who use your Tool/Machine/Robot balk at the price, when their goal is met by the lower utility commodity.
If you don't care about lead and tolerance, those fake resistors are just fine for hobbyist use. If your project needs lead free and 0.001% then you really need to pay the premium for originals, not fakes.
And ASML is a niche, but an example of a generic problem in their line of reasoning. For almost any "industry" you care to mention from coathanger bending to lobster fishing, there is somebody making good profit selling what the coathanger-wrangler, or fisherman would call "a tool"
ie: cute argument, but actually... sorry.. niches are everywhere.
I called out an exception. I like the article. I think the point is well made. I would say it a bit differently: commodities are very hard to make profitably once the exemplar is good enough, and cheap enough. If you seek higher utility then don't be surprised if the people who use your Tool/Machine/Robot balk at the price, when their goal is met by the lower utility commodity.
If you don't care about lead and tolerance, those fake resistors are just fine for hobbyist use. If your project needs lead free and 0.001% then you really need to pay the premium for originals, not fakes.
And ASML is a niche, but an example of a generic problem in their line of reasoning. For almost any "industry" you care to mention from coathanger bending to lobster fishing, there is somebody making good profit selling what the coathanger-wrangler, or fisherman would call "a tool"
ie: cute argument, but actually... sorry.. niches are everywhere.
The broad brush market is alive and well, though.
Here's another multibillion dollar example https://en.wikipedia.org/wiki/Lam_Research
I mean, you're using a computer to shitpost here. Everything in that computer, let alone the computer as a whole, is a tool that made its manufacturer and any vendors involved lots of money.
Yeah, I also wear clothes (kids go into garments/fashion), drink coffee (kids start coffee shops), and buy many other items besides. By this logic every market is a lucrative one, if people buy stuff from it. So, just go and build computers, companies make money selling them. Start the new Apple or Dell.
It's almost as if this missed the point of TFA entirely, and as if "things are sold in that field and there are some player making lots of money in it" is useless by itself for evaluating a field/niche regarding to its money making opportunities.
Let me introduce you to a wonderful concept: https://en.wikipedia.org/wiki/Opportunity_cost
Not bad for shitposting eh?
It's almost as if this missed the point of TFA entirely, and as if "things are sold in that field and there are some player making lots of money in it" is useless by itself for evaluating a field/niche regarding to its money making opportunities.
Let me introduce you to a wonderful concept: https://en.wikipedia.org/wiki/Opportunity_cost
Not bad for shitposting eh?
If your tool is an integral part of a larger chain, as is the case for ASML's machines being in the chain of silicon chips, that's a different story than a tool which simply eases the process by a small factor.
UV lithography is the only way to make advanced chips required for various other tools and appliances, but an OS debugger (as the author puts it) is just a nicer printf.
There's big money in one, and small money in the other.
UV lithography is the only way to make advanced chips required for various other tools and appliances, but an OS debugger (as the author puts it) is just a nicer printf.
There's big money in one, and small money in the other.
In software tools. Hardware tools are hardware.
There are lots of profitable software tool companies, aren't there? Atlassian, JetBrains, Microsoft, most of them are smaller than this but still "money"; I live in open-source land, but everyone else I know works somewhere that spends money on all kinds of tools.
It's just that the fruit in software tools doesn't hang so low, I think.
It's just that the fruit in software tools doesn't hang so low, I think.
I think there are a few things at work.
There is no money in tools because employers (and to some extent employees) don't value them. Also, folks who make tools maybe value them too much? And maybe tools are too easy to copy?
I remember decades ago when purify came out. I tried the demo at work on a project I was working on. Within a few hours, I found an enormous number memory errors. It was enormously helpful. Purify had a high per-seat license cost, and work wouldn't buy it.
Many current employers are a little smarter about things, but I mainly see things like code coverage tools and version control systems, but no real spending on more personal/fashionable tools, even if employees are more efficient.
I think tools that might work better would be those employees bought them themselves - think jetbrains - and priced accordingly. The analogy might be auto mechanics building up their collection from snap-on.
also, warts and all, linux is a nice environment because you can get to the bottom of most things and tools might be less industrial quality, but there are plenty of them.
There is no money in tools because employers (and to some extent employees) don't value them. Also, folks who make tools maybe value them too much? And maybe tools are too easy to copy?
I remember decades ago when purify came out. I tried the demo at work on a project I was working on. Within a few hours, I found an enormous number memory errors. It was enormously helpful. Purify had a high per-seat license cost, and work wouldn't buy it.
Many current employers are a little smarter about things, but I mainly see things like code coverage tools and version control systems, but no real spending on more personal/fashionable tools, even if employees are more efficient.
I think tools that might work better would be those employees bought them themselves - think jetbrains - and priced accordingly. The analogy might be auto mechanics building up their collection from snap-on.
also, warts and all, linux is a nice environment because you can get to the bottom of most things and tools might be less industrial quality, but there are plenty of them.
Employers do not value tools because the employees say the cost-benefit is not worth it. The problem is that the employees do the cost-benefit calculations remembering their time as a poor student when their time was worthless.
When your time is worth $0, a product must be 10x better to be worth paying 10x as much. When your time costs $200,000/yr, a 1% improvement is trivially worth $2,000/yr.
People in software do not internalize this because we can build our own tools. We can solve the problem given time unlike say a chemist who can not build a reaction vessel. When we were poor, we built our own tools and we keep that habit even after we have money and our time is valuable.
It is absolutely ridiculous; the average janitor spends more on tools than a software developer. Truck drivers, secretarys, artists, mechanics, all spend more on tools even though their time is significantly less costly on average. In every other engineering profession companys literally spend $30,000, $100,000, and more a year keeping their high value workers productive.
Only in software do we think that $3,000/yr for Visual Studio, the tool that you literally use the entire time when doing Windows development, or $99 per year to do iOS development are “steep” costs. $99 is like one hour of time; it is ludicrously cheap.
$25,000/yr for a 25% productivity improvement is a no-brainer. The problem is that software people think it is obviously wrong, but the business people think it is obviously right. We do not need to struggle with hand trowels, the smart business people want to give us excavators, we just need to stop telling them no.
When your time is worth $0, a product must be 10x better to be worth paying 10x as much. When your time costs $200,000/yr, a 1% improvement is trivially worth $2,000/yr.
People in software do not internalize this because we can build our own tools. We can solve the problem given time unlike say a chemist who can not build a reaction vessel. When we were poor, we built our own tools and we keep that habit even after we have money and our time is valuable.
It is absolutely ridiculous; the average janitor spends more on tools than a software developer. Truck drivers, secretarys, artists, mechanics, all spend more on tools even though their time is significantly less costly on average. In every other engineering profession companys literally spend $30,000, $100,000, and more a year keeping their high value workers productive.
Only in software do we think that $3,000/yr for Visual Studio, the tool that you literally use the entire time when doing Windows development, or $99 per year to do iOS development are “steep” costs. $99 is like one hour of time; it is ludicrously cheap.
$25,000/yr for a 25% productivity improvement is a no-brainer. The problem is that software people think it is obviously wrong, but the business people think it is obviously right. We do not need to struggle with hand trowels, the smart business people want to give us excavators, we just need to stop telling them no.
My software hand trowels don't get randomly totally wrecked by some fresh grad UX guru who decided that they needed to rework everything to justify their employment or because they felt they urgently need to chase the latest design fads, because the old ones are so last year.
Most tools in those other professions don't have that problem. Once you buy them, they're generally static.
But yeah, your point about devs being hilariously stingy and not valuing their time is pretty spot on. I just don't think that's the main reason they don't want to buy commercial tooling.
Most tools in those other professions don't have that problem. Once you buy them, they're generally static.
But yeah, your point about devs being hilariously stingy and not valuing their time is pretty spot on. I just don't think that's the main reason they don't want to buy commercial tooling.
Not sure what it is like for other professions but in software development I've found getting approval for buying tools to be a cumbersome process. The most ridiculous case was a $10 license that took several months to get approved and involved multiple 'C' level executives at a company with about 30k employees. At the same company, I was denied access to a cloud service that the company already had a subscription to because my process would use $0.03 of compute time per month, which we were quite willing to pay. There were no security concerns or anything else, just two mid-level managers trying to extract resources out of each other.
Those are extremes but even simple things like getting a $100 plugin to parse a particular data format ends up being a time-wasting process that far exceeds the cost of the plugin, and if drawn out enough, the savings it would have produced over building it ourselves. As a result, building it ourselves tends to end up being the default choice.
Those are extremes but even simple things like getting a $100 plugin to parse a particular data format ends up being a time-wasting process that far exceeds the cost of the plugin, and if drawn out enough, the savings it would have produced over building it ourselves. As a result, building it ourselves tends to end up being the default choice.
That is because the budget for you or your team has a big fat $0 for tooling. It is a shameful and degrading state of affairs.
Developer time is so expensive it is insane to spend more than a few minutes thinking about a $100 purchase. Your team should have thousands of dollars of open budget per member per year for tooling or critical purchases. Obviously it needs to be rectified regularly to evaluate if the purchases are actually worthwhile, but needing complex process for trivial expenditures like $1,000 is insane.
The funniest thing is that operations, cloud deployment, and data science teams do get serious tooling budget. Even though the fields are relatively adjacent and usually include some degree of software development, the fact that they are less capable of building their own tools means they get orders of magnitude more tooling spend.
Everybody knows that developer time is too valuable to spend when we can just throw more machines and tools at the problem. But in actual software development, we just forget that and go right back to wasting valuable developer time by cheaping out on tools. It is amazing.
It is about time we get some respect for our time and demand tooling budgets.
Developer time is so expensive it is insane to spend more than a few minutes thinking about a $100 purchase. Your team should have thousands of dollars of open budget per member per year for tooling or critical purchases. Obviously it needs to be rectified regularly to evaluate if the purchases are actually worthwhile, but needing complex process for trivial expenditures like $1,000 is insane.
The funniest thing is that operations, cloud deployment, and data science teams do get serious tooling budget. Even though the fields are relatively adjacent and usually include some degree of software development, the fact that they are less capable of building their own tools means they get orders of magnitude more tooling spend.
Everybody knows that developer time is too valuable to spend when we can just throw more machines and tools at the problem. But in actual software development, we just forget that and go right back to wasting valuable developer time by cheaping out on tools. It is amazing.
It is about time we get some respect for our time and demand tooling budgets.
>When we were poor, we built our own tools and we keep that habit even after we have money and our time is valuable.
This is a lesson in life that most people really only realize once they start getting old and see more and more people around them die.
Time is money, and as we get older we (hopefully) have more money but we only lose and lose whatever time we have left. Money is theoretically infinite, time is finite. You can borrow money, but you can't borrow time.
At a certain point in life, paying money to save your time (aka use someone else's time) is worth it no matter how expensive it is.
This is a lesson in life that most people really only realize once they start getting old and see more and more people around them die.
Time is money, and as we get older we (hopefully) have more money but we only lose and lose whatever time we have left. Money is theoretically infinite, time is finite. You can borrow money, but you can't borrow time.
At a certain point in life, paying money to save your time (aka use someone else's time) is worth it no matter how expensive it is.
I don't even think cost benefit is the driving factor. I think making tools is why a lot of devs maintain their interest.
For any given task you'll find 5 commercial solutions that basically make it 1-click, if you're lucky a few FOSS equivalents, and 5000 abandoned github repos that implement one or two specific features in the commercial ones.
Devs(In general, not myself) don't want debuggers, they want printf. They might want Linux, but they probably like their own DIY hobby OS more.
For any given task you'll find 5 commercial solutions that basically make it 1-click, if you're lucky a few FOSS equivalents, and 5000 abandoned github repos that implement one or two specific features in the commercial ones.
Devs(In general, not myself) don't want debuggers, they want printf. They might want Linux, but they probably like their own DIY hobby OS more.
> Employers do not value tools because the employees say the cost-benefit is not worth it. The problem is that the employees do the cost-benefit calculations remembering their time as a poor student when their time was worthless.
> People in software do not internalize this because we can build our own tools.
I think there's another aspect. Those tools are doing things we can do anyway. Here's a story of something I did recently:
I wanted to download a series from viki.com. They have switched from just sending you the video file to sending you an encrypted video file along with the encryption key. This is "more secure".
So, there's a process involving detecting your own request for the encryption key so you can replay it and get the response, decrypting the video file, decrypting the audio file, and then muxing them together with the unencrypted subtitle file using ffmpeg. After figuring this all out (which took a significant chunk of time), I could manually run the process in 5 minutes for one video.
I wrote a powershell script to automate all the parts of it that weren't finding the URL of the three media streams, detecting the request for the encryption key or echoing the request and getting the key. Those stayed manual. But the script handled downloading, decryption, and assembly and it cut my time per video from 5 minutes to 2 minutes. That's a 60% decrease! But it's also easy to say "hey, 5 minutes was already pretty fast; what do I care about 3 minutes?". The payoff from automation only looks good if you're doing a lot of it.
(The automation also made it obviously worthwhile to go nuts on adding metadata to the final video file, where that would have been more annoying in the all-manual process.)
> People in software do not internalize this because we can build our own tools.
I think there's another aspect. Those tools are doing things we can do anyway. Here's a story of something I did recently:
I wanted to download a series from viki.com. They have switched from just sending you the video file to sending you an encrypted video file along with the encryption key. This is "more secure".
So, there's a process involving detecting your own request for the encryption key so you can replay it and get the response, decrypting the video file, decrypting the audio file, and then muxing them together with the unencrypted subtitle file using ffmpeg. After figuring this all out (which took a significant chunk of time), I could manually run the process in 5 minutes for one video.
I wrote a powershell script to automate all the parts of it that weren't finding the URL of the three media streams, detecting the request for the encryption key or echoing the request and getting the key. Those stayed manual. But the script handled downloading, decryption, and assembly and it cut my time per video from 5 minutes to 2 minutes. That's a 60% decrease! But it's also easy to say "hey, 5 minutes was already pretty fast; what do I care about 3 minutes?". The payoff from automation only looks good if you're doing a lot of it.
(The automation also made it obviously worthwhile to go nuts on adding metadata to the final video file, where that would have been more annoying in the all-manual process.)
Purify was glorious. As was Saber-C before it.
My employer paid for both. But we sold even more expensive development software ourselves, and bought very expensive workstation and server hardware to run it on, so we were accustomed to the price points.
I like that great software is now open source, and most any kid or adult around the world can run it on a cheap decade-old PC. But I do miss some things about that software development era.
My employer paid for both. But we sold even more expensive development software ourselves, and bought very expensive workstation and server hardware to run it on, so we were accustomed to the price points.
I like that great software is now open source, and most any kid or adult around the world can run it on a cheap decade-old PC. But I do miss some things about that software development era.
The barriers are things like integration effort, carrying cost (keeping up with updates and the toil to keep the tool going), poor signal / noise ratio and subsequent triage and debugging effort, additional feedback to dev team (valuable!) but often with no additional time to address it (demoralising), driving workflow changes for whole dev team (not just 1 or 2 enthusiasts) is hard, training & learning costs, and finally an inertia effect - the business shipped successful product without $tool in the past (otherwise there would likely be no money to buy it) and this is a trivial proof that $tool is not "essential".
You must be in a line of work absolutely foreign to high level programming. In language ecosystems like Java, C#, JavaScript, and so forth people live and die by tools. You are not hired as an employee for your ability to program, but for your years of mastery on some bloated tool. Just look at any job board.
Jobs posted to HN by start ups are frequently comical in this regard. On one hand they claim to be solving tough problems and hiring only the sharpest people around, but more than half the time the hiring criteria just the same tool insanity as the corporate world. If I can make more doing half the work and dealing with the same stupidity in the corporate world then why would I want to work your start up? Maybe I just like being punished.
Talking bad about some tools often brings upon the same emotional fire as burning holy scripture in public. For many developers if not for certain tools, for example Java Spring or the big JavaScript frameworks, they could not program at all. They simply lack the skills to practice without such tools.
In these cases tools are more than valued. They are as essential to air and food. Its unfortunate, because economically that value is entirely artificial. An employer can easily eliminate these large tools, eliminate half the associated jobs, retrain the rest of the staff to move forward without those tools and pocket the difference without lose of productivity.
Jobs posted to HN by start ups are frequently comical in this regard. On one hand they claim to be solving tough problems and hiring only the sharpest people around, but more than half the time the hiring criteria just the same tool insanity as the corporate world. If I can make more doing half the work and dealing with the same stupidity in the corporate world then why would I want to work your start up? Maybe I just like being punished.
Talking bad about some tools often brings upon the same emotional fire as burning holy scripture in public. For many developers if not for certain tools, for example Java Spring or the big JavaScript frameworks, they could not program at all. They simply lack the skills to practice without such tools.
In these cases tools are more than valued. They are as essential to air and food. Its unfortunate, because economically that value is entirely artificial. An employer can easily eliminate these large tools, eliminate half the associated jobs, retrain the rest of the staff to move forward without those tools and pocket the difference without lose of productivity.
Purify was a fantastic tool and I would have cheerfully paid $100K in late-90's dollars. We demoed it and the amount of memory errors it found in our legacy C and nascent C++ code stack was simultaneously depressing and exciting. We bought licenses and went to town. There was a markable product quality improvement as a result, at a time when our sales were exploding. I've never found a memory tool since that was as capable. We were running SGI IRIX boxes at the time.
I think about this often because it feels like my peak ability and productivity s a developer happened back on classic MacOS (7.5 through 9). There was a program called MacsBug that you could install, and with just two keys on the keyboard, invoke a stop-the-world debugger, complete with disassembly, stepping, and heap inspection. You could twiddle a bit here and there, even in the operating system. MacsBug shipped with a Forth interpreter, if I remember right, so you could even automate some of the debugging process. I still have my MacsBug reference guide.
I don't think there's a day that goes by that I don't wish I had those capabilities again.
But, operating systems also have another important function: they have to defend against malicious software. In the days of Macsbug, every program had full read/write access to every other program's heap. That would be horrifying today. Every tactic that's been developed in the name of secure computing has made OS-level debugging significantly harder.
I don't think there's a day that goes by that I don't wish I had those capabilities again.
But, operating systems also have another important function: they have to defend against malicious software. In the days of Macsbug, every program had full read/write access to every other program's heap. That would be horrifying today. Every tactic that's been developed in the name of secure computing has made OS-level debugging significantly harder.
Thanks for reminding me about this. MacsBug was great. Besides being a useful development tool, MacsBug was used extensively to break shareware as well by the mac warez community. I remember seeing some guides about it that were very illuminating about how program execution and object code actually worked that made things click for me.
It was also a time when any consumer could issue a system interrupt with the same two keystrokes because it was built into mac OS, and see the cpu registers and enter hex to change the program flow. Since there was not as clean a system/program barrier as there was today and single threaded OS (I think?), before OS 8 the system froze all the time because of apps, and even non-expert users memorized the hex code for the system exit() trap.
It was also a time when any consumer could issue a system interrupt with the same two keystrokes because it was built into mac OS, and see the cpu registers and enter hex to change the program flow. Since there was not as clean a system/program barrier as there was today and single threaded OS (I think?), before OS 8 the system froze all the time because of apps, and even non-expert users memorized the hex code for the system exit() trap.
0xA9F4, if memory serves.
While this essay serves as an interesting critique of the current state of operating systems and their development, it perhaps falls prey to the very flaw it seeks to highlight - the tendency to oversimplify the complexities inherent in systems software development and to yearn for a 'perfect' solution that is largely a figment of the author's imagination.
It's worth noting that when a VC says "there's no money in X", their definition of what constitutes "money" is very different from a normal persons. It's more accurate to say "there are no wildly market-inefficient profit margins to capitalize on and make hundreds of millions from in tools".
> The code used to build operating systems has only the most primitive of data types (lists, hashes, the occasional tree), while the libraries used in applications are a veritable cornucopia of modern data types.
Really? Is this only Linux (which gets a lot of eyeballs)? Mac OS? Maybe he's only talking about the kernel?
Really? Is this only Linux (which gets a lot of eyeballs)? Mac OS? Maybe he's only talking about the kernel?
He is definitely talking about the kernel (as well as things that run in kernel space like drivers). Userspace is all applications from this perspective, and can be debugged with a normal debugger.
If you use an OS with a tiny, well-tested microkernel, you debug drivers and such with the user-level debug tools. I wrote FireWire camera drivers for QNX debugging them that way, and doing compiles on the same machine upon which I was testing. Without rebooting.
That's not new. The first user-space driver development was for the Michigan Terminal System on the System/360 model 67. That was the first computer with virtualization hardware. People have been doing this in IBM land for over half a century.
That's not new. The first user-space driver development was for the Michigan Terminal System on the System/360 model 67. That was the first computer with virtualization hardware. People have been doing this in IBM land for over half a century.
The founders at a tool-making startup I worked for came back from a sales meeting one day looking very sad. I asked them why the long faces. They said they pitched the tool as something which would save their engineers a lot of pain.
"Eh?" said the prospect. "What? Who cares how much pain the engineers are in? Its their job to endure pain."
Clearly, the founders hadn't fully perfected their sales pitch....
"Eh?" said the prospect. "What? Who cares how much pain the engineers are in? Its their job to endure pain."
Clearly, the founders hadn't fully perfected their sales pitch....
Look at the catalog of ComponentSource (https://www.componentsource.com), which is mostly ... tools. Are Jetbrains and Progress (to name two offhand) doing badly?
This is very simple.
Tools in the "real world" generally work on one protocol: physics.
Yes.. there are standards and industries and the like in physical tools, but those are much more static and unchanging. Because physics and material world. A tool will generally be relevant until it wears out, and often/sometimes is useful for decades after "obsolescence".
In software, where (looks at the Javascript ecosystem) things change in a year. Sometimes less. Version churn and feature churn means your tool is obsolete/useless (if you are lucky) within 3-4 years. Often less.
In regular tools, it is pretty clear the distinction between a tool and the end product.
But in software, a tool is SOFTWARE, with all the problems of software: version churn, feature drift, underlying OS support, and all the other quicksand.
Tool markets are also inherently a fraction the endproduct market: The produced product volume is say 10x to 100x the value of the cost of the tool in the physical world before it needs replacement or wears out. Software being what it is, you can get 100000x - infinite the tool. That value proposition means the market for the tool is the inverse of the value factor of the tool. In software, you would get one tool and use it for 100000 products / views / etc. So the market for it is lower.
Perplexingly to my point about software tools going out of support, is that REALLY established tools are long lived and never wear out. So a hot gee-whiz tool will be competing against well-featured tools (often free after a while because tool vendors go out of business and open source it), so there isn't much gee-whiz anymore.
OpenAI is an interesting "tool". We'll see where it goes commercially. But most tools have reproducibility. AI tools do not, at least currently they don't.
Free software tools seem more durable, because the user population sustains it generally with contributions, bug reports, feedback, and demand. And sometimes the minimal money to support the org. And sometimes marketing.
Software is also generally about building a bandwagon. Gatekeeping tools are not good for bandwagon building. So software tools are generally free to make jumping on the bandwagon easier.
Tools in the "real world" generally work on one protocol: physics.
Yes.. there are standards and industries and the like in physical tools, but those are much more static and unchanging. Because physics and material world. A tool will generally be relevant until it wears out, and often/sometimes is useful for decades after "obsolescence".
In software, where (looks at the Javascript ecosystem) things change in a year. Sometimes less. Version churn and feature churn means your tool is obsolete/useless (if you are lucky) within 3-4 years. Often less.
In regular tools, it is pretty clear the distinction between a tool and the end product.
But in software, a tool is SOFTWARE, with all the problems of software: version churn, feature drift, underlying OS support, and all the other quicksand.
Tool markets are also inherently a fraction the endproduct market: The produced product volume is say 10x to 100x the value of the cost of the tool in the physical world before it needs replacement or wears out. Software being what it is, you can get 100000x - infinite the tool. That value proposition means the market for the tool is the inverse of the value factor of the tool. In software, you would get one tool and use it for 100000 products / views / etc. So the market for it is lower.
Perplexingly to my point about software tools going out of support, is that REALLY established tools are long lived and never wear out. So a hot gee-whiz tool will be competing against well-featured tools (often free after a while because tool vendors go out of business and open source it), so there isn't much gee-whiz anymore.
OpenAI is an interesting "tool". We'll see where it goes commercially. But most tools have reproducibility. AI tools do not, at least currently they don't.
Free software tools seem more durable, because the user population sustains it generally with contributions, bug reports, feedback, and demand. And sometimes the minimal money to support the org. And sometimes marketing.
Software is also generally about building a bandwagon. Gatekeeping tools are not good for bandwagon building. So software tools are generally free to make jumping on the bandwagon easier.
Tools matter where UX matters. Stuff that you set up once and a computer interfaces with in an automated fashion has no UX. That's almost the entire premium of every software tool in existence.
Being against a kernel debugger is one of the worst decision Linux ever made. Thankfully those days are behind us.
https://lwn.net/Articles/270089/
Linus Torvalds, in paraphrase:
> I happen to believe that stone knives and bear skins force people to think about their problem on a different level.
This is exactly as ridiculous as it sounds.
https://lwn.net/Articles/270089/
Linus Torvalds, in paraphrase:
> I happen to believe that stone knives and bear skins force people to think about their problem on a different level.
This is exactly as ridiculous as it sounds.
It's ridiculous, but at the simplest level, it's not wrong. Your tools or the lack thereof will change how you think about things, for better or worse depending on the situation.
If your goal is putting venison on the table tonight, vs understanding deer behavior and herd migration and when deer are even in the area in enough numbers and so can even be thought of as a viable food source, a stone knife and a bear skin as tools might promote (or force through necessity) better understanding than a hunting rifle and a large puffy jacket.
That doesn't mean it's "better" or that everyone should be using shitty tools, but it is food for thought, and that perhaps doing it a few times is a learning experience that isn't all negative.
If your goal is putting venison on the table tonight, vs understanding deer behavior and herd migration and when deer are even in the area in enough numbers and so can even be thought of as a viable food source, a stone knife and a bear skin as tools might promote (or force through necessity) better understanding than a hunting rifle and a large puffy jacket.
That doesn't mean it's "better" or that everyone should be using shitty tools, but it is food for thought, and that perhaps doing it a few times is a learning experience that isn't all negative.
Yeah, I think he overstates the case. I do see people who have become overly dependent on debuggers and that has adversely affected their debugging skills -- but that isn't an indictment of debuggers.
How is this ridiculous? I don't think the core of what he is saying (that tools affect how you think) is a matter of opinion. I can certainly see how people would disagree on his conclusion (therefore everyone must use the harder tools).
Tools that force you to hold a model of what's going on in your head and reason about its state require you to think completely differently than tools which hold the state or context for you outside of your head. Think about how different it feels to compose a message in your head, or with a pencil and paper, compared to using a word processor. You can even notice this effect by just typing text with your eyes closed as compared to staring at the screen as you type. The amount of context in your head affects how you think about things. At least for me, the tools I use greatly affect how much context I absorb into my head, and as a result what kind of theory generation and problem solving I can do, especially when a problem is hard enough that it requires several sessions of debugging.
Tools that force you to hold a model of what's going on in your head and reason about its state require you to think completely differently than tools which hold the state or context for you outside of your head. Think about how different it feels to compose a message in your head, or with a pencil and paper, compared to using a word processor. You can even notice this effect by just typing text with your eyes closed as compared to staring at the screen as you type. The amount of context in your head affects how you think about things. At least for me, the tools I use greatly affect how much context I absorb into my head, and as a result what kind of theory generation and problem solving I can do, especially when a problem is hard enough that it requires several sessions of debugging.
kernel debugging on linux sucks (IMHO)
I've tried it and it is really hard to get even the most basic debug output - a symbol'd stack trace of a crash.
If you can't get it at runtime, you need a kernel dump, and trying to get one is a mess. Once you have it, it is hard to match up the dump to the symbols. I believe some of the tools were kernel-version specific and I think when kernel 5 came out they didn't work.
I've tried it and it is really hard to get even the most basic debug output - a symbol'd stack trace of a crash.
If you can't get it at runtime, you need a kernel dump, and trying to get one is a mess. Once you have it, it is hard to match up the dump to the symbols. I believe some of the tools were kernel-version specific and I think when kernel 5 came out they didn't work.
I have run "crash" against a live kernel, i have run systemtap against a live kernel, kprobes and written my own C debugging linux kernel issues.
I can't speak for other distros, but you install the relevant debuginfo(symbol files) and then use your tooling, and its been pretty straight forward, at least for the last 10 years.
Hit me up if you ever need assistance doing it on standard x86-64, I'll do what i can to help.
I can't speak for other distros, but you install the relevant debuginfo(symbol files) and then use your tooling, and its been pretty straight forward, at least for the last 10 years.
Hit me up if you ever need assistance doing it on standard x86-64, I'll do what i can to help.
So not at all ridiculous. I prefer the method of dev Linus follows.
> A venture capitalist once told me, “There is no money in tools.”
That suggests he's saying there is not upside in a portfolio of tool companies. Following average VC advice gives average VC results. If everyone listened to this kind of stuff we'd be awash is generic SaaS CRM stuff...oh
That suggests he's saying there is not upside in a portfolio of tool companies. Following average VC advice gives average VC results. If everyone listened to this kind of stuff we'd be awash is generic SaaS CRM stuff...oh
I like writing tools and make all my money doing that. It's a niche market, but it's mine and worked out amazingly well.
[deleted]
The article does actually say that in highly specialised niches tool making is fantastically lucrative. So, don't mistake the headline for ground truth.
Also there is good reason to believe tool making since flint knapping has been high status, and demands surplus production to liberate skilled labour to making tools which have immensely high use value: tools are magnifiers of human productivity and are worth a lot.
The point is that when commodities are cheap, and tools like gdb can be considered a commodity, the relative advantage of "a better one" has to be weighed up with using a screwdriver as a hammer. Thus printf() debugging.