Automate Everything - the hacker way(tomblomfield.com)
tomblomfield.com
Automate Everything - the hacker way
http://tomblomfield.com/post/17092502705/automate-everything
50 comments
I agree with you completely.
As engineers, our instinct is to build things. Business discipline requires focusing on profitable uses of time.
It's highly unlikely that building expense tracking software was the best use of time for the original author. He writes that thinking about the POTENTIAL future costs of expense tracking software ($1000/year) upset him.
If one realistically considers the costs to "roll your own" in terms of time, maintenance overhead, hosting & administrative cost, and opportunity cost the obvious conclusion is that this is not a good use of time and energy.
If Tom had instead spent that time fixing a bug, doing hard sales/marketing work it would have been worth much more to him.
No for us, engineers, doing hard, repetitive work is the real discipline. Automating things and finding systems that can be made more efficient is the easy work. Picking up a phone and making a sales call, trying to improve ad words advertising....these are things that are likely more valuable than rolling your own expenses email system.
As engineers, our instinct is to build things. Business discipline requires focusing on profitable uses of time.
It's highly unlikely that building expense tracking software was the best use of time for the original author. He writes that thinking about the POTENTIAL future costs of expense tracking software ($1000/year) upset him.
If one realistically considers the costs to "roll your own" in terms of time, maintenance overhead, hosting & administrative cost, and opportunity cost the obvious conclusion is that this is not a good use of time and energy.
If Tom had instead spent that time fixing a bug, doing hard sales/marketing work it would have been worth much more to him.
No for us, engineers, doing hard, repetitive work is the real discipline. Automating things and finding systems that can be made more efficient is the easy work. Picking up a phone and making a sales call, trying to improve ad words advertising....these are things that are likely more valuable than rolling your own expenses email system.
Except, by rolling our own systems and automating things we want to, we expand our potential and learn new things, which gives us somewhat difficult to quantify value for the future. One can't become an engineer without solving nontrivial (for one's own skills) problems.
"Software you solely maintain" and "software you pay someone else for" are not the only two options. You may have heard of something called "open-source software", which is a way that the load of maintaining a program can be spread around the community of its users — like paying a vendor to maintain it, but you pay in code instead of money, and you still get to use it even if you don't pay. Ruby, for example, the language he wrote this in, is open-source software. Much of the internet is built with it.
There are probably other companies in England that also need to report expenses to accounting.
There are probably other companies in England that also need to report expenses to accounting.
I know what you mean, and I'm torn. This project was built in about 4-5 hours on a Saturday, mainly as a chance to learn about mail handling. It's ugly as hell and there are no tests (!), so I don't know how well it will hold up as a long-term solution - I guess we'll see.
I just find the $9/month per employee a bit repugnant - if we grow to 10 people, we'll be paying over $1000 a year for a piece of software that parses subjects out of emails and stores attachments. Doesn't seem like a good deal.
I just find the $9/month per employee a bit repugnant - if we grow to 10 people, we'll be paying over $1000 a year for a piece of software that parses subjects out of emails and stores attachments. Doesn't seem like a good deal.
As a rough guide, you might factor in another 8-10 hours, because, according to Fred Brooks, a proper programming "product" takes about x3 as long as a "program".
Taking a wider view, there's an appeal to "cloud components", where all components are rented - including ones as simple/trivial as this. For this to work, it has to be cheap enough to be a no-brainer (esp for projects made of such components). Say, 1/100 - pennies - e.g. 9 cents /m/e. In the long-term, you'll see free open source alternatives, and maybe even free hosting (cloud providers supply it free, to support their hosting of your actual app)... and maintained.
Taking a wider view, there's an appeal to "cloud components", where all components are rented - including ones as simple/trivial as this. For this to work, it has to be cheap enough to be a no-brainer (esp for projects made of such components). Say, 1/100 - pennies - e.g. 9 cents /m/e. In the long-term, you'll see free open source alternatives, and maybe even free hosting (cloud providers supply it free, to support their hosting of your actual app)... and maintained.
I will give you the answer now -- it won't hold up :). Parsing and handling email is notoriously finicky, and every app I've written or maintained to do it has a constant stream of little bugs and edge cases that must be tested and handled that grow over time.
Would be interested to see a followup in 6-12 months.
Would be interested to see a followup in 6-12 months.
What about expensify - baseline activity is free. More frequent and complex usage isn't. https://www.expensify.com/upgrade
Instead of simply looking at the absolute value of the cost per employee, compare it to the time lost to the current method of accounting. Do you lose $9/month in work per employee on expense processing?
If it takes employees an average of 5 minutes per month, you don't have a problem. If it takes them 2 hours per month, investing $9/employee/month is a net win.
If it takes employees an average of 5 minutes per month, you don't have a problem. If it takes them 2 hours per month, investing $9/employee/month is a net win.
Hosting your own on Heroku isn't cheap either. Not to mention even more time savings when you go the paid route (interfaces into accounting software and what not).
It's also worth noting that for the specific problem he mentions, texthog is free for individuals and expensify is free for individuals and small groups.
This kind of thinking is as useful to non-tech founders as it is dangerous to technical founders.
For the non-techs, it's important for them to get their heads around the possibility that they may be wasting large quantities of worker-days doing what could be automated with a bash script.
For the techs, just because you can doesn't mean you should. Deciding to automate something when there's a cheap SAAS offering is often a destructive thing, mainly because even developers are prone to underestimate complexity and maintenance costs, and to goldplate a solution that's already "good enough".
I'm not saying Don't Automate. I'm just suggesting to be very cautious about undertaking any automation initiative. The things to automate are the baggage which you will take on your journey, and as such, they should be directly related up your core product/service. Something like a mailing list handler is probably not, and typical of the kind of underestimation that can take place (what happens when you discover your homebrew mailing list is blocked by every big mail provider's spam filter?).
For the non-techs, it's important for them to get their heads around the possibility that they may be wasting large quantities of worker-days doing what could be automated with a bash script.
For the techs, just because you can doesn't mean you should. Deciding to automate something when there's a cheap SAAS offering is often a destructive thing, mainly because even developers are prone to underestimate complexity and maintenance costs, and to goldplate a solution that's already "good enough".
I'm not saying Don't Automate. I'm just suggesting to be very cautious about undertaking any automation initiative. The things to automate are the baggage which you will take on your journey, and as such, they should be directly related up your core product/service. Something like a mailing list handler is probably not, and typical of the kind of underestimation that can take place (what happens when you discover your homebrew mailing list is blocked by every big mail provider's spam filter?).
I agree with you - I think most programmers have this mentality automatically, whereas the non-tech guys often don't. We're hiring for non-technical operations/customer-service/biz-dev roles a the moment and we find ourselves leaning towards people with a technical background.
I just don't know how much this "instinct to automate" can be taught.
I just don't know how much this "instinct to automate" can be taught.
I'll also add that this is the real value behind learning at least one language REALLY well. For me it's Python, for this guy it's Ruby. That means the language, it's ecosystem of platforms, and the standard lib.
Might be important to mention in these days where a new significant language seems to break onto the scene every 8 months or so.
I submit that any working programmer ought to know (or be about the business of knowing) at least one language well enough that they can go from idea to working prototype without spending a ton of time reading documentation and wondering about the canonical way to do something. One language where you can automate stuff and solve small problems.
Might be important to mention in these days where a new significant language seems to break onto the scene every 8 months or so.
I submit that any working programmer ought to know (or be about the business of knowing) at least one language well enough that they can go from idea to working prototype without spending a ton of time reading documentation and wondering about the canonical way to do something. One language where you can automate stuff and solve small problems.
For me it's shell scripts. Yeah they aren't ultimately as flexible as something like python, but piping together grep, sed, awk, and other command-line utilities, and some simple control-flow capabilities (if/then, loops, case statements) you can do a lot quickly.
And you can always write a little piece in Python, if any of the standard utilities are quite what you need.
And you can always write a little piece in Python, if any of the standard utilities are quite what you need.
What about Ruby and Perl and their ability to shell out as easily as you do in a shell? (with backticks)
You have a real language available with (arguably) fewer weird things and warts to be aware of or remember, better string manipulations and lots of other advantages, but if you really just want to send something through `awk` or `bc` you can do that just as easily as you do in bash and capture the result in a Ruby or Perl variable.
Seems like a win-win situation. (Unless you have an exceptional circumstance with requirements about running anything but shell scripts or something like that.)
You have a real language available with (arguably) fewer weird things and warts to be aware of or remember, better string manipulations and lots of other advantages, but if you really just want to send something through `awk` or `bc` you can do that just as easily as you do in bash and capture the result in a Ruby or Perl variable.
Seems like a win-win situation. (Unless you have an exceptional circumstance with requirements about running anything but shell scripts or something like that.)
I write quite a few Ruby utilities (for myself) and this is what I do: use backticks to harness the power of shell utilities, but have Ruby glue to do more complex tasks with those tools.
For example, I recently wrote a quick Ruby script to test various parts of my internet connection (ping default gateway, test DNS resolution, etc) because I was bored and my internet connection was down. I used backticks to do things like grab a nameserver from /etc/resolv.conf and then ping the server, but I had the surrounding Ruby code doing things like parsing the output using regular expressions and doing some logic on the results.
Sure, I could have written the thing in pure bash, but Ruby has better text manipulation capabilities when it comes to things like parsing (in my opinion), and with the backticks, I still was able to harness the power of the shell anyways.
I'm not sure how portable such solutions are, but for a home-grown testing program written on a whim out of boredom, it was certainly enjoyable to have both the power of Ruby and the shell available.
For example, I recently wrote a quick Ruby script to test various parts of my internet connection (ping default gateway, test DNS resolution, etc) because I was bored and my internet connection was down. I used backticks to do things like grab a nameserver from /etc/resolv.conf and then ping the server, but I had the surrounding Ruby code doing things like parsing the output using regular expressions and doing some logic on the results.
Sure, I could have written the thing in pure bash, but Ruby has better text manipulation capabilities when it comes to things like parsing (in my opinion), and with the backticks, I still was able to harness the power of the shell anyways.
I'm not sure how portable such solutions are, but for a home-grown testing program written on a whim out of boredom, it was certainly enjoyable to have both the power of Ruby and the shell available.
I agree that shell scripts are great. Add in something like Groovy or Scala to get access the multitude of java libraries out there and it's easy to do a lot of work very quickly.
This is a perfect example of the old saying that a good programmer is a lazy programmer. Lazy programmers hate doing the same monotonous things over and over and will go to great lengths to avoid it!
This is a close cousin to what my grandfather used to call "working to get out of work" in other words spending a lot of mental effort to avoid manual labor. This usually involves fashioning a tool or device to move something that could have been moved more quickly with simple, but back-breaking manual labor.
I tend to do both of these things a lot!
This is a close cousin to what my grandfather used to call "working to get out of work" in other words spending a lot of mental effort to avoid manual labor. This usually involves fashioning a tool or device to move something that could have been moved more quickly with simple, but back-breaking manual labor.
I tend to do both of these things a lot!
Lazysmart people are good people.
People who are eager to get up in the morning and toil in the sweat of their brows are deeply suspect in my book.
I don't see much good coming out of that kind of attitude - when will they stop and ponder the inescapable nuances of life?
The world needs more lazy inventor types laying around daydreaming on lazyboys, not extra widget crankers milling around mindlessly in labyrinthine office hives.
IMNSHO, of course :)
People who are eager to get up in the morning and toil in the sweat of their brows are deeply suspect in my book.
I don't see much good coming out of that kind of attitude - when will they stop and ponder the inescapable nuances of life?
The world needs more lazy inventor types laying around daydreaming on lazyboys, not extra widget crankers milling around mindlessly in labyrinthine office hives.
IMNSHO, of course :)
This is how and, more importantly, why I learned to program. My office uses a horribly unintuitive centralized database from which employees 'pull reports' to prove to supervisors the work they've done.
One of the guys in the office was maintaining a monthly report that involved him burning 3 hours a week pulling and compiling reports into a needlessly complex Excel spreadsheet. I offered to help out, and wrote an Excel macro that did the job for him.
From there, I figured 'hell, why should everyone have to spend time doing this?' So I downloaded a copy of VB and learned enough of it to turn the Excel macro into a simple deployable application that turned an absolute waste of manpower into an easy-to-use desktop app.
In hindsight, a native application wasn't the best way to do it. Our office has a strictly controlled internet/intranet policy, and the people maintaining the database are likely too busy doing just that — maintenance — to build a more robust data reporting system. That might be the beauty of learning on your own, you quickly gain insight into how things should and could work.
It wasn't his point, but simonw is right: building things IS fun. I got started and I haven't looked back.
One of the guys in the office was maintaining a monthly report that involved him burning 3 hours a week pulling and compiling reports into a needlessly complex Excel spreadsheet. I offered to help out, and wrote an Excel macro that did the job for him.
From there, I figured 'hell, why should everyone have to spend time doing this?' So I downloaded a copy of VB and learned enough of it to turn the Excel macro into a simple deployable application that turned an absolute waste of manpower into an easy-to-use desktop app.
In hindsight, a native application wasn't the best way to do it. Our office has a strictly controlled internet/intranet policy, and the people maintaining the database are likely too busy doing just that — maintenance — to build a more robust data reporting system. That might be the beauty of learning on your own, you quickly gain insight into how things should and could work.
It wasn't his point, but simonw is right: building things IS fun. I got started and I haven't looked back.
Perfect is the enemy of good.
If innovation had to sit around and wait for the "best solution" or be overly concerned with how things "should" work, you'd be cooking your dinner by pounding it with a rock and we'd all be in awe of enterprise software architecture.
Innovation is about solving problems with the tools and resources that are available, and kudos to you for doing just that. If more people would be bothered (and, indeed, allowed) to learn a little programming, even in something like VB, the productivity of an average office worker would explode.
If innovation had to sit around and wait for the "best solution" or be overly concerned with how things "should" work, you'd be cooking your dinner by pounding it with a rock and we'd all be in awe of enterprise software architecture.
Innovation is about solving problems with the tools and resources that are available, and kudos to you for doing just that. If more people would be bothered (and, indeed, allowed) to learn a little programming, even in something like VB, the productivity of an average office worker would explode.
I posted this on the blog but I'll copy it here as well. Please forgive the shameless plug.
I think the OP is right in that wanting to automate repetitive tasks is inherent to humans. And the problems discussed in this post are exactly the kind my startup http://loggur.com aims to address. The goal is to let non-programmers automate and also to let programmers automate certain tasks (mainly data manipulation, analysis, triggers, and reports/notifications) in a fraction of the time it would take to otherwise hand-code it.
I should also mention that I am fully aware that what Loggur attempts to do is far from new, but I have yet to see an implementation that isn't a pain to work with. And it's designed with the current (and predicted future) state of things in mind and to be flexible enough to adapt to whatever the needs of the future may be.
I think the OP is right in that wanting to automate repetitive tasks is inherent to humans. And the problems discussed in this post are exactly the kind my startup http://loggur.com aims to address. The goal is to let non-programmers automate and also to let programmers automate certain tasks (mainly data manipulation, analysis, triggers, and reports/notifications) in a fraction of the time it would take to otherwise hand-code it.
I should also mention that I am fully aware that what Loggur attempts to do is far from new, but I have yet to see an implementation that isn't a pain to work with. And it's designed with the current (and predicted future) state of things in mind and to be flexible enough to adapt to whatever the needs of the future may be.
One thing that always surprises me is how many times companies reinvent the wheel and the need for it. In your post you mention an expenses system; something pretty much every company will have to implement. Since HMRC require this information you'd think they'd provide a tool that everyone could "plug into", giving them full visibility of the data they require in a standard format with little effort, and allowing businesses to also benefit by just downloading the relevant information from this service into their own systems for any internal needs. If all companies were using the same service this would also be one less thing for new employees to learn, since they'd be familiar with the expenses process from previous jobs.
On a persona note, I'm definitely in the automater category; on the few occasions when I've left a task unautomated because it was only supposed to be a one off it's come back to bite me repeatedly when month after month that same one off task pops up (admittedly after 2-3 repeats I then ignore the advise that it's a one off and automate anyway). These days if I can automate a task in slightly over the time it would take to perform that task, I automate by default. If there's a significant difference I weigh up the likelihood of repetition (by myself and/or others) against the extra time required to automate it.
On a persona note, I'm definitely in the automater category; on the few occasions when I've left a task unautomated because it was only supposed to be a one off it's come back to bite me repeatedly when month after month that same one off task pops up (admittedly after 2-3 repeats I then ignore the advise that it's a one off and automate anyway). These days if I can automate a task in slightly over the time it would take to perform that task, I automate by default. If there's a significant difference I weigh up the likelihood of repetition (by myself and/or others) against the extra time required to automate it.
With tasks I only perform occasionally, automation becomes my record of how to do those tasks. I find this extremely valuable for complex tasks infrequently performed. Taking the time to write a script ensures that (1) I pay attention to what I'm doing, not just run into a snag->google->get the fix->forget (or never investigate!) what the hell the problem was; and (2) allows me to quickly check that my procedure is repeatable; and (3) gives me a step-by-step record of exactly what needs to be done, with no steps skipped (like when I take notes) because they're "obvious." What is obvious when you've been working a couple of hours to solve a problem is very different than what is obvious when you haven't done something in a couple of months.
It's OT, but I've found that expense report submission requirements are often (either willfully, our just willfully not improved) difficult in order to reduce the rate at which they are submitted.
That's interesting. You'd think that would be irrational for the company as a whole, no? If your employees are paying for their own housing, meals, transportation, massages, etc., instead of the company taking those on as business expenses, then for every US$1000 the company saves on expenses, it has to pay its employees US$1300 or so in higher salaries to make up for it, with the other US$300 going to income tax, payroll tax, etc. What's wrong with my thinking, or is this really true?
Its just one of the dozens of sneaky ways that less than moral companies say they're paying employees competitive rates and then... don't.
...polls our pop3 server once per minute...
...attachments are saved to Amazon S3...
...details are stored in Postgres...
This is the hacker way? Was it designed? It is not elegant. I think it was grown. It uses technologies which are cool and hip but also dependent on third party services.
POP3 is not a reliable delivery protocol for unique messages.
I'll leave it at that.
This is the hacker way? Was it designed? It is not elegant. I think it was grown. It uses technologies which are cool and hip but also dependent on third party services.
POP3 is not a reliable delivery protocol for unique messages.
I'll leave it at that.
The "hacker way" was a small nod to the recent Facebook filing - I don't it necessary implies any kind of elegant design.
Zuckerberg explains "hacking just means building something quickly... Hackers believe that something can always be better, and that nothing is ever complete. They just have to go fix it — often in the face of people who say it’s impossible or are content with the status quo...Hackers try to build the best services over the long term by quickly releasing and learning from smaller iterations rather than trying to get everything right all at once."
Zuckerberg explains "hacking just means building something quickly... Hackers believe that something can always be better, and that nothing is ever complete. They just have to go fix it — often in the face of people who say it’s impossible or are content with the status quo...Hackers try to build the best services over the long term by quickly releasing and learning from smaller iterations rather than trying to get everything right all at once."
Email is super-convenient though, and honestly I can't say that I've ever had an email just vanish into the ether, though certainly that's possible. How often does Accounting lose an expense report or receipt? I bet it's more often than emails get lost... especially intra-company emails.
A simple web page on the company intranet that let you upload the scan, and key in a description and amount, would be more reliable but somewhat less convenient. It's a trade-off.
A simple web page on the company intranet that let you upload the scan, and key in a description and amount, would be more reliable but somewhat less convenient. It's a trade-off.
The informality of the solution seems at odds with the BigCo expense reporting story told in the problem description: "Presumably, hundreds of millions of people have to do this every month, costing millions hours of lost productivity. And in most companies, it’s because a well-meaning HR or Finance person has said that It Must Be Done."
This email-based expense system hinged on specific keywords would be great for a small business, but if you're a multinational with lots of legal oversight...
This email-based expense system hinged on specific keywords would be great for a small business, but if you're a multinational with lots of legal oversight...
>This email-based expense system hinged on specific keywords would be great for a small business, but if you're a multinational with lots of legal oversight
Its still not necessarily bad, such a system could be easily expanded to be an quick input method to a system that makes the beancounters and legal happy too. Say I snap a picture of a till receipt with my cellphone and send it off to the expense account. The backend could also hook into our timesheet webapp. I already have to choose the account I'm billing my hours to and some accounts require I put in a note. My expense could show up right in the list of hours I've clocked, then when I enter my hours at the end of the day I double check all the expenses are there, are for the right amount, choose the account to charge and pop in a description.
Its still not necessarily bad, such a system could be easily expanded to be an quick input method to a system that makes the beancounters and legal happy too. Say I snap a picture of a till receipt with my cellphone and send it off to the expense account. The backend could also hook into our timesheet webapp. I already have to choose the account I'm billing my hours to and some accounts require I put in a note. My expense could show up right in the list of hours I've clocked, then when I enter my hours at the end of the day I double check all the expenses are there, are for the right amount, choose the account to charge and pop in a description.
Does anyone know if (in the UK at least) HMRC require companies to submit the actual receipts (physical bits of paper) or do scanned images suffice?
http://blog.receipt-bank.com/?p=428 quote HMRC as saying
"Records may be preserved on optical imaging systems, and the originals discarded, provided that what is retained in digital form represents a complete and unaltered image of the underlying paper document."
giving the reference as http://webarchive.nationalarchives.gov.uk/20110720224136/htt...
"Records may be preserved on optical imaging systems, and the originals discarded, provided that what is retained in digital form represents a complete and unaltered image of the underlying paper document."
giving the reference as http://webarchive.nationalarchives.gov.uk/20110720224136/htt...
Call the tax authorities and ask. At least then, you have some cover for your ass if it turns out their don't suffice.
I don't know about UK, but in the USA scanned images are legally adequate for this sort of thing.
I sure hope so. I'll phone up tomorrow and ask.
To my mind the meta-hack here would be to do away with the need for expense reporting entirely by giving everyone a corporate credit card like 37signals did. http://37signals.com/svn/posts/893-workplace-experiments
Your start-up might not be in a position to do that, but removing the task entirely always seems preferable to me.
Your start-up might not be in a position to do that, but removing the task entirely always seems preferable to me.
I'm not a programmer, but I do like to automate and my favorite way at the moment is using macros, bash scripts and WinAutomation - I can't believe how much can be automated this way. Email replies, online searches, software updates, website testing, testing content for uniqueness, proof reading and renaming files, taking screenshots, adding tasks to the calendar - everything can be automated using existing software and something to connect it all together (in my case, it's WinAutomation).
Obviously, it's not scalable, but anything that saves you an hour of sorting 500 files by category is a win in my book...
Obviously, it's not scalable, but anything that saves you an hour of sorting 500 files by category is a win in my book...
Sir, you are definitely a programmer.
I write shell scripts to do just about everything. Good point about teaching these skills to non-programmers.
I think a balance between automation & manual gives the best efficiences. Particularly in the rapid building pre production stage of a solution.
My context is data processing matlab (automation) & excel (manual).
My context is data processing matlab (automation) & excel (manual).
Interestingly, the Tibetan word for meditation (Gompa) means, literally, a repetitive mental task.
Thus far, we have machines that think for us in countless ways, none yet that meditate for us.
Thus far, we have machines that think for us in countless ways, none yet that meditate for us.
Tom, you might be interested in http://wappwolf.com/ :)
The code would be fun to see if you have a moment.
[deleted]
I find myself having to fight against this instinct almost every day. There are a couple of problems with hacking for a couple of hours to save a few bucks a month.
Firstly, you now have an additional piece of software to maintain - you're committing yourself to an unknown quantity of future work.
Secondly, your software won't get any better without you actively improving it. The nice thing about software you pay someone else for is that it gets better over time.
It's a tough instinct to fight though. Building things is Fun. It's just that there are probably other things you should be building that are more important to your company.