Ask HN: Which recurring tasks have you automated?
31 comments
I have done a lot of scripts/tools to mostly automate or simplify it for non-techs to take over, but probably the one that I am most proud of was a 'mailbox reaper' task that was fully automated by crawling the filesystem in a Netapp storage engine (NAS).
A prior version was a simple python script that ran as a cronjob but it needed lot of hand-holding as it would run at times when the filer was too heavy, and it was annoying to babysit it all the time esp. at odd hours.
The new version used snmp data from the filer to 'gauge' how idle or busy it was (as a feedback loop), and it would self-throttle or 'accelerate' itself in various stages as necessary. Saved the company a lot of grief in storage costs, and it ran like a charm with with hardly human intervention. It kept various stats and logs which could be queried as needed. Also saved a lot of time for other engineers/ops on-call team to focus on other areas.
Looking back at it, it used some crude mathematical models to gauge the busyness of the filer, but this portion was designed to be plugable and independent of how the filesystem was being crawled.
This simple program (set up as a forever loop daemon) got me thinking on autonomous agents and 'self-healing' software in general but never had time to pursue it in a more academic setting.
A prior version was a simple python script that ran as a cronjob but it needed lot of hand-holding as it would run at times when the filer was too heavy, and it was annoying to babysit it all the time esp. at odd hours.
The new version used snmp data from the filer to 'gauge' how idle or busy it was (as a feedback loop), and it would self-throttle or 'accelerate' itself in various stages as necessary. Saved the company a lot of grief in storage costs, and it ran like a charm with with hardly human intervention. It kept various stats and logs which could be queried as needed. Also saved a lot of time for other engineers/ops on-call team to focus on other areas.
Looking back at it, it used some crude mathematical models to gauge the busyness of the filer, but this portion was designed to be plugable and independent of how the filesystem was being crawled.
This simple program (set up as a forever loop daemon) got me thinking on autonomous agents and 'self-healing' software in general but never had time to pursue it in a more academic setting.
In my first "real" job, I worked at a publisher manually entering orders from paper and email into their production system. Multiple orders for multiple products required A LOT of repetitive typing: menu button pressing, addresses, costs, products.
I installed AutoHotKey (take that corporate security!) and over a few weeks wrote up a script that would let you supply an excel file with summarised stats about the orders you had that day. It would then physically automate the rest of the process using mouse/keyboard macros, like the office equivalent of a player piano.
It even had a little finite-state machine that knew where it was in the production system. For each order it knew where it had to be, and thus it could get from/to every other area of the production system by sending simulated keyboard/mouse-clicks.
I don't care what anyone says, sure an actual program would have been faster, but that wasn't an option.
In full flight it was awesome to behold. Like all the promises of labour saving technology made manifest! Of course, I soon learnt that labour saving was not the point of a corporation, but for the next few months until I quit, it was pretty cool...
I installed AutoHotKey (take that corporate security!) and over a few weeks wrote up a script that would let you supply an excel file with summarised stats about the orders you had that day. It would then physically automate the rest of the process using mouse/keyboard macros, like the office equivalent of a player piano.
It even had a little finite-state machine that knew where it was in the production system. For each order it knew where it had to be, and thus it could get from/to every other area of the production system by sending simulated keyboard/mouse-clicks.
I don't care what anyone says, sure an actual program would have been faster, but that wasn't an option.
In full flight it was awesome to behold. Like all the promises of labour saving technology made manifest! Of course, I soon learnt that labour saving was not the point of a corporation, but for the next few months until I quit, it was pretty cool...
AutoHotKey and AutoIt are the most used programs for "novice" developers i suppose.
For AutoIT you actually have a forum with a lot of scripts available: http://www.autoitscript.com/forum/forum/9-example-scripts/
For AutoIT you actually have a forum with a lot of scripts available: http://www.autoitscript.com/forum/forum/9-example-scripts/
I have a few sites that are on production, staging, and local servers. Each site is powered by its own SQL Server database.
There have been times when I have been working on a feature, and have noticed that the CMS content on local/staging is out of date with what is on live.
To fix this, I created a Powershell script to create a backup of the live db, copy it over to the staging server, backup the old db and restore the live db on staging, and to do the same on the local database. It's a bit awkward, but I wanted a one-click method that would sync all three databases, while ensuring that it would fail gracefully and would create backups in case something went wrong.
This works well for my personal projects, but I am looking to rewrite it for work at some point. I work at an agency, so a tool that can sync our db servers would be fantastic.
There have been times when I have been working on a feature, and have noticed that the CMS content on local/staging is out of date with what is on live.
To fix this, I created a Powershell script to create a backup of the live db, copy it over to the staging server, backup the old db and restore the live db on staging, and to do the same on the local database. It's a bit awkward, but I wanted a one-click method that would sync all three databases, while ensuring that it would fail gracefully and would create backups in case something went wrong.
This works well for my personal projects, but I am looking to rewrite it for work at some point. I work at an agency, so a tool that can sync our db servers would be fantastic.
Looks a lot like my PowerShell script integrated with my Visual Studio IDE.
I have an XML which contains the different servers (testing, dev and production).
I'd just type UpdateDatabases -T in package manager console :)=> so testing servers are updated with the latest migration
I have an XML which contains the different servers (testing, dev and production).
I'd just type UpdateDatabases -T in package manager console :)=> so testing servers are updated with the latest migration
Sounds fantastic! Ever considered putting the source up? I've considered pushing mine up on GitHub, but yours sounds better, and would work really well in an agency setting.
Starting a new project from a boilerplate and renaming/replacing/cleaning up the things.
It's a CLI Python script that copies templates to your desired location and asks you for replacement values. It works with regular expressions as well as strings and executes commands after cloning (e.g. initiate a git repo). It works of course for any language and any framework.
I started open-sourcing it and adding some things that will be useful for more people (e.g. files/folders to exclude, i18n, developer output, multiple template paths, install/setup script, etc.).
It's a CLI Python script that copies templates to your desired location and asks you for replacement values. It works with regular expressions as well as strings and executes commands after cloning (e.g. initiate a git repo). It works of course for any language and any framework.
I started open-sourcing it and adding some things that will be useful for more people (e.g. files/folders to exclude, i18n, developer output, multiple template paths, install/setup script, etc.).
Mmmm.. Probably could do this as well.
Is it template specific (eg. Bootstrap?)
Is it template specific (eg. Bootstrap?)
I needed a way of backing up mysql without writing a script to lock down the front-end (fancy isdown page), stop the webapp (just a php script that pulls data out of various APIs), stop mysql, perform the backup, then start everything back up.
So I run two instances of mysql, one is PROD and the other is read only replication. I can backup the read only replicated mysql instance without having to stop anything in production.
I am looking into the same for my nosql db's right now.
Regards,
So I run two instances of mysql, one is PROD and the other is read only replication. I can backup the read only replicated mysql instance without having to stop anything in production.
I am looking into the same for my nosql db's right now.
Regards,
3D file format conversion. It never works perfectly first time.
Currently, I have a Python script for format-converting and reapplying textures after an export from 3DS Max to an abandonware project called Mach Studio Pro, and I'm working on something similar to sort out opacity problems between Daz3D and Octane Render.
I've also made heavy use of a 3rd party script to mash converted World of Warcraft levels into a usable format for Max.
Currently, I have a Python script for format-converting and reapplying textures after an export from 3DS Max to an abandonware project called Mach Studio Pro, and I'm working on something similar to sort out opacity problems between Daz3D and Octane Render.
I've also made heavy use of a 3rd party script to mash converted World of Warcraft levels into a usable format for Max.
Just an example, from time to time we check all our clients pagerank.
I have a script that extracts our database of clients and checks their pagerank through their mail (of any contact) or the "website" field.
The pagerank gets saved in the clients "internal note" field
I have a script that extracts our database of clients and checks their pagerank through their mail (of any contact) or the "website" field.
The pagerank gets saved in the clients "internal note" field
Is there an API you're using, or is this scraped from somewhere?
The code isn't much, i added something to github which was like v1 (now it's more integrated and more extensive) and should make it clear to you.
https://github.com/NicoJuicy/GetPageRankWinForms/blob/master...
You get the PageRank by this:
https://github.com/NicoJuicy/GetPageRankWinForms/blob/master...
You get the PageRank by this:
int Result = GetThePageRank.DLL.GooglePR.MyPageRank(URL);
I can't think of any physical tasks that I've automated, but I try to use calendar reminders to save me mental space. For instance, I've got scheduled reminders to change my bedclothes, take the rubbish out the night before bin day, replace parts on my bike annually, etc.
I should integrate this to though ;-)
I renew my library books from a script in ruby. I haven't had a fine in over a year now.
ooh plenty
resizing images i'd receive from clients for their websites to a standard size. it just runs through the folder and resizes.
image manipulation to add borders
automating a continuous integration environment, built a website where developers could enter ticket details and a front end component they require delivered to the lab, which then kicks off the build, makes the components and uploads to the client's server and sends mail
db2 package creation by looking up files on a unix filesystem and providing them to the dba's packaging tool
and everything i install i typically have a bash script for.
resizing images i'd receive from clients for their websites to a standard size. it just runs through the folder and resizes.
image manipulation to add borders
automating a continuous integration environment, built a website where developers could enter ticket details and a front end component they require delivered to the lab, which then kicks off the build, makes the components and uploads to the client's server and sends mail
db2 package creation by looking up files on a unix filesystem and providing them to the dba's packaging tool
and everything i install i typically have a bash script for.
I am a photographer, and was migrating from Aperture to Lightroom. I wrote a node script to recursively rename files and folders exporting from Aperture sorting by date, which would otherwise take days as I transferred about 10000 photos
I created an API directory as a side project a few months ago. I manually posted new APIs submitted to the site on the Twitter account at first. It was kind of tedious, so I wrote a script to auto-post new APIs to Twitter.
Pulling down all my employee hours from Harvest (I run a consultancy), updating a MySql database, running a bunch of burn-down / project reports monthly and emailing our customers.
Over the past month and a half, I've built a glorified to-do list that keeps me on track. It automatically updates today's tasks, etc.
I automated Windows and SQL Server installations / configurations using Powershell. I initially started with Ruby but eventually switched over to Powershell due to all of the things like being able to use .NET objects from within Powershell, etc.
If you work with Windows, this is a really awesome tool.
If you work with Windows, this is a really awesome tool.
Yeah, Powershell is indeed extremely powerfull. Do you use it for something else also?
I use powershell to automaticly update multiple databases to the latest migration (but that's mostly because i'm using Entity Framework with .Net projects).
I use powershell to automaticly update multiple databases to the latest migration (but that's mostly because i'm using Entity Framework with .Net projects).
I'm still learning of all of the cool things that can be done with Powershell. So far, it's just been for server builds and configurations.
It would be great if, in the future, Powershell could be used for web automation. They already have WatiN, but you need to use C# to use it. While that is no problem, it would be great to be able to use Powershell for that as well.
It would be great if, in the future, Powershell could be used for web automation. They already have WatiN, but you need to use C# to use it. While that is no problem, it would be great to be able to use Powershell for that as well.
Didn't know of WatIn, thanks!
Knowing programmers, they could spend a lot of time automating boring repitive tasks.
So i'm wondering, what tasks have you automated and how much time does it save you?