Ask HN: What set your hair on fire in 2012?
I'm curious to know what serious technological / IT problems you encountered in 2012. What things made you work late at night, scrambling for a solution? EC2 outages did it for some people. What else? What failed unexpectedly? For what problems do you still not have a solution, and just hope for the best?
29 comments
Trying to find the formula to help companies with no MySQL experience (yet running mission critical MySQL instances) get stable, redundant, & backed up fast.
There's nothing like being paged 10 times an hour at 11pm to get you motivated to get things fixed.
There's nothing like being paged 10 times an hour at 11pm to get you motivated to get things fixed.
When people in tech say "paged" do they/you literally mean it? Or do you just mean text message but the terminology has stuck as a throwback to the days when pagers were how people were on emergency call.
I suspect there still is a small pager business out there somewhere, perhaps for doctors and adhering to some legacy hippa rules, but they must be sparse.
I suspect there still is a small pager business out there somewhere, perhaps for doctors and adhering to some legacy hippa rules, but they must be sparse.
I carry around an honest-to-goodness old-school pager. I love it. It works even when I get next to no cell signal, it works when I screw up the newest ROM of my phone. It works when I'm out of battery. It runs weeks on a AA.
I have messages go to SMS as well, but when the servers are down, damn it, I need to know.
I have messages go to SMS as well, but when the servers are down, damn it, I need to know.
It's a combination of growl/SMS/email. The terminology has stuck around (and I know doctors still use actual pagers for their reliability).
And given our issues lately with our phone carrier silently blocking our messages as spam, the thought of going back to an old school pager for reliability is still there.
And given our issues lately with our phone carrier silently blocking our messages as spam, the thought of going back to an old school pager for reliability is still there.
Paypal payflow integration. There is a zoo of paypal products, and outdated paypal integration docs that obfuscate the process.
Online payments systems in South America.
Timezones in javascript. We ended up stitching together 3 libraries into something that sort of looks like 10% of joda-time. Definitely not a solved problem.
Which libraries did you end up using?
Yeah, which ones, I also have timezone hair pulling for the front end.
So far moment.js seems to be the best solution for me.
Getting require.js to play nicely with rails 3 asset pipeline on heroku. It would all work fine locally but compiling js always had some dependency issue. Finally got it working some week or two later.
Jython. It's an unfinished research project/toy/proof of concept that never should have been used in a production environment under any sort of load. Unfortunately my predecessors made the wrong choice and I inherited that technical debt.
Can't you easily rewrite the critical parts in Java? (That should be much easier than writing C extensions for CPython.)
The problems are in the core implementation of Jython, not in using Jython to solve problems. For example: http://bugs.jython.org/issue1634167
well for me "hair on fire" means "what were you really inspired about/by" to the point you stay up late working on it. For me:
ML for the working programmer - started down this road as I wanted to make sure I really "get" what is being expounded upon by haskell/shen etc.
shen/klambda - every time I dive into this bottomless pool I come back up with a new insight to how programming could/will someday look.
amber smalltalk - close to my ideal collaborative programming environment - the new helios ide and compiler progress bring it even closer!
edn - seriously I don't think enough hype has been made about what edn can bring to the table, specifically w/ "tagged" data. Stayed up one night and wrote jsedn so I could start to use edn in projects.
datomic - There are no silver bullets in database programming, but datomic is damn close. Trying to wrap my head around the specifics culminated in prototyping datomicism, my smalltalk esque environment for working w/ datomic.
minecraft - which has led to working on a full blown 2d/3d schematic system (stay tuned for http://schematic.io) and a rediscovery of "primitive" computer geometry (namely "geometric principles and procedures for computer graphic applications" by Sylvan H. Chasen). This will hopefully serve as a nice open source example of how to meld coffeescript/backbone/datomic and a edn based DSL for helping kids "code by osmosis" as it follows the bret victor principle of showing the code change as the objects are manipulated (and vice versa).
ML for the working programmer - started down this road as I wanted to make sure I really "get" what is being expounded upon by haskell/shen etc.
shen/klambda - every time I dive into this bottomless pool I come back up with a new insight to how programming could/will someday look.
amber smalltalk - close to my ideal collaborative programming environment - the new helios ide and compiler progress bring it even closer!
edn - seriously I don't think enough hype has been made about what edn can bring to the table, specifically w/ "tagged" data. Stayed up one night and wrote jsedn so I could start to use edn in projects.
datomic - There are no silver bullets in database programming, but datomic is damn close. Trying to wrap my head around the specifics culminated in prototyping datomicism, my smalltalk esque environment for working w/ datomic.
minecraft - which has led to working on a full blown 2d/3d schematic system (stay tuned for http://schematic.io) and a rediscovery of "primitive" computer geometry (namely "geometric principles and procedures for computer graphic applications" by Sylvan H. Chasen). This will hopefully serve as a nice open source example of how to meld coffeescript/backbone/datomic and a edn based DSL for helping kids "code by osmosis" as it follows the bret victor principle of showing the code change as the objects are manipulated (and vice versa).
Yes stay tuned for schematic.io - I am doing the UI and UX for this w/ shaunxcode.
What sets my hair on fire is coupling UX with BDD and the rapid development chain of realtime UX and programming. Having UX write the UI cucumber scenarios helps a lot. But this is for pet projects. I'd really like to see the world of business accept and start trying out using the UX process as a precursor to BDD and allowing devs to work more freely on what they are good at with a constant pump of UX feeding them the things they don't want to think about.
What sets my hair on fire is coupling UX with BDD and the rapid development chain of realtime UX and programming. Having UX write the UI cucumber scenarios helps a lot. But this is for pet projects. I'd really like to see the world of business accept and start trying out using the UX process as a precursor to BDD and allowing devs to work more freely on what they are good at with a constant pump of UX feeding them the things they don't want to think about.
Can you describe a bit more about schematic.io? I quick google search didn't turn up much. What is this going to be?
Basically a CAD tool for minecraft in which you can create schematics for structures in a 2d editor w/ a live 3d preview. The "use case" is for sharing a complex build plan with multiple users on a "layer by layer" basis so that many people can collaborate either in real or elapsed time. e.g. I may start building a castle and leave it w/ a sign in the center giving the schematic.io url so that another user can come along and make progress on it according to said schematic.
Right now this works for "block" structures e.g. there is not yet the ability to place redstone wires/circuits - but that is coming soon. The use case there being to increase the level of complexity and radness users can build w/o having to "frame by frame" a youtube video on building complex piston doors!
Right now this works for "block" structures e.g. there is not yet the ability to place redstone wires/circuits - but that is coming soon. The use case there being to increase the level of complexity and radness users can build w/o having to "frame by frame" a youtube video on building complex piston doors!
Not knowing how to research 'hair on fire' problems other people have, so that I could create a solution, add value, make money, one-man-startup style. I know it's kinda meta.
Reading Kalzumeus and other entrepreneurs opened the door a little. Lean startup pushed me along the way as well.
Reading Kalzumeus and other entrepreneurs opened the door a little. Lean startup pushed me along the way as well.
Funny, the "new" technological problems I think OP is referring to are rarely things that are that difficult to deal with.
But these are the things that continue to keep me up at night year after year:
But these are the things that continue to keep me up at night year after year:
10. How clever to get with javascript in the browser without going overboard.
9. How much javascript to depend upon on the client.
8. Optimistic or pessimistic database locking?
7. Do it at compile time or run time?
6. Do it on the client or the server?
5. Keep data on the client or make the round trip?
4. Mouse, keyboard, either one, or something new?
3. Which browsers and/or platforms to support?
2. Simplicity vs. capability in the user interface.
1. How to deal with difficult/stupid peopleThe prevalence of #1 on your list (among other reasons) are why I always have and always will believe that improvements to education yield by far the most return on investment vs any other allocation of resources. Improving education throughout the world is the single best way to increase economic output.
Now, things I did not say:
* "Put more money into our education system as it is."
We are in need of vast improvement, and we should be investing time and money towards innovation and improvement.
* "Stop funding and/or stop using our existing educational system."
Yes, there are many, many flaws, but our educational systems have improved our society a great deal, and are fantastic avenues and methods of distribution for educational innovation.
* "Edtech is the best industry!"
I'm still unsold on the edtech space as a way of improving education. Particularly, I fear that our existing models for business, when applied to education, will exclude the vital underpriveleged demographic who have some of the most potential for growth. Aditionally, much of the edtech I see used in schools does little to actually aid student learning—but it certainly does look good on a quarterly newsletter where administrators list what fantastic things they're doing for their school. Whoever figures out how to make equitable, effective educational technology that turns a profit will be one of the preeminent innovators of our time.
(Note: If you believe you or your company are doing this, I have Ruby on Rails skills, am willing to learn any technology you work with, and I want to work for you.)
In sum: The act of improving the effectiveness of the systems which themselves improve the effectiveness of human capital yields exponential and undervalued economic gains. I hope in the future it receives more focus.
Now, things I did not say:
* "Put more money into our education system as it is."
We are in need of vast improvement, and we should be investing time and money towards innovation and improvement.
* "Stop funding and/or stop using our existing educational system."
Yes, there are many, many flaws, but our educational systems have improved our society a great deal, and are fantastic avenues and methods of distribution for educational innovation.
* "Edtech is the best industry!"
I'm still unsold on the edtech space as a way of improving education. Particularly, I fear that our existing models for business, when applied to education, will exclude the vital underpriveleged demographic who have some of the most potential for growth. Aditionally, much of the edtech I see used in schools does little to actually aid student learning—but it certainly does look good on a quarterly newsletter where administrators list what fantastic things they're doing for their school. Whoever figures out how to make equitable, effective educational technology that turns a profit will be one of the preeminent innovators of our time.
(Note: If you believe you or your company are doing this, I have Ruby on Rails skills, am willing to learn any technology you work with, and I want to work for you.)
In sum: The act of improving the effectiveness of the systems which themselves improve the effectiveness of human capital yields exponential and undervalued economic gains. I hope in the future it receives more focus.
The problems as you've listed here have no context, which will make them extremely difficult to answer, if not impossible.
For example:
For example:
> How clever to get with javascript in the browser
> without going overboard.
edw519 has not listed problems to be solved but dilemmas/trade-offs to be managed.
Sending payments via Paypal. Yes, I understand it is very simple, but Paypal has restrictions in my country. And i only have debit card and no credit cards. And debit cards don't work online apparently. I can connect my back account but that requires my pan info (like SSN) and a bank confirmation. My PAN info is still not confirmed. It surprises me how difficult it can be to send money online when you are in a particular country. No only that but I was unable to use my debit card anywhere on the entire internet. I wanted to buy windows 8 and had to use my uncle's credit card who does not live in this country. Had to figure out a way to send him money too. It's ridiculous how you become a second class citizen if you don't have a credit card. This is 2013 (in my timezone it is) and there has to be a better way to transfer money.
Prepaid Visa/Mastercard cards work and they're pretty much the equivalent of a debit card. You put money on them and use them to buy stuff online.
AFAIK we don't have them here. Even if we had they won't work online due to government restrictions. I live in India and they had a few problems with how paypal can be used to launder money.
[deleted]
1. Authorize.net's ARB (integrated with it before really good solutions came into view)
2. A chaotic - but good - life (difficulty in regaining my daily routine due to startup + girlfriend + other commitments)facebook APIs. Random breakages, incomprehensible changes, unanswered PRs etc. And AWS outages. And Heroku outages. And OAuth integrations.
Basically, everything not 100% under my control keeps going wrong. And so does everything under my control, but apparently causing smaller fires.
Basically, everything not 100% under my control keeps going wrong. And so does everything under my control, but apparently causing smaller fires.
Yup.
Just about everybody else on this thread seems to be thinking at the wrong granularity.
I've been thinking about your problem all year. http://akkartik.name/blog/libraries2
Just about everybody else on this thread seems to be thinking at the wrong granularity.
I've been thinking about your problem all year. http://akkartik.name/blog/libraries2
Scaling. Oh god. Scaling.
My biggest pain in the ass this year was as simple as disk space constantly running out on our VPS servers. That, and automating postgres backups.
We put all our databases on a mapped /data drive that is larger and allocated in the san for this reason. I'm not the sysadmin and the policy isn't mine, but it seems to work....
How did you automate your postgres backups? I'm going to be tackling this for a client soon and have a few ideas, but would love to hear how you did it.
Well, first of all I mistakenly assumed our VPS backups would be sufficient, but the first step is actually dumping the database every interval you'd like. If you don't dump it, you might end up with a corrupt database should you be relying on the VPS backups.
Then, we set up a process to zip the database dump and send it off to tarsnap. We delete the backup done 30 days ago each time we run.
It works fine for us, but I much prefer using Heroku and not having to worry about it :)
Then, we set up a process to zip the database dump and send it off to tarsnap. We delete the backup done 30 days ago each time we run.
It works fine for us, but I much prefer using Heroku and not having to worry about it :)
By an large, candles. Candles and long hair are a bad combination.
On a serious note, a piece piece of software called Player/Stage which I am working with is driving me up the wall. Learning how three.js and the collada spec work to try and get the two playing nicely. Cross-browser CSS3/HTML5. Weirdness troubleshooting a flash app running with a PHP backend on a hundred android 1.6 tablets you don't have code access to.
On a serious note, a piece piece of software called Player/Stage which I am working with is driving me up the wall. Learning how three.js and the collada spec work to try and get the two playing nicely. Cross-browser CSS3/HTML5. Weirdness troubleshooting a flash app running with a PHP backend on a hundred android 1.6 tablets you don't have code access to.
Chef Scripts. But now that I have Vagrant the fire is out. Seriously. Vagrant is amazing to the point of being nearly a silver bullet like Rails was. Funny thing was, when I first looked at it, I couldn't even figure out why I would want to use it, but now I can't live without it.
Wordpress E-commerce plugin. Stay. Away. From. It.
There's a reason why it has such terrible ratings and reviews and complaints. It has a lot of features that the others don't, and that tricks you into thinking it's right for your online store. Only to be 1 month in, and start discovering all the terrible programming, glitches, bugs, php errors, and short commings.
Stay away from WP E-Commerce. You've been warned.
There's a reason why it has such terrible ratings and reviews and complaints. It has a lot of features that the others don't, and that tricks you into thinking it's right for your online store. Only to be 1 month in, and start discovering all the terrible programming, glitches, bugs, php errors, and short commings.
Stay away from WP E-Commerce. You've been warned.
Especially if your theme, or anything close to your wordpress install (other plugins, etc) use any javascript libraries that might share variables or have different version numbers. Agreed.
Refactoring one of my old websites (200,000 lines of code). Code was horrible. I think in 5 years I will tell the same story about my recently refactored code =)
After building online business for the last 16 years, I took serious venture capital for the first time in 2012. So managing that relationship and meeting someone else's expectations has shaped my 2012 (set my hair on fire). I have a very patient investor, and it's still not easy, I don't know how people deal with the real sharks of VC.
Sencha Touch + Phonegap, as it turns out it's not very pleasant to try to make a good looking responsive multiplatform app, specially with android browser weirdness and bugs.
Lesson learn, if you care about the user experience there is no other way than native.
Lesson learn, if you care about the user experience there is no other way than native.
Couldn't second this more.
What killed me:
Ms11-100 security patch. Utter disaster that broke our entire cache and UI layer. Literally our kit jumped from 10% utilisation to 90% overnight resulting in a last minute rollback. Didn't show up in load testing.
business analysts lack of grasp on reality "its only a few billion rows" (to download our audit data via HTTP in CSV format - not going to happen).
SQL Server 2012 pricing.
Mark shuttleworth being an ass over a amazon integration causing us to move all our Linux stuff to debian.
VMware 2tb lun limit on iscsi. This sucks badly.
Compilation speed of c# - this is knackering us. Java is much better as it doesn't have to create monolithic dependency units (assemblies).
Exploding kit.
Ms11-100 security patch. Utter disaster that broke our entire cache and UI layer. Literally our kit jumped from 10% utilisation to 90% overnight resulting in a last minute rollback. Didn't show up in load testing.
business analysts lack of grasp on reality "its only a few billion rows" (to download our audit data via HTTP in CSV format - not going to happen).
SQL Server 2012 pricing.
Mark shuttleworth being an ass over a amazon integration causing us to move all our Linux stuff to debian.
VMware 2tb lun limit on iscsi. This sucks badly.
Compilation speed of c# - this is knackering us. Java is much better as it doesn't have to create monolithic dependency units (assemblies).
Exploding kit.
How to combat frauds with e-commerce transactions:
1. Pro-actively block Tor nodes and all hide-my-ass type of open proxy. To e-commerce merchants, tor node users are not freedom fighters, but fraudsters, every-single-one-of-them.
2. For credit card transaction, always check the return "credit card" country
3. Shopping carts need to track IP address. Fraudsters always can bounce from Brazil to UK to Italy in a single cart session.
4. Have to come up with a score system to determine fraud risk of transactions. Maxmind API seems to be helpful.
5. When in doubt, ask user to take a picture of their credit card, and upload to your site.
1. Pro-actively block Tor nodes and all hide-my-ass type of open proxy. To e-commerce merchants, tor node users are not freedom fighters, but fraudsters, every-single-one-of-them.
2. For credit card transaction, always check the return "credit card" country
3. Shopping carts need to track IP address. Fraudsters always can bounce from Brazil to UK to Italy in a single cart session.
4. Have to come up with a score system to determine fraud risk of transactions. Maxmind API seems to be helpful.
5. When in doubt, ask user to take a picture of their credit card, and upload to your site.
You really cannot be serious about asking a user to take a photo of their credit card and upload it.
The only other alternative recommended by payment gateway, would be to ask you to fax your passport. The point is, to prove that you really do process the card, not getting it through some shady sites.
Try registering an hosting account on many "managed hosting" company, you are required to fax your passport to them. I will be offended if somebody asks me to do so.
Uploading credit card image is definitely less evil than faxing your passport.
Try registering an hosting account on many "managed hosting" company, you are required to fax your passport to them. I will be offended if somebody asks me to do so.
Uploading credit card image is definitely less evil than faxing your passport.
As bad as this solution is, I'd choose this form of user identification (id photographed and uploaded) over the last 5-10 captchas I have done. They have been killers for me lately, and one took a day of on and off trying to get past.
I'm from Russia and I tell you he's serious. I've been asked that a few times.
This year I switched to Ruby on Rails after years of doing mostly Java. I'm quite happy with it, but by far the biggest pain point has been Ruby's dependence on native code libraries. Dealing with CLASSPATHs and Maven was never fun, but it sure beats trying to maintain a working set of Ruby gems, many of which have cascading dependencies on C++ compilers, Python interpreters, SSL libraries, Javascript runtimes, etc.
Realizing that I would have to use Javascript.
Finding freelancer peers to sub-contract and refer work to. When you're overflowing with work, it's more difficult than ever to find people through events / traditional slow networking, and there's no good online alternative.
I wish there were a site that wasn't transaction-focused, that provided a way to find people with specific skill sets, and that provided search / profile data that was specific to freelancers vs job seekers (e.g. what's your hourly rate? what's your availability? are you looking for sub-contracting / referrals / collaborations?). And most importantly, that connected to the social graph and provided a good reputation system. If the people I've worked with before aren't available, the next best is people they've worked with before, and rated positively.
I wish there were a site that wasn't transaction-focused, that provided a way to find people with specific skill sets, and that provided search / profile data that was specific to freelancers vs job seekers (e.g. what's your hourly rate? what's your availability? are you looking for sub-contracting / referrals / collaborations?). And most importantly, that connected to the social graph and provided a good reputation system. If the people I've worked with before aren't available, the next best is people they've worked with before, and rated positively.
Clients changing schemas or replacing knowledgeable employees at critical stages of projects. Microsoft security issues, licensing and lack of vision with new products. Network and SAS provider outages. Technical debt at current PoE.
The incredible price of large (i.e. > 128GB) SSD's for Ultrabooks (my development machine of choice).
I got to clean up after a developer went on vacation just before setting up a nightly Java/Spring/log4j/FTP/MySQL/SQLServer process which went crazy over the fourth of July when it couldn't download a file from a business partner and went into an infinite loop.
It managed to send out 250K emails before it was terminated. Everyone's inboxes were flooded. The ops team didn't notice since their mobile devices were out of power, presumably from trying to download the emails...
It managed to send out 250K emails before it was terminated. Everyone's inboxes were flooded. The ops team didn't notice since their mobile devices were out of power, presumably from trying to download the emails...