Ask HN: What "infrastructure" do you use at your startup/company?
12 comments
We're a Microsoft shop, so the sweet spot for us is a small number of fast servers. We pay for tools and server licenses, but surprisingly less than you might expect.
Hardware: Half a cage at a colo with one fast box for web/db server, and a few cheap pizza boxes for build server and misc. This used to be two boxes in a garage behind a business DSL until Twiddla went big.
Content storage & delivery: S3/Cloudfront
Heavy lifting: EC2. We spin up a half dozen boxes each night for a few hours of processing for S3stat.
Email: Google Apps (they're terrible, but free and functional)
Tools: MSDN + EmpowerISV = $400 for all the dev tools for the shop. Small Business Server for the boxes in the cage. ReSharper, CodeSmith & RedGate's SQL stuff are actually the biggest per dev cost, but they make life so much better that they're worth it.
All in, it works out to about $500/month to keep up and running. Total, across all our sites (and there are quite a few). The colo is expensive, but it's so nice. Even with the cloud stuff where it is today, I'd still put a box in a datacenter if I had it to do over again.
Hardware: Half a cage at a colo with one fast box for web/db server, and a few cheap pizza boxes for build server and misc. This used to be two boxes in a garage behind a business DSL until Twiddla went big.
Content storage & delivery: S3/Cloudfront
Heavy lifting: EC2. We spin up a half dozen boxes each night for a few hours of processing for S3stat.
Email: Google Apps (they're terrible, but free and functional)
Tools: MSDN + EmpowerISV = $400 for all the dev tools for the shop. Small Business Server for the boxes in the cage. ReSharper, CodeSmith & RedGate's SQL stuff are actually the biggest per dev cost, but they make life so much better that they're worth it.
All in, it works out to about $500/month to keep up and running. Total, across all our sites (and there are quite a few). The colo is expensive, but it's so nice. Even with the cloud stuff where it is today, I'd still put a box in a datacenter if I had it to do over again.
"until Twiddla went big"
How big to have to change?
"half dozen boxes ... for S3stat"
What kind of boxes do you use and how much do you need to process?
(I use 2 of your services and I love it)
How big to have to change?
"half dozen boxes ... for S3stat"
What kind of boxes do you use and how much do you need to process?
(I use 2 of your services and I love it)
We moved into the Colo when Twiddla got accepted into the SXSW Interactive Awards. Good thing too, because we won our category, got written up by every major tech blog simultaneously, and spent a few days dealing with 1000 pageviews per second. This was before EC2 came out with Windows instances, so the machine was pretty much on its own. Today if the same thing happened, we could spin up a dozen EC2 boxes to deal with the spike.
That one box handled the load remarkably well, especially considering that I hadn't done much backend optimization at all, and that we were still persisting every single message to the database. It went a long way towards solidifying my faith in the MS stack, and making me scratch my head whenever I see somebody's blog fall over from a minor HackerNewsing.
As to S3stat, it runs on EC2 m1.small instances. Towards the end of the month, when each bucket takes longer to process, we'll step up the number of workers we fire up each night, and occasionally pull in a few c1.medium instances to speed things along.
It's a fun architecture, and I've been meaning to write it up for a while now. It uses six distinct AWS offerings to do its thing.
That one box handled the load remarkably well, especially considering that I hadn't done much backend optimization at all, and that we were still persisting every single message to the database. It went a long way towards solidifying my faith in the MS stack, and making me scratch my head whenever I see somebody's blog fall over from a minor HackerNewsing.
As to S3stat, it runs on EC2 m1.small instances. Towards the end of the month, when each bucket takes longer to process, we'll step up the number of workers we fire up each night, and occasionally pull in a few c1.medium instances to speed things along.
It's a fun architecture, and I've been meaning to write it up for a while now. It uses six distinct AWS offerings to do its thing.
Business: Social games for gamers (http://www.ea2d.com)
Languages: Java backend, AS3 frontend, Python + Ruby misc
Note that we're very new (and hiring!), so much of this is still being built out.
Software lifecycle: git for SCM (GitHub as central repo); ant for builds; ivy for dependency management; Bamboo for build automation; Crucible for code review; JIRA for issue tracking; Greenhopper for sprint planning; Vagrant for containerized development environments
Systems infrastructure: AWS (EC2 w/ELB+ASG and S3 for now, more services to come); nginx-fronted Tomcat for JVM; Cassandra for persistent DB; chef for configuration management (currently chef-solo, but evaluating Opscode Platform); PoolParty+Fabric for one-click deploy server fleets; Nagios + Pingdom for monitoring; Mixpanel + Kontagent for analytics; Dynect (likely) for DNS, possibly for GSLB; Akamai (likely) for CDN, possibly for GSLB
Misc: PagerDuty for on-call stuff; Google Apps for email, shared docs, etc; Confluence for wiki; OpenFire (Jabber) for chat
Languages: Java backend, AS3 frontend, Python + Ruby misc
Note that we're very new (and hiring!), so much of this is still being built out.
Software lifecycle: git for SCM (GitHub as central repo); ant for builds; ivy for dependency management; Bamboo for build automation; Crucible for code review; JIRA for issue tracking; Greenhopper for sprint planning; Vagrant for containerized development environments
Systems infrastructure: AWS (EC2 w/ELB+ASG and S3 for now, more services to come); nginx-fronted Tomcat for JVM; Cassandra for persistent DB; chef for configuration management (currently chef-solo, but evaluating Opscode Platform); PoolParty+Fabric for one-click deploy server fleets; Nagios + Pingdom for monitoring; Mixpanel + Kontagent for analytics; Dynect (likely) for DNS, possibly for GSLB; Akamai (likely) for CDN, possibly for GSLB
Misc: PagerDuty for on-call stuff; Google Apps for email, shared docs, etc; Confluence for wiki; OpenFire (Jabber) for chat
Sounds neat, but is a subsidiary of EA really considered a "start up"?
I definitely wouldn't consider EA2D a startup, but fortunately the OP left it open ("startup/company").
That being said, I do think we (EA2D) have a lot in common with later-stage startups. Three big factors are our autonomy (near-complete), self-sufficiency (product, dev, marketing, ops, etc. is internal to our studio) and size (just 27 people across all disciplines). I bet our stack more closely resembles that of a startup than that of an enterprise.
That being said, I do think we (EA2D) have a lot in common with later-stage startups. Three big factors are our autonomy (near-complete), self-sufficiency (product, dev, marketing, ops, etc. is internal to our studio) and size (just 27 people across all disciplines). I bet our stack more closely resembles that of a startup than that of an enterprise.
* Code Management/Bug tracking: http://unfuddle.com - I love these guys. The bug tracking is fairly solid and they serve as an excellent source code host
* Hardware : Amazon EC2. We also use cloudfront (CDN) and a couple of other services. It's worked out amazingly well really.
* Accounting: My wife (PhD in Accounting). She uses Quickbooks I think. We use ADP for our payroll for quite awhile and that was nightmare. They were late with W2's and failed to pay the proper taxes they said they were paying. Just awful. My wife does payroll now.
* Email: We use authsmtp as our gateway. They've been very solid. Very few spam box issues. We also use mailchimp to manage our actual contact lists.
* Analytics: we use a combination of homegrown tools, google analytics, UserFly and CrazyEgg for usability and conversion analysis.
* Email: we use Google Apps. That's also worked out extremely well for us. No real issues and we rely on Google Docs quite a bit.
* Other services: We use Twilio to handle phone-tracking (it's an integral part of our product). We used UserVoice at one point and that was excellent.
I'll update with other stuff as I think about it.
* Hardware : Amazon EC2. We also use cloudfront (CDN) and a couple of other services. It's worked out amazingly well really.
* Accounting: My wife (PhD in Accounting). She uses Quickbooks I think. We use ADP for our payroll for quite awhile and that was nightmare. They were late with W2's and failed to pay the proper taxes they said they were paying. Just awful. My wife does payroll now.
* Email: We use authsmtp as our gateway. They've been very solid. Very few spam box issues. We also use mailchimp to manage our actual contact lists.
* Analytics: we use a combination of homegrown tools, google analytics, UserFly and CrazyEgg for usability and conversion analysis.
* Email: we use Google Apps. That's also worked out extremely well for us. No real issues and we rely on Google Docs quite a bit.
* Other services: We use Twilio to handle phone-tracking (it's an integral part of our product). We used UserVoice at one point and that was excellent.
I'll update with other stuff as I think about it.
Billing: Merchant accounts and Authorizenet's gateway, plus PayPal
Accounting: For analysis/forecasting, a custom programmed dashboard. For bookkeeping, Quickbooks.
Newsletters: MailChimp
Everythign else e-mail: Rackspace Email
Banking: The local bank
IT: If hardware breaks, Softlayer, Linode and Amazon take care of that. Anything else is my job.
Accounting: For analysis/forecasting, a custom programmed dashboard. For bookkeeping, Quickbooks.
Newsletters: MailChimp
Everythign else e-mail: Rackspace Email
Banking: The local bank
IT: If hardware breaks, Softlayer, Linode and Amazon take care of that. Anything else is my job.
Business : Cirrii.com
Email: Google Apps works fine for everything we need.
Code Management : Local Redmine install. Mutli repo and multi project is just what we need for a bunch or side projects or prototypes. Github for personal stuff. The issue tracking is not where it needs to be yet for an production project yet but getting there.
Deployments: Fabric and Chef-Solo. Internally a lot of virtualbox vm's.
EC2, Rackspace and hosteurope.de with : Haproxy lb's, powerDNS homegrown GeoDNS, AWS Cloudfront CDN.
HTH
Email: Google Apps works fine for everything we need.
Code Management : Local Redmine install. Mutli repo and multi project is just what we need for a bunch or side projects or prototypes. Github for personal stuff. The issue tracking is not where it needs to be yet for an production project yet but getting there.
Deployments: Fabric and Chef-Solo. Internally a lot of virtualbox vm's.
EC2, Rackspace and hosteurope.de with : Haproxy lb's, powerDNS homegrown GeoDNS, AWS Cloudfront CDN.
HTH
[deleted]
I use Codebase to for source code management (mercurial) and for project ticketing.
I use Google Wave for most in-project communications.
Python/Django + jquery for dev.
WebFaction for web hosting <--- Opportunity here, there's definitely an opportunity to make simpler hosting for Django site. I believe there are people working on it already, though.
I use Google Wave for most in-project communications.
Python/Django + jquery for dev.
WebFaction for web hosting <--- Opportunity here, there's definitely an opportunity to make simpler hosting for Django site. I believe there are people working on it already, though.
Servers:
Linode, Slicehost, EC2
Project Management/Version Control: Unfuddle, Github
Billing and customer tracking: freshbooks, highrise
Project Management/Version Control: Unfuddle, Github
Billing and customer tracking: freshbooks, highrise
I saw your comment and though, damn they use Exact same tools as us. Then I saw your username. :)
Billing: Recurly.
Accounting: Quickbooks
Provisioning: Home grown on EC2/GoGrid/Rackspace
Banking: Local bank
Newsletter: AWeber
IT: Depends. Most I handle, but some of the mundane is outsourced.
SCM: HG @ Codebase
If I had to do it over again, I wouldn't obsess as much as I did over the infrastructure. Check and re-check your backup & disaster recovery plan.
Accounting: Quickbooks
Provisioning: Home grown on EC2/GoGrid/Rackspace
Banking: Local bank
Newsletter: AWeber
IT: Depends. Most I handle, but some of the mundane is outsourced.
SCM: HG @ Codebase
If I had to do it over again, I wouldn't obsess as much as I did over the infrastructure. Check and re-check your backup & disaster recovery plan.
Is Recurly worth it? One of their team just sent me an email suggesting I check 'em out.
I would never, ever trust those guys again.
They pulled a massive bait-and-switch on their first users. Overnight, without notice, they changed their long time published pricing from pay-as-you-go (fee per transaction) to high monthly fees. For many, the cost was orders of magnitude higher. They arbitrarily decided the API would only remain accessible if you bought the highest priced plan.
It'd be fine if they gave notice of the change or grandfathered people in, but at least initially, they didn't. They just wrote condescending, insulting blog posts telling us all how we should appreciate their rate hikes. If you weren't happy with (or able to pay) the higher prices and had already integrated Recurly with your site, you were essentially held hostage. They have all your customer billing information, not you. If you want to keep billing customers, your revenue!, while you strip out Recurly and implement some other payment system and get all your users to sign up again, you've gotta pay to keep your account open in the meantime.
Eventually they figured out that screwing with their users like this wasn't a good idea after all, backtracked and came up with a new pricing scheme more like the original, but it's really too late at that point. Revenue is what makes a business. You don't play games with who you trust that to. And nobody should ever trust it to Recurly, since they're more than willing to play games with you.
If you want to outsource subscription management, check out Spreedly and Chargify. They've proven themselves more trustworthy than Recurly by far. Really, I recommend doing it yourself though, it's not that bad. PayPal subscriptions and IPN can take you a long way with minimal coding.
They pulled a massive bait-and-switch on their first users. Overnight, without notice, they changed their long time published pricing from pay-as-you-go (fee per transaction) to high monthly fees. For many, the cost was orders of magnitude higher. They arbitrarily decided the API would only remain accessible if you bought the highest priced plan.
It'd be fine if they gave notice of the change or grandfathered people in, but at least initially, they didn't. They just wrote condescending, insulting blog posts telling us all how we should appreciate their rate hikes. If you weren't happy with (or able to pay) the higher prices and had already integrated Recurly with your site, you were essentially held hostage. They have all your customer billing information, not you. If you want to keep billing customers, your revenue!, while you strip out Recurly and implement some other payment system and get all your users to sign up again, you've gotta pay to keep your account open in the meantime.
Eventually they figured out that screwing with their users like this wasn't a good idea after all, backtracked and came up with a new pricing scheme more like the original, but it's really too late at that point. Revenue is what makes a business. You don't play games with who you trust that to. And nobody should ever trust it to Recurly, since they're more than willing to play games with you.
If you want to outsource subscription management, check out Spreedly and Chargify. They've proven themselves more trustworthy than Recurly by far. Really, I recommend doing it yourself though, it's not that bad. PayPal subscriptions and IPN can take you a long way with minimal coding.
@dangrossman I'd like to address your comment:
Allow me to introduce myself. I'm Dan Burkhart, the President of Recurly.
There was a very early pricing change, which was admittedly not handled well. We subsequently reached out to all of our customers twice via email surveys, and personally via telephone to all that indicated that they were willing to speak with us. Our latest pricing went over extremely well, AND we did grandfather any customer to prior pricing if they wanted. (A relatively small percentage chose to be grandfathered, since our new pricing has been favorable for the broad majority).
You can see that our new pricing posted on July 1 went over extremely well. (There was only one comment on the blog post). http://blog.recurly.com/2010/07/new-recurly-pricing/
Our customer base continues to grow rapidly, and since we raised money, we have hired full time customer support. We treat our customers extremely well. I suggest you speak to Retargeter, SlideShare, KissMetrics, Unbounce, WordTracker, SproutSocial, GroupSpaces, GOOD.is, and many others to hear from them how they have been treated, and whether they would recommend Recurly.
One last point. Understand that Recurly stores credit cards (in a PCI compliant) vault. The benefit to our customers is 1) that as their business scales, we uniquely offer the ability for our customers to easily transition from one payment gateway to another without ANY business interruption. As your business grows, you will surely be able to negotiate more favorable rates with your gateway, and we give you the leverage to do so. 2) We are also able to better address credit card declines and remediate the process by resubmitting additional information to satisfy the reason for the decline. This results in lower customer churn and for businesses doing any kind of transaction volume, this is a very big benefit. (Compare Recurly costs to services like Litle & Co who offer the same service for big $$).
Cheers.
Dan
Allow me to introduce myself. I'm Dan Burkhart, the President of Recurly.
There was a very early pricing change, which was admittedly not handled well. We subsequently reached out to all of our customers twice via email surveys, and personally via telephone to all that indicated that they were willing to speak with us. Our latest pricing went over extremely well, AND we did grandfather any customer to prior pricing if they wanted. (A relatively small percentage chose to be grandfathered, since our new pricing has been favorable for the broad majority).
You can see that our new pricing posted on July 1 went over extremely well. (There was only one comment on the blog post). http://blog.recurly.com/2010/07/new-recurly-pricing/
Our customer base continues to grow rapidly, and since we raised money, we have hired full time customer support. We treat our customers extremely well. I suggest you speak to Retargeter, SlideShare, KissMetrics, Unbounce, WordTracker, SproutSocial, GroupSpaces, GOOD.is, and many others to hear from them how they have been treated, and whether they would recommend Recurly.
One last point. Understand that Recurly stores credit cards (in a PCI compliant) vault. The benefit to our customers is 1) that as their business scales, we uniquely offer the ability for our customers to easily transition from one payment gateway to another without ANY business interruption. As your business grows, you will surely be able to negotiate more favorable rates with your gateway, and we give you the leverage to do so. 2) We are also able to better address credit card declines and remediate the process by resubmitting additional information to satisfy the reason for the decline. This results in lower customer churn and for businesses doing any kind of transaction volume, this is a very big benefit. (Compare Recurly costs to services like Litle & Co who offer the same service for big $$).
Cheers.
Dan
I've been happy with them. The API is simple to use. Isaac is a great down to earth guy. Handled the pricing uproar well earlier this year well.
I'd also suggest Chargify and Braintree.
I'd also suggest Chargify and Braintree.
WordPress (and a lot of plugins)
DropBox
Remember the Milk
Basecamp
Linode + Virtualmin/Webmin for hosting
Paypal
G Adwords
G Analytics
G Apps for emails and calendar
Actually I forgot Zoho invoice, Coda, Subversion and OpenOffice
When we started, we had a single Linux server and some laptops. Over time we added a VMWare server for testing and a VPS. Now our main file server is a Mac (as we didn't want to run Windows servers and needed PGP Whole Disk Encryption), we have a test VM Server and a couple of VPSs.
Code - Github
Billing - Recruly
Accounting - Freshbooks
Newsletter - Mailchimp
Email Delivery - Sendgrid
What tools/services do you currently use? If you had to do it all over again, what would you do differently?