How a Potato Saved Shopify's Internet(shopify.com)
shopify.com
How a Potato Saved Shopify's Internet
http://www.shopify.com/technology/3754222-how-a-potato-saved-shopifys-internet
3 comments
The problem was that all cabling had to be done underground at our place, and phone lines were the only thing we had in abundance. Coax, fibre, anything else basically meant busting up a path to the road in a major urban area. Plus it was the middle of winter which makes road work even harder.
Not every internet provider realised this immediately. We started down the path to better internet a lot of times with companies that said "no problem, we can get a wire to your building just fine" -- and then backed down weeks later when they saw our situation.
After one or two of these delays, we applied the shotgun approach, asking every single internet provider to give us an estimate all at once, to make the delays concurrent rather than successive. Every last one came back negative, or just stopped returning our calls(!). So Potato was my response to our complete lack of options.
The multiple IPs thing was an issue for some sites, but it turned out to work fine in general. If there was a site that really had a problem with it, it was also typically a low volume site, and didn't need heavy load balancing. So we had one of our proxies set up to not be balanced and just pick a single working link, and used that for those sites.
Not every internet provider realised this immediately. We started down the path to better internet a lot of times with companies that said "no problem, we can get a wire to your building just fine" -- and then backed down weeks later when they saw our situation.
After one or two of these delays, we applied the shotgun approach, asking every single internet provider to give us an estimate all at once, to make the delays concurrent rather than successive. Every last one came back negative, or just stopped returning our calls(!). So Potato was my response to our complete lack of options.
The multiple IPs thing was an issue for some sites, but it turned out to work fine in general. If there was a site that really had a problem with it, it was also typically a low volume site, and didn't need heavy load balancing. So we had one of our proxies set up to not be balanced and just pick a single working link, and used that for those sites.
How did Linux bonding not work? Did you try balance-xor and did you set miimon and arp_interval to something besides the default 0?
The problem is that DSL around here is PPP over Ethernet, rather than just plain Ethernet. So there's no MII, no ARP, no regular ethernet link bonding. It's hard to escape PPPoE for DSL around here due to phone company politics.
The type of bonding I talked about in the article was all PPP link bonding, which apparently still has a lot of issues to work out in Linux. Our two-line bond was pretty unreliable, and our attempts at a three-line bond ended up with a lot of return packets going nowhere. Didn't matter which lines we picked, either.
The type of bonding I talked about in the article was all PPP link bonding, which apparently still has a lot of issues to work out in Linux. Our two-line bond was pretty unreliable, and our attempts at a three-line bond ended up with a lot of return packets going nowhere. Didn't matter which lines we picked, either.
Brilliant, very impressed. Though I'm not sure I understand it even after reading how he did it. ;)
And you still have the issue of different connections from the same machine will show up from different IP addresses, which could certainly cause issues with some of the more paranoid sites.