If you've been a user of my Data Science Toolkit or OpenHeatMap sites, you may have noticed they've been a bit flakey recently. The back story is that I've started a new company (of which more soon) and I had to cut back on how much I was spending on my own servers. It was costing me over $1,200 a month on all the different systems I'd set up over the last three years! This was mostly because it was quicker and easier to set up a new instance than worry about jamming it onto an existing one, and I never got around to cleaning things up. I doubt there are many people who've been as lazy about this as me, but if you're looking at cutting costs, I would start by figuring out if there's servers you can merge.
By cutting out several like twitter.mailana.com and fanpageanalytics.com that were no longer being updated or heavily used, I was able to cut that in half, but I then reached two sites that have decent traffic. I started by merging DSTK and OHM onto one large server, which mostly worked but caused some hiccups. That got me down to around $400 a month, including another small instance for some legacy Mailana labs sites.
I then decided to switch to 'spot instances', Amazon's auction model for buying spare server capacity cheaply. Most of the time it's only about 12 cents an hour, a third of the normal price. I switched, but then started to experience some serious price spikes that kept taking the server down, and required manual intervention to set everything up again. At some points, the price went to $15 an hour, so there's obviously capacity limits being hit. That's very different from my experiences just a few months ago when prices seemed a lot more stable. At this point I'd never recommend spot instances for user-facing servers, the downtime seems too high. They're still a great deal for things like MapReduce backend processing.
I need 64 bit support for a lot of the frameworks DSTK relies on, so I couldn't go down to a small instance, but I did realize that micro instances were x86_64, so my next step was to try running both sites off one of those. Rather predictably, the processing requirements and lack of memory crippled the tiny instance, so the site was extremely flakey. It would have been only $14 a month though, so I spent some time trying to fix the issues, by configuring swap space for instance. My conclusion was that micros are too limited for anything but light web serving.
Today I finally bit the bullet and bought a one year reserved m1.large instance, which cost me about $900 up front and another $1,100 over the next year in costs. I also rolled in my web.mailana.com small instance into the same server, so I'm down to about $170 a month! This is around a $100 less than the unreserved cost, so I'd seriously look at reserving servers as a way of managing your costs, if you can stomach the up-front deposit.
I've also added a link to my O'Reilly books to the sites, in the hope I'll cover some of my costs:
Improve your data skills (and keep this server running!) by buying my guides:
I'm happy I've found a solution that should allow me to keep offering the DSTK and OpenHeatMap services without breaking my bank account. Apologies to all the users who have suffered through the transition, but things should be a lot more stable from now on.
Comments