My goal with this blog

I write about relevant changes in the way that people use the web and how startups are built to provide services and products for this ever changing wonderful thing we still know as "the web." As a former entrepreneur turned early-stage investor, my greatest hope is for this to be useful to other folks that are like me in the hopes that they can avoid some of the mistakes I've made.

Getting all Pollyana over Amazon's web services

Much has been made of Jeff Bezos's talk this week at some MIT conference on web services, and specifically S3 (the storage service) and E2C (CPU service). Wade Roush from MIT Tech Review interviewed him and everyone who saw the interview felt compelled to announce to the world that all of us little startups need to move all of our backend processing and storage to Amazon's cloud. Even the ever present canary in the coal mine, Nick Carr, seemed relatively positive about it.

This just doesn't pass the smell test for me. I know that other little companies that I respect like SmugMug have raved, especially about S3 the storage service, but I have very real concerns about the viability and scalability of this approach and I certainly feel like any commercial companies that put a key piece of infrastructure under S3 are a bit like the test pilots who fly the monster jets that have previously only "flown" in simulation. [Disclosure: for big compute applications (which I am not familiar with) E2C may actually make sense so the rest of this post is really about S3, and any other infrastructure tier services like SQS).

First, anyone who has worked on a big website that has grown organically over a period of years knows just how rickety and duck-tapey the whole infrastructure can get. This has got to be especially true for Amazon which got started before we learned about how to scale on the web. And it's actually a fine thing for most sites– because just like the pilot who still flies mail routes on a DC-3 with bolts popping out of it, most sysadmins/developers at these big sites learn to put their hand on the fuselage and just know where the bolts might be about to pop.

However, I'm not sure that this institutional knowledge will export well across SLAs and web service APIs. More importantly, at the end of the day Amazon is a retailer that makes its money from selling stuff which necessarily means that all of its best and brightest are probably working on that side of the fence. Even if there is a small and talented group pushing the infrastructure web services initiative, they've still got to deal with 11 years of bolts that have have come loose and been tightened under the pressure of having to keep Amazon's site and sales up.

Second, people seemed to make a big deal out of the fact that Amazon has built infrastructure that is often at less than 10% utilization which must mean that they have plenty of capacity for third parties. But they certainly didn't build a system intending it to stay at 10% utilization– they've instead built all of this infrastructure to handle peak load at certain times of the year (say Christmas). What happens if as a consumer of AWS, your business cycle also peaks during the holidays (as most output-based businesses do including ours). When it's about serving your files or that gif of the Davinci Code, how is the decision made? And even if it's evenly distributed, do you really want your traffic to be mushed in the stream with the 35M people trying to buy last minute Christmas presents going through the app is best tailored to use these services? How many optimizations have the developers at Amazon put in that you just won't know about when the load spikes on that compute grid?

I don't want to sound like one of those off-the-grid guys who wants to produce his own electricity– however, I'm not sure I'd want to buy my electricity from Costco just because they have big generators at the back of their store that sit idle right until the middle of summer.

Finally, what really gets me about the whole proposition is not that it's a bad idea– it certainly makes a lot of sense to have services provided by infrastructure providers up in the cloud– but that it seems misguided in design and then promoted under false assumptions. If I were Amazon and wanting to get hardcore about the web services space, I would look at my assets and start from either my payment processing capabilities or my huge content archive of product and sales data. When Amazon launched its web service API (the one that let you get products and prices) a few years ago, I thought for sure this is where they were going, and so it comes as a surprise to see them now recasting themselves as an infrastructure utility computing player. Even extending into identity management (as Yahoo did this week) would make more sense from a core competency perspective.

And to the folks that argue that most of us building software are writing 5-10% of software that is truly value-add and 90-95% that is just rote, I would absolutely agree. However the right solution for this problem is open source, not managed services. It really doesn't take that much to go out and buy a rack at a tier-1 datacenter, shove some whitebox 1Us into it and install Linux/Apache/MySQL/YourFavoriteAppStack on it. And if you make a half decent set of stack choices, you'll be able to scale in much the same way that AWS will let you. The challenges come when you have to start getting deep into your infrastructure to understand how your particular application's workload puts stress on it, how you can rework pieces of both the app and the infrastructure to meet your scaling/costs needs.