(this is page 94 of 121)





Latest

Die ugly database, die

Last week's awesome encounter with Alan Kay left me with quite a few anecdotes; during one particular moment in the conversation, he reminded me in tone of the caught villains on Scooby Doo who always end the episode by claiming that they would have gotten away with it if it wasn't for "those meddling kids." Except of course that he was talking about the stagnant state of the software industry and the meddling kids are all of us failing to read the literature.


I am as guilty of this as the next guy which is why I decided to catch up on my list of queued papers on a recent cross country flight. Oracle HQ The theme of this bunch: the limitations and fallacies of the current web application development stack, and specifically the relational database. In particular, I wanted to point out two interesting articles in the "your architecture sucks" department:

1. In "The End of an Architectural Era" Stonebreaker et al. argue that using relational databases for most of today's web-scale information processing tasks is like throwing a buggy-whip under the hood of the car and expecting it to go fast. It's a good read (at least right up to the point which they start talking about their plan for a distributed transaction processing system) because it covers a lot of the history and design motivations behind the original RDBMS (back in the day of some very IBM sounding thing called "System R") and makes interesting points about places where web scale computing that is done on the back on an RDBMs is taking on unnecessary overhead. My only criticism of the paper is that in the proposed alternative the authors are willing to just toss away one of the most important benefits of a relational database— the ability for developers/administrators/whoever to perform ad-hoc queries on the dataset without having to drop to writing code.

2. Not having been crazy about the proposed alternative in the Stonebreaker paper, I was excited to come across "Dynamo: Amazon's Highly Available Key-value Store" by a bunch of smart dudes at Amazon, because of how well it was written, how simple and elegant the solution they propose seems to be, and most importantly, because it describes a system that is in actual production use every day. The basic concept is BerkleyDB on steroids, distributed across an arbitrary number of nodes in a fault-tolerant and self-healing design. In English: you can have persistent, reliable hashtables even in the midst of a semi-reliable infrastructure (node outages and network partitions), all at web scale. After pooh-poohing the relational database (sorry RDBMS, you are just not having a good month!), they go on to describe a system that supports an "always writeable" datastore and fails only in the rarest of cases. Best concept of the whole paper: the "vector clock" that follows each write operation around the network of nodes (a "vector clock" is a pairing of a write's version # with the node that initially takes that write) to solve conflicts across the system. I wish everything I ever did on any computing device came with its own vector clock!

Alan is right— anyone building a website that could someday live at web-scale (millions of users, billions of transactions) should be reading these papers rather than simply taking on the intellectual challenge of re-inventing some of these schemes from scratch (we were guilty of some of this at Tabblo). Fortunately we've got a lot of resources around to help the cause; if you haven't seen the always-entertaining, and frequently excellent "High Scalability" blog (source for both of these papers), you should subscribe.

Comments Permalink

Two years ago I blogged about this fabulous new writer I had found named Richard Morgan because I found him very clever in how he used his genre (sci-fi) to relax constraints in a way that created very thoughtful fiction. In his first book, Altered Carbon, Morgan sets a basic detective Whodunit in a world where people find it difficult to die due to the complete separation of consciousness from the physical body. The book is great fun and quickly turned into a trilogy that— like all good trilogies— loses steam in the middle, and picks it up again at the end.

Well this summer Morgan came back with another excellent novel— Thirteen— which is the conceptual reciprocal of Altered Carbon. This time he's put the classic Fugitive plot in a world that is populated by "variants" (of the genetic variety) or humans that were bred for specific tasks. I'm not going to ruin the cleverness of the types that he breaks down (let's just say the run to some pretty base human qualities), but will instead recommend it as a highly entertaining (and thought-provoking) twist on the age-old question of nature versus nurture.

I've personally found that where people sit on the nature-nurture question tends to track the dynamic of the old Disraeli quote about 20 year-old conservatives being heartless and 40 year-old liberals being brainless, with the schism rooted right around when people have children. Though all of us want to believe we can be whoever we think to be, there is nothing like seeing fundamental character differences expressed from such an early age in all sorts of ways to make us parents feel that the effects of nurture are sort of irrelevant and that it's mostly about the code running in your chromosomes. And so it is especially entertaining to watch Morgan play with the ideas of nature fully constraining who we are in such a complete and fatalistic way. By the end of the book, you can't help but feel that there is a little bit of variant in all of us.

He also rocks at inventing new terms— in fact so much so that I am sure this guy is turning out idiom in Scotland as we speak). My favorites from 13? Twist and Cudlip. Go read it to find out just what they mean.

Comments Permalink

So everyone is on fire about the fact that Apple seems to be bricking cracked iPhones with the new 1.1.1 software upgrade, which is a bit surprising to me. Did people not expect one of the most proprietary companies in the industry to take this path when its walled garden is under attack?

In fact, I think that the Apple of today— which is to say the Apple that launched OSX— is actually much more clue-full about how to enable something which is critical for companies in their position: key axes of user/developer participation that enrich the platform without threatening the very aspects that gives it its edge (stability and perception of control for partners). Which is why I'm left wondering why this is such a big deal, and why we can't just all focus our energy on getting Apple to support the right vector of participation for the rest of us, by which I mean enhancing their web appplication platform to the point where developers won't really care about the fact that they can't install binaries to run on the iPhone.

Below are the three things I think Apple could do to cut this gordian knot of control versus user-led innovation. In providing an "SDK" that allowed web developers access to these 3 APIs I think Apple would (as they did with the "open for business" OSX strategy) unleash a torrent of creativity while keeping the relevant bits of their platform closed for the sake of their small-minded (but critical) partners, the phone companies and the content holders.

1. Access to the camera's current "roll:" though it would be ideal to be able to control the camera itself, a great start is being able to take pictures off of the current set via Javascript/DOM scripting. For instance, I would love to be able to write:

var img = document.camera.roll[0].images[0]['thumbnail'];

for manipulating images inside of the browser for say posting as a file upload to a server. One of the coolest things about mobile imaging is the ability to completely bypass the PC; Shozu on Series 60 showed us this in a big way. I'd like to be able to send photos up not only to Flickr but also to all sorts of other cool web applications that could take advantage of basic image recognition to provide all sorts of new functionality.

2. Growl-like notification hooks, preferably tied to the SMS pipeline: this one is hard to describe for people that haven't used the fabulous Growl OSX service, but I'll try anyway. A portable computer like the iPhone (more on this below) needs to be really good at messaging which means that it needs to support as a core capability the ability to have messages "pushed" to the end device. Currently on the iPhone this comes as either phone calls or SMS messages. Why not let developers of web applications register "listeners" for particular SMSes that can the point mobile Safari to specific pages, or, at the very least, get the user's attention. When we look at mobile Meebo or mobile Facebook on the iPhone, the biggest limitation is their inability to push notifications up to the user that is not currently on their respective pages. Sure, each of these apps can send generic SMS messages to a particular phone number, but wouldn't it be so much easier on the user if these SMSes could drive specific behaviors on the phones? The idea of copying a system-wide notification like Growl goes all the way back to the Cocoa framework, so I'm sure Apple would have no problem doing it.

3. Minimal control of the local radio-based networking: Apple is already heading in this direction by giving us Bonjour (local multicast) support inside of mobile Safari in 1.1.1. Taking this further would allow application developers to find out more about the local networking environment, and where appropriate, to make connections with locally available devices for sharing information that do not require connecting to the greater Internet.

Notice that none of the three vectors of functionality I describe above touch either of the two sacred-cow business models that Apple is trying to protect: the carrier lock for using cellular networks, or the content providers' lock on the music they push to the device.

I often tell new iPhone users that the best way to always be happy with their new toy is to not think of the iPhone as cellphone-on-steroids, but to think of it as a laptop in your pocket. When looked at from this perspective features like a battery that lasts all day and a form factor that lets you surf the web in a device that weighs less than 6lbs and doesn't require a keyboard seem more like magic than say the touch screen replacing the number pad, or the constant limitations of the AT&T network. As they build out vectors for extensibility, it would behoove Apple to remember the fact that they are indeed seeding portable computers and not locked phones.

Comments Permalink


"The best way to predict the future is to invent it."
—Alan Kay

It was the moment I read that quote a few years ago that I decided that Alan Kay was one of my absolute top software heroes. Of course it also had to do with his pioneering work at Xerox PARC (read all about it in Dealers of Lightning), around object oriented programming, conceiving the first laptop, and a general adherence to both elegance and execution in software design that make him truly stand out among an already pretty impressive class of peers at PARC.

Yesterday I attended a TTI Vanguard conference on behalf of HP and had the chance not only to meet Alan, but to spend 90 minutes talking with him about software design and the state of engineering, as well as a whole bunch of related topics. I've been unwinding the conversation for most of the past twelve hours, and if I have one regret, it's that I don't have a transcript of it so that I could spend some time really digesting what he was getting across to me. He has a way of using very concise terms that carry a tremendous amount of meaning and then backing them up with references to work done by colleagues across a broad range of disciplines that is incredibly dense. If Alan himself were a Smalltalk object, I think he might need a little refactoring on the messages he sends; they are compact for sure, but depend on such a rich shared context of meaning that it can be hard for the rest of us to follow.

And speaking of compactness, I really dig his latest project. He's gotten funding from the NSF (and some other folks) to rebuild an entire personal computing system in 20,000 lines of code. And by personal computing system he doesn't mean a VM like the JVM or .NET, but in fact the "whole stack—" from the interface the user sees to the instruction set on the processor. Yeah, crazy right? When he first said it to me, I wasted the first 10 minutes trying to understand what kind of "whole system" definition he was going to use to cheat his way to the 20K LOC constraint, but it soon became clear that he was deadly serious about doing this soup-to-nuts.

Why? Because according to Alan, the edifice that is any major computing "stack" (Windows, Linux, OSX + drivers + frameworks + applications) can easily run into 100-300M lines of code— far too much for any one person to even hope to begin to understand (20K lines is by comparison, about the equivalent of a 400 page book). And if we can't understand it, there is no way that we could ever hope to begin to fix the entropy that is slowly eating these systems from the inside out, or to innovate enough in software development practices to allow software to experience its own Moore's law-like exponential increase in power per line of code written.

I'm torn over whether I think that working the sort of alchemy that Alan & team are going to have to undertake to pull off this Herculean task so that one person can truly understand the entire computing environment is going unleash the type of revolution that he hopes it will. On the one hand, I love the notion that building this type of system will usher in new tools and ways of thinking about software development that will allow us to keep teams small and productive. I've always been very proud of the small size of our team at Tabblo (especially relative to what we are able to do), and have been a little shocked since joining HP about how many other "lab managers" scoff when I tell them that our team is fewer than 10 people, following it with some statement of size about their own multi-hundred person team. It shouldn't be this way— on this front both Alan and Google are absolutely correct. Small teams make the magic happen; in fact, I can not remember the last piece of software that I was blown away by that had more than 25 people working on the core of it (one of my favorite analogies that he used while we chatted yesterday was that of the pyramids, "hunks of rubble covered with limestone," that took thousands of people years to build and could not stand up to the simple Roman arch built by 2-3 masons).

On the other hand, one of my favorite things about working in software is how well abstractions work to isolate me from the stuff that I don't care to know about. As I type this, I have a vague idea of what the CPU and GPU are doing together to make the characters appear on the screen, but most of the time I don't want to have to think about it. And if I wanted to build a new kind of word processor, I'm not sure I'd really want to think about it either. Furthermore, there is a whole generation of people just like me who probably don't have the training and experience to think that deeply about the low levels of what the OS and the hardware are doing to provide us with our computing environments— and each generation of kids coming out of school knows less and less about this arcane stuff. Today's PHP hacker wants to build the next Facebook, but he is likely to know very little about how PHP executes, how a webserver is built, or even how TCP works to send bytes all over the Internet. Should he have to worry about this if his goal is to build social applications?

Obviously, I am simplifying his argument as I think that what he would argue is that in a properly self-describing, self-bootstrapping system, it's turtles all the way down which would make it a lot easier for our PHP-hacker friend to understand the system to its core.

In fact, it is the pursuit of this elegance that is the most inspiring part of Alan's new project— and of his whole life's work. The fact that he is always looking to make things more logical and concise, to find a new kind of science (and art) in the way that most of us will build software in the future is a very good thing indeed.

And in the meanwhile, the rest of us still working on the pyramids should take a pause to think a bit about how we could move towards that arch.

[Postnote: After writing this, I went and read his NSF proposal. I'm not an expert in grant writing, but this proposal is so good that anyone looking to write any sort of pitch should read it (especially people writing business plans for risky new ventures). It's grand while remaining incredible humble in what is known and what is really hard to do. It covers the depth of experience the team has concisely, and gives a great history of "water under the bridge." But most of all, it inspires with its broad vision of what computing could be for everyone, and why it's so important that we be commissioning this type of work. I don't know who you are NSF person who approved this, but you have definitely spent my tax dollars well here!]

Comments Permalink

More platform on the brain

Hot on the heels of my wondering why I was beginning to hate the term platform, I came across Marc Andreessen's awesome post on the three kinds of things you see labeled as "Internet platforms" these days. Despite only agreeing with half of it, I would say that this is a superb post— the kind of essay that makes you think you should be paying for it. He does a great job of dissecting the entire evolution of the web-as-platform, going through three stages: 1. APIs (Flickr), 2. Plug-ins (Facebook), and 3. runtime (Salesforce), explaining at each step how the developer's work changes. On top of being a very lucid synthesis of what has been going on over the last 10 years in the web, and not being sloppy in his use of the term (platform), he also has a whole bunch of golden nuggets about how stuff gets built today that are worth reading. I can not recommend it highly enough.

Now for the part that I don't totally buy— and I'll mention this only because Marc opens the door at the end of the piece when he argues for programmers to be the ultimate arbiters of what a platform is. To the regular software engineer, platforms have always been about one thing primarily: distribution. When developers were abandoning the Mac for Windows in the early 90s it wasn't about more features and functionality, but about being able to write apps that more people could install and run. When Marc talks about the "ultimate" type of Internet platform (a runtime where developers don't have to worry about running their own infrastructure), I think he overstates the importance of that relative to the relevance of having real reach once you've written an application for a particular platform.

He does mention Facebook, but only as the 2nd type of platform, going so far as to argue for the eventual domination of platforms that also provide their own runtime over those that allow for plug-ins or just provide APIs. What is more interesting still is that Amazon EC2 and S3 get labeled as quasi-type-3 platforms (as does Akamai's new EdgeComputing runtime) because they are attempting to provide some runtime capabilities, but not much more of an API. I think he's got this wrong in that Amazon to me seems to realize that the value in a platform service can either come from distribution or from providing a core set of infrastructure services. I like to think the good folks at Amazon know that the Facebooks of the world will bring the access to distribution, but that it's going to take services like S3 and EC2 to allow developers who win to scale these services out. If you want an example, you need to look no further than Renkoo's wildly successful send-a-drink Facebook application, which is being scaled on the back of Amazon's infrastructure.

Developers are naturally prone to distrust "runtimes," mostly because in the past they've implied lock-in. As more folks try to roll out these Internet scale platforms, they would do well to remember this, and build out accordingly.

Comments Permalink