My goal with this blog

I write about relevant changes in the way that people use the web and how startups are built to provide services and products for this ever changing wonderful thing we still know as "the web." As a former entrepreneur turned early-stage investor, my greatest hope is for this to be useful to other folks that are like me in the hopes that they can avoid some of the mistakes I've made.

Why the Cylons will first arrive through the Twitter stream

Over the holidays, I wrote myself a link shortener to take advantage of the fact that I've been paying for a vanity domain ( and to refresh my memory of HTTP redirect etiquette and the Googlebot. Watching the stats over the last month, I've been amazed at what I had erroneously perceived as amazing Twitter juice as I would easily get 1-3K clicks on any link I shared with was even mildly interesting. This morning though, I became suspicious when within 1 minute of an un-Godly early morning moment, I had received over 200 distinct clicks on my Twitter link.

A quick sampling of this first minute shed some light on just who listens that early in the morning:

2012-02-14 06:33:10184.106.83.185EventMachine HttpClient
2012-02-14 06:33:1050.56.19.192EventMachine HttpClient
2012-02-14 06:33:1150.19.36.2Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) yolinkBot
2012-02-14 06:33:1150.57.137.74EventMachine HttpClient
2012-02-14 06:33:1166.228.54.132InAGist URL Resolver (
2012-02-14 06:33:11184.72.46.156RockMeltEmbedService
2012-02-14 06:33:11173.192.79.101NING/1.0
2012-02-14 06:33:1189.151.116.53Mozilla/5.0 (compatible; TweetmemeBot/2.11; +
2012-02-14 06:33:1150.18.23.200JS-Kit URL Resolver,
2012-02-14 06:33:1146.20.47.43Mozilla/5.0 (compatible
2012-02-14 06:33:1169.63.180.247Mozilla/5.0 (compatible; FriendFeedBot/0.1; +Http://
2012-02-14 06:33:1150.18.121.54UnwindFetchor/1.0 (+
2012-02-14 06:33:12199.59.149.165Twitterbot/1.0
2012-02-14 06:33:12184.72.47.46UnwindFetchor/1.0 (+
2012-02-14 06:33:13212.238.124.233Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv: Gecko/20100405 Namoroka/3.6.3
2012-02-14 06:33:1565.52.0.205Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
2012-02-14 06:33:1750.17.86.151Summify (Summify/1.0.1; +
2012-02-14 06:33:1894.212.250.64PycURL/7.21.6
2012-02-14 06:33:2172.55.158.72Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)
2012-02-14 06:33:2250.17.85.42Java/1.6.0_07
2012-02-14 06:33:34216.52.242.14LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +
2012-02-14 06:33:42174.37.79.60Mozilla/5.0 (compatible; ScribdReader/1.0; +
2012-02-14 06:33:4350.19.205.250Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +
2012-02-14 06:34:0064.12.237.20Jakarta Commons-HttpClient/3.1

What is most fascinating about this is the sheer number of bots (software programs written by humans) wired into my feed (you can tell that these are what are visiting by the "useragent" column which usually identifies one of the main browsers but here identifies hand-written scripts). I expect that a couple of these are from Twitter itself and perhaps a handful of blessed partners (LinkedIn for instance) with access to the firehose, but for the most part, the bot melee must be comprised of software agents that have been written to at some point explicitly follow my account.

Given the sheer number of bots now authoring tweets, I wonder how long Twitter would continue to function bot-to-bot if every human being on the planet stopped tweeting instantly with software agents posting links that other software agents would dereference and then reply to. In Battlestar Galactica the Cylons were big toaster-like robots but maybe our artificial overlords will show up in a much more subtle way. Let's just hope if that is the case that they end up being less self-promotional than their human progenitors when it comes to Twitter!