Scaling technology is one of the nasty problems that most people never have to concern themselves with. The average individual pulls facebook.com up in their default web browser, scans the activity stream and goes back to work. Behind the scenes is a swirling mess of technology that assembles the realish-time updates from the nearly 130 people this individual has friended. This system tackles the same problem for Facebook's nearly 800 million users and does so at least once (if not a dozen times) a day. I didn't pull this activity data out of thin air -- the usage statistics come direct from Facebook. Imagine now that Facebook is legally required to add a new process in the middle of this workflow where any reference to a unbounded list of domain names has been censored from your view of the activity stream.

Today I read that a United States Federal judge in Nevada has asked Bing, Facebook, Google, Twitter, and Yahoo! to "de-index" roughly 700 domain names. First of all we can all enjoy a good laugh at the naïve idea that Yahoo!'s search index even matters. Snickering aside, this is a very serious slippery slope.

There are the obvious impacts on freedom of speech and the hit on the simplistic worldview that the internet is an unmoderated network of nodes. Let's also consider that a blacklist of this nature creates a barrier to entry that inhibits competition. Setting aside those reasons, a purely technical assessment suggest that a list of "unindexable" domain names doesn't scale. Eventually the list of blacklisted domain names will reach into the tens of thousands. Sufficiently competent developers will always find a solution for big data scaling problems but your average developer will not.

I've had some personal experience with government mandated blacklist. In a previous job I ran into the Office of Foreign Assets Control (OFAC) at the US Department of the Treasury:

OFAC administers and enforces economic and trade sanctions based on US foreign policy and national security goals against targeted foreign countries and regimes, terrorists, international narcotics traffickers, those engaged in activities related to the proliferation of weapons of mass destruction, and other threats to the national security, foreign policy or economy of the United States

You'd think with a mission like this that your average developer working on run-of-the-mill web projects would never cross paths with an institution such as OFAC. You'd be dead wrong. In many cases developers have their interaction with OFAC abstracted away from them via services such as PayPal. In short OFAC doesn't want you to sell, trade or transfer materials (including digital materials) to a select individuals, groups, countries, etc. This means that in his heyday, Osama Bin Laden should've been prevent from buying this Americana-themed mural of George Bush because Etsy's merchant account service should've denied "transferring materials" to a known terrorist that was on the Specially Designated Nationals and Blocked Persons (SDN) list. Getting on the SDN, or any other OFAC-banned entity list, typically involves doing something incredibly stupid or having a foreign policy (i.e. Iran) that doesn't play nicely with our own. Pissing off luxury goods manufacturer Chanel would not be sufficient grounds for inclusion.

I encountered OFAC when I was trying to help a university technology transfer office build a technology licensing system that would allow them to transfer digital intellectual property to individuals who were interested in repurposing this IP in their own research. The university's export control officials insisted that we had to do our "due diligence" to ensure that we weren't transferring these materials to any entity on the SDN. It paralyzed our development efforts for a few weeks while we tried to figure out exactly what the scope of OFAC was and in particular how it applied to transferring non-fee IP to academic researchers. There's almost no literature that helps with these edge cases and it certainly acted as stop-energy in our project. Parsing the SDN list into something easily usable by a web application in real-time is also a pain-in-the-ass, but we'll leave that aside as it is a solvable problem. Regardless this one form of a blacklist causes scalability problems both in the software developer lifecycle and on the technical side.

While OFAC heavily impacts a fairly small percentage of developers. Banning domain names from being interacted with by the internet at large impacts a far larger percentage. At the end of the day it likely the ruling passed down from the Federal judge will be challenged and parts over-turned. What's a bit more troubling is that governments seek to reshape the internet with blacklists. Eventually they'll move to a whitelist-model and then internet will basically be an App Store. Except instead of company that largely intends to safeguard its users from daft implementations, we will have a significantly corrupt system that shelters you from domain names that couldn't afford to sufficiently lobby their representatives.

(Even worse is that this post is a lot of words discussing existing precedent and we haven't gotten into SOPA.)

Last Updated: 2011-12-01T01:53:02-08:00