I received a comment from Blake Ross on the legal changes Facebook have recently made, and I wanted to highlight that response. I recommend you look at the document itself, along with both of our commentaries on it, and make up your own mind.
I work for Facebook, but this comment should not be construed as an official company statement.
Your interpretation of this document isn't correct and frankly doesn't make much sense. If our goal were to make it difficult for startups to succeed using Facebook data, we wouldn't have launched an open API that provides access to all of our data; we wouldn't have launched the fbFund to fund startups that are built on top of this API; and we wouldn't host an annual developer conference to help startups use this API. The very future of our platform is predicated on the notion that we can help other companies improve their products by leveraging the social graph.
This crawling document exists because we've had problem where shady companies would try to scrape user information in aggregate and use it for malicious purposes. For instance, these companies would scrape page by page from http://www.facebook.com/family/ and then try to resell these bulk lists.
thanks for taking the time to comment. I might be willing to give Facebook the benefit of the doubt on this, if it wasn't for the $14,000 legal bill I just paid.
The fact is that you've made all the information on the profile pages public, complete with micro-format data to help crawlers. There's some simple technical fixes you could add to solve the specific problem you mention, starting with amending robots.txt and changing your ID formats: http://petewarden.typepad.com/searchbrowser/2010/...
You've chosen to leave all that information out in the open so you can benefit from the search traffic, and instead try to change the established rules of the web so you can selectively sue anyone you decide is a threat.
I'm really pretty bummed about this because I've been a long-time fan of Facebook, you can see me raving about your XHProf work here: http://petewarden.typepad.com/searchbrowser/2009/...
The sad fact is, your leadership has decided to change the open rules that have allowed the web to be such an interesting and innovative place for the past decade.
Facebook has always been a closed system where developers are expected to live in a culture of asking permission before doing anything, and existing at the whim of the company's management. The web I love is an open world where you are free to innovate as long as you stick to the mutually agreed rules. This is a land grab by Facebook, they've moved into the open web for the commercial benefits they'll reap, but want to change the rules so they can retain absolute control.