The first step to increasing your site’s visibility on the top search engines such as Google, Yahoo! and MSN is to help their respective robots crawl and index your site.

To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file. Conversely and importantly, webmasters can also notify the search engines about the existence and importance of pages with a sitemap.xml file. (Both files are placed in the root directory of the domain.)

Fortunately for the webmaster, the major search engines provide various tools to help manage both Sitemap and Robot files.

To gain an understanding of both ‘protocols’, I’ll discuss them briefly below.

Sitemaps (Inclusion Protocol)

The Sitemaps protocol allows a webmaster to inform search engines about URLs on a website that are available for crawling. A Sitemap is an XML file that lists the URLs for a site. It allows webmasters to include additional information about each URL: when it was last updated, how often it changes, and how important it is in relation to other URLs in the site. This allows search engines to crawl the site more intelligently. Sitemaps are a URL inclusion protocol and complement robots.txt, a URL exclusion protocol.

The webmaster can generate a Sitemap containing all accessible URLs on the site and submit it to search engines. Since Google, MSN, Yahoo!, and Ask use the same protocol now, having a Sitemap would let the biggest search engines have the updated pages information.

Sitemaps supplement and do not replace the existing crawl-based mechanisms that search engines already use to discover URLs. By submitting Sitemaps to a search engine, a webmaster is only helping that engine’s crawlers to do a better job of crawling their site(s). Using this protocol does not guarantee that web pages will be included in search indexes, nor does it influence the way that pages are ranked in search results.

The following is a cut-down version of the sitemap.xml for this website. WordPress, via a plugin, automatically updates this file each time a new post or page is written.

<urlset xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
<url>
<loc>http://www.simonwhatley.co.uk/</loc>
<lastmod>2008-10-08T14:50:16+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>
http://www.simonwhatley.co.uk/big-city-little-people
</loc>
<lastmod>2008-10-08T14:50:16+00:00</lastmod>
<changefreq>monthly</changefreq>
<priority>0.1</priority>
</url>
</urlset>

More information about sitemaps can be found on the Sitemaps.org website.

Robots (Exclusion Protocol)

The robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorise and archive web sites. The standard complements Sitemaps, a robot inclusion standard for websites.

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorisation of the site as a whole.

The protocol, however, is purely advisory. It relies on the cooperation of the web robot, so that marking an area of a site out of bounds with robots.txt does not guarantee privacy. Some web site administrators have tried to use the robots file to make private parts of a website invisible to the rest of the world, but the file is necessarily publicly available and its content is easily checked by anyone with a web browser.

For example, the following tells all crawlers not to enter four directories of a website:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/

Exclusion can also be achieved on a page-level basis using a Meta-tag. This is a tag that would be placed in the HTML head of of a web page. The robots attribute controls whether search engine spiders are allowed to index a page, or not, and whether they should follow links from a page, or not.

A common example could be as follows:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" 
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> 
<html xmlns="http://www.w3.org/1999/xhtml" dir="ltr" lang="en-GB" xml:lang="en"> 
 <head profile="http://gmpg.org/xfn/11"> 
	<title>Simon Whatley</title>
	<meta http-equiv="robots" content="index,follow" />
</head>
<body>
</body>
</html>

A word of caution though, Meta tags are not the best option to prevent search engines from indexing content of your website.

More information about Robots.txt files can be found on the Robotstxt.org website.

Webmaster Tools

The top 3 search providers all have their own webmaster tools admin interface. The Google offering is the most advanced, but it’s good practice to use and submit information to all three.

Links to their services are provided below:

Ask doesn’t have an interface. However, you can still ping their Submission Service using the URL http://submissions.ask.com/ping?sitemap= in conjunction with your sitemap URL.

Further Information

With the advent Google Chrome there has been a lot of media coverage regarding the browser’s uptake and how it will compete with Internet Explorer, Firefox and Safari. This is where the User Agent becomes most valuable. It can be used in analytics software to determine the browser share and consequently aid the development of the website.

But what is a User Agent? A User Agent is the client application used with a particular network protocol; the phrase is most commonly used in reference to those which access the Web. Web user agents range from web browsers and e-mail clients to search engine crawlers (spiders), as well as mobile phones, screen readers and braille browsers used by people with disabilities. When Internet users visit a web site, a text string is generally sent to identify the user agent to the server. This forms part of the HTTP request, prefixed with user-agent: and typically includes information such as the application name, version, host operating system, and language. Bots, such as web crawlers, often also include a URL and/or e-mail address so that the webmaster can contact the operator of the bot.

By simply typing about:version into Chrome’s address bar you will be presented with the following information:

Google Chrome
0.2.149.29 (1798)
Official Build
Google Inc.
Copyright © 2006-2008 Google Inc. All Rights Reserved.
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13

As you can see Chrome’s version information provides limited detail about the browser. The last line is the important one. It is the HTTP User-Agent header:

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13.

If you know the RFC 2616 specification on the HyperText Transfer Protocol — which incidentally, I gladly don’t — you would know that the User Agent, or more formally, product token, should be short and to the point:

Product tokens SHOULD be short and to the point. They MUST NOT be used for advertising or other non-essential information. Although any token character MAY appear in a product-version, this token SHOULD only be used for a version identifier (i.e., successive versions of the same product SHOULD only differ in the product-version portion of the product value).

Clearly this isn’t the case! One of Google’s reason’s behind creating the Chrome browser was to start afresh. It would have therefore been truely amazing if they had made the string simply Chrome/0.2.149.27.

Unfortunately, browser sniffing makes an ever-growing UA string the path of least resistance for browser vendors.

So, what does Chrome’s User Agent string actually mean:

  • Mozilla/ - This means that browser has the kind of capabilities that Netscape 1.1 had compared to Mosaic and Lynx.
  • 5.0 - This means that the browser engine is from the post-Browser War Web Standards era as opposed to being from the Browser War era.
  • (Windows; - This means that general windowing system flavor the browser runs on is Windows (as opposed to, for example, Apple and X11).
  • U; - This means that the browser has at least the level of cryptographic capability / encryption strength that U.S. versions of browsers had in the late 1990s.
  • Windows NT 6.0; - This indicates the operating system the browser is running on. In this instance, the browser is running on Vista.
  • en-US) - This indicates the user interface language of the browser (U.S. English in this case). This may be used to choose between different content languages even though HTTP has a different header for that purpose.
  • AppleWebKit/ - This indicates that the engine of the browser is WebKit as opposed to being Gecko. Developers should not do user agent sniffing as a rule, but if they still do, this is what they should be sniffing.
  • 525.13 - This is the WebKit version from which Chrome branched its copy. Site admins could use this to detect old versions with known bugs.
  • (KHTML, like Gecko) - This introduces the substring Gecko into the UA string while pointing out to human readers that Webkit was forked from KHTML. Without this substring, Chrome might be put in the same category as IE and Netscape 4.
  • Chrome/ - This string identifies the browser as actually Google Chrome.
  • 0.2.149.27 - This is the Chrome version. This could be used to detect old versions with known bugs.
  • Safari/ - This means that the browser is like Safari as opposed to being like Firefox.
  • 525.13 - This just repeats the WebKit version in order to have some version but not the irrelevant Safari.app version.

On 1st 2nd September 2008 Google launched a new opensource browser project named Chrome.

UPDATE: You can download the beta project from the Google Chrome website.

Instead of me talking you through the project, Google and Scott McCloud have put together a cool little cartoon.

However, as a brief summary:

  • Google Chrome is Google’s open source browser project.
  • The browser will use the popular Webkit HTML rendering engine used in Safari and Adobe AIR.
  • The browser will include a brand new JavaScript Virtual Machine called V8.
  • The browser will include Gears to allow developers to enhance the user experience.
  • Google Chrome will use special tabs, like more traditional browsers, but set above the address and menu bar.
  • Each browser tab will run on its own process. If one tab fails for some reason, the whole browser will not need to be restarted, losing valuable work or tabs. This is similar to functionality found in Internet Explorer 8.
  • The browser has an address bar which includes a more intuitive auto-completion feature called ‘omnibox’. It is said to be less ‘irritating’ than current auto-complete/suggest functionality common to Firefox 3 or Google Suggest.
  • As a default homepage Chrome presents you with a kind of speed dial feature, similar to the one found in Opera.
  • Chrome has a privacy mode, which allows you to create an incognito window and nothing that occurs in that window is ever logged on your computer. Again, this is similar to functionality found in Internet Explorer 8.
  • Web apps can be launched in their own browser window without address bar and toolbar, much like Mozilla’s Prism project.
  • To fight malware and phishing attempts, Chrome constantly downloads lists of harmful sites.

(Click on the images to see a larger view)

Google Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser ProjectGoogle Chrome - Opensource Browser Project


You can find out more information from Google’s blog post on the subject.

Adobe is slowly but surely increasing its online presence with the addition of four web-based tools; Buzzword, Share, Photoshop Express and Brio. Although these four applications currently function independently from each other, they have very similar user interfaces and with a small amount of work, these tools could be tied together, offering a new and unique online suite worth noticing.

So why the big deal?

Software is moving from being packaged, where you develop for a particular operating system and put it in a box, to being developed and distributed over the internet and being designed to run across operating systems. That’s where all the innovation has moved to. Software isn’t as OS-specific anymore, it’s moving to rich internet applications. It’s a sea change in how software in general is being built.

Adobe’s Kevin Lynch on AIR’s Open-Source Road to the Desktop.

What is Adobe offering?

Adobe hasn’t developed a cohesive online suite like Google Docs and Zoho, but they are developing a series of applications that will, given time, challenge for position.

Buzzword

Buzzword, originally developed by Virtual Ubiquity, is a web-based, highly collaborative word processor built on Adobe’s ubiquitous Flash platform. This online editor really excels in “what you see is what you print” (WYSIWYP) functionality. Unlike the slightly clunky Google Docs and Zoho Writer, using Flash allows Buzzword to handle page layout in a way that is not possible with HTML. Buzzword also offers online collaboration via its sharing feature, which, like Google Docs, allows users to invite others to read, edit or comment on documents in realtime. Buzzword stores files online so that they are available in a single repository for document collaboration. Work is underway to support Adobe AIR to allow for offline work.

Adobe BuzzwordAdobe BuzzwordAdobe BuzzwordAdobe Buzzword

(click on the images for more detail)

You can find more information about Buzzword on the Adobe Labs website.

Share

Share is a free web-based service that makes it easy to share, publish and organize your important documents. Each document you upload to your Share account is assigned a unique website address. To share a document with someone, select the document you want to share, enter the person’s email address and an optional message, and set whether the files will be publicly accessible or restricted only to the recipients. Recipients will get an email with a link they can click on to download the document. You can also link to your documents, or embed flash previews on your own website, blog or wiki. This concept is not new, with Scribd and Issuu being an alternatives.

Adobe ShareAdobe ShareAdobe ShareAdobe Share

(click on the images for more detail)

You can find more information about Share on the Adobe Labs website.

Photoshop Express

Adobe Photoshop Express is an online Rich Internet Application (RIA) where you can polish, sort, store, and show off up to 2GB of photos. Furthermore, you can crop, rotate, smudge, tweak, twirl, pinch, correct — or any combination you like — the images. The tool isn’t like its more powerful offline sister, it is more like the photo editing website Picnik. What’s interesting about the Adobe offering, is the fact that Photoshop Express comes with 2GB of free storage for your photos, which makes it less of just an online tool, and more of an online service. The 2GB trumps Picassa’s current 1GB.

Adobe Photoshop ExpressAdobe Photoshop ExpressAdobe Photoshop ExpressAdobe Photoshop Express

(click on the images for more detail)

You can find more information about Photoshop Express on the Adobe Labs website.

Brio

Brio, currently in Beta, is a personal web-conferenceing service that enables you to instantly communicate and collaborate using your own online meeting room. Brio offers screen-sharing, full multi-party video, VoIP, teleconferencing, whiteboarding, chat and shared notes; all via the browser.

To start a meeting, just go to your meeting room and invite others to join you at the same URL. As the host, you will need to download a small Brio add-in in order to share your screen. Meeting attendees will not need to download any software unless they will also be sharing their screen. There is no need to schedule meetings in advance.

Adobe BrioAdobe BrioAdobe BrioAdobe Brio

(click on the images for more detail)

You can find more information about Brio on the Adobe Labs website.

Integration and Offline Access

Although each of these tools work independently of one another, using different sign-ons, it is a very real possibility that Adobe will adopt a similar route to that of Google, Microsoft, Yahoo and Zoho and integrate their online products into a single cohesive unit with one sign-on; the Adobe ID.

Plans are already afoot to integrate the Buzzword and Share tools, both of which sit naturally together. What would be more interesting would be the integration of Photoshop Express with these tools so that you can, for example, edit images embedded in a Buzzword document.

The Future

Adobe has stiff competion from the offline, desktop applications. This is where AIR enters the picture. Adobe said, as far back as September 2007, that they would create a version of Buzzword in AIR. This has yet to be envisaged, but the rumblings from Adobe suggest that this development is still in the works. Bringing Buzzword to the desktop would be an extremely significant step, making it a very real alternative to desktop word processors.

All that is required now for Adobe is to implement a spreadsheet and presentation application. Whether they buy in these tools, or use their existing skill set is the question. On current form, and if the acquisition of Virtual Ubiquity and its Buzzword product is an indication, Adobe are likely to be keeping a keen eye on existing technologies being developed by third parties. For example SlideRocket is a viable contender for presentations - built in Flash and with an AIR client; the user interface even looks similar to the above products. Or there is blist for spreadsheets that again is built on Flex/Flash technology.

Keep an eye on Adobe Labs for their latest developments. You will notice developments in areas such as RSS with myFeedz, colour theming with Kuler, and a competitor to Microsoft’s Sharepoint and Google’s Sites called JamJar.

Data Portability

Social network portability is one of several user-interface ideas and suggestions in the area of data-portability. As users, our identity, photos, videos and other forms of personal data should be discoverable by, and shared between our chosen (and trusted) tools or vendors. When you join a new site, you should be able to import or preferably subscribe to your profile information and your social network from any existing profile of yours. We need a DHCP for Identity. A distributed File System for data. The technologies already exist, we simply need a complete reference design to put the pieces together. This problem is solved by a number existing technologies and initiatives: Microformats, OpenID, OAuth, RDF, RSS, OPML and APML.

Data Portability Technologies

Data Portabilities mission is to put all existing technologies and initiatives in context to create a reference design for end-to-end Data Portability. To promote that design to the developer, vendor and end-user community.

This post serves a brief primer to each of these technologies.

Microformats

Designed for humans first and machines second, microformats are a set of simple, open data formats built upon existing and widely adopted standards. Instead of throwing away what works today, microformats intend to solve simpler problems first by adapting to current behaviours and usage patterns (e.g. XHTML, blogging).

Examples include:

People and Organizations
hCard
Calendars and Events
hCalendar
Opinions, Ratings and Reviews
VoteLinks, hReview
Social Networks
XFN
Licenses:
rel-license
Tags, Keywords, Categories
rel-tag
Lists and Outlines
XOXO

If you use Flickr, Technorati, Upcoming, Last.fm, Twitter, Cork’d or any number of other services, you can conceivably share data between the different services providers automatically.

More details can be found on the microformats website.

OpenID

OpenID is an open, decentralized framework for user-centric digital identity. OpenID takes advantage of already existing internet technology (URI, HTTP, SSL, Diffie-Hellman) and realizes that people are already creating identities for themselves whether it be at their blog, photostream, profile page, etc. With OpenID you can easily transform one of these existing URIs into an account which can be used at sites which support OpenID logins.

In other words, OpenID allows users to login using shared credentials across different services. It also allows users to decide what information to share between services. For example, you can allow the use of your address on one service, but not another. You can think of OpenID as an extension to the single sign on used by Google or Yahoo! to access their various services.

More details can be found on the OpenID website.

OAuth

The OAuth protocol is less about authentication, which is the realm of OpenID, but rather authorisation. OAuth is an open protocol to allow secure API authorisation in a simple and standard method from desktop and web applications. For consumer developers, OAuth is a method to publish and interact with protected data. For Service Provider developers, OAuth gives users access to their data while protecting their account credentials.

A number of services have already been implemented. These include Fire Eagle, Open Social, Pownce, Get Satisfaction and Magnolia.

More details can be found on the OAuth website.

Resource Description Framework (RDF)

RDF is a family of World Wide Web Consortium (W3C) specifications originally designed as a metadata model but which has come to be used as a general method of modeling information, through a variety of syntax formats.

The RDF metadata model is based upon the idea of making statements about resources in the form of subject-predicate-object expressions, called triples in RDF terminology. The subject denotes the resource, and the predicate denotes traits or aspects of the resource and expresses a relationship between the subject and the object. For example, one way to represent the notion “The sky has the color blue” in RDF is as the triple: a subject denoting “the sky”, a predicate denoting “has the color”, and an object denoting “blue”. RDF is an abstract model with several serialization formats (i.e. file formats), and so the particular way in which a resource or triple is encoded varies from format to format.

This mechanism for describing resources is a major component in what is proposed by the W3C’s Semantic Web activity: an evolutionary stage of the World Wide Web in which automated software can store, exchange, and use machine-readable information distributed throughout the web, in turn enabling users to deal with the information with greater efficiency and certainty. RDF’s simple data model and ability to model disparate, abstract concepts has also led to its increasing use in knowledge management applications unrelated to Semantic Web activity.

More details can be found on the W3C website.

Really Simple Syndication (RSS)

RSS is a family of Web feed formats used to publish frequently updated content including, but not limited to, blog entries, news headlines, and podcasts. An RSS document, which is called a “feed” or “web feed” or “channel”, contains either a summary of content from an associated web site or the full text. RSS makes it possible for people to keep up with web sites in an automated manner that can be piped into special programs or filtered displays.

RSS content can be read using software called an “RSS reader”, “feed reader” or an “aggregator”. The user subscribes to a feed by entering the feed’s link into the reader or by clicking an RSS icon in a browser that initiates the subscription process. The reader checks the user’s subscribed feeds regularly for new content, downloading any updates that it finds.

More details can be found on the RSS Board website.

Outline Processor Mark-up Language (OPML)

OPML is an XML format for outlines. Originally developed by Radio UserLand as a native file format for an outliner application, it has since been adopted for other uses, the most common being to exchange lists of web feeds between web feed aggregators.

The OPML specification defines an outline as a hierarchical, ordered list of arbitrary elements. The specification is fairly open which makes it suitable for many types of list data.

More details can be found on the OPML website.

Attention Profiling Mark-up Language (APML)

APML allows you to share your own personal Attention Profile in much the same way that OPML allows the exchange of reading lists between News Readers. The idea is to compress all forms of Attention Data into a portable file format containing a description of your ranked interests.

Services that have adopted APML include Bloglines, Cluztr, Dandelife, Engagd, Idiomag, OpenLink Data Spaces and Particls.

More details can be found on the APML website.

Securely transfering personal data around the web has become an increasingly important concept to not only users of the web, but service providers. Both Plaxo and Six Apart have been working on a system to allow the transferral of data. However, since Google announce Open Social and the Open Social API, the mantle has been handed over and there is now a strong commitment to realising data portability.

In the late 1990s, a large multi-national technology corporation, hoping to become a major force in online advertising, bought a small start-up in a sector that was believed to be the next big thing. That corporation was Microsoft and the start-up was Hotmail. Hotmail and Microsoft established web-based email as a must-have application for personal use. The addition of Hotmail to the Microsoft inventory promised to increase the companies online revenues that were being dominated by Yahoo!, Google and AOL amongst a host of others.

A decade later it was the turn of a much-evolved AOL to speculate with the purchase of a small and upcoming social networking website, Bebo, for $850m (£425m). This has raised a number of eyebrows since AOL has been a struggling web-portal after its merger with Time Warner, added to the fact that the real value of social networking has yet to be realised or understood.

Social Networking Websites

Both deals in their respective decades offer to the casual observer a paradox of the Internet revolution. Whilst both email and social networking have the premise of being the next big thing which aides revenue generation, it is dangerous to assume that each service can standalone and generate revenue in its own right. Webmail, now over a decade old illustrates this perfectly. Microsoft, Yahoo!, Google and AOL all have their respective webmail services with advertisements stratefically placed to entice the user to click through, but these are a small part of the bigger networks. The offer of email, free archiving, address book and calendar is cheap to deliver, but its primary purpose is to keep the user engaged with the brand and its associated websites, making users more likely to visit the affiliated pages where advertising is more effective.

For instance, I am a fully signed up member of Google and access their email, chat, documents, analytics, webmasters, adsense, adwords, calendar and checkout applications, etc, some of which have advertising and all of which support the core Google search pages through branding. A similar example can also be said of Yahoo!. I again frequently use Yahoo!s MyBlogLog, Flickr and Upcoming services, which serve to re-inforce the Yahoo! brand and web portal.

Social networking will become a ubiquitous feature of online life, but that does not mean it is a business.

From whence came webmail now comes social networking. The implicit values of social networking services such as MySpace, Facebook and Bebo have been increased by the big internet and media companies such as News Corporation, with their purchase of MySpace for $580m (£290m) in 2005 and Microsoft’s $260m (£130m) investment for a 1.6% share in Facebook, in late 2007 (valuing it at an enormous $15bn/£7.5bn). But valuing these online services so highly does not mean that there is a valuable revenue model; Facebook’s revenue for 2007 was a mere $150m (£75m). Sergey Brin of Google also admitted that the monetisation of their Orkut service and social networking in general was proving to be problematic (they also have a contractual agreement with News Corporation to offer advertising on their MySpace service).

Facebook has also been met with criticism and difficulty when trying to monetise its service with a project called Beacon. Facebook’s idea was to inform users’ networks whenever an item was purchased therefore creating what is in effect a recommendation system, or algorithmic word-of-mouth. Users rebelled and privacy advocates shouted loudly, the service was axed and Mark Zuckerberg, Facebook’s founder, was left to apologise for an innovative idea badly implemented.

Whilst social networking does have oportunities to make money, it is unlikely that it will be pots and pots of money. The value of the service, however, is not monetary, but as its genre suggests, it is social. We have already seen how people can connect to past and present friends, but a social networkings strength is in its ability to forge new relationships, business or personal. Social networking has made explicit the connections between people, which has lead to a whole ecosystem of applications built on their APIs which allow users to interact.

But should users really have to visit a specific website to be social?

I often comment that there is something profoundly wrong when people are forced to spend their lives updating their profile to keep in touch with their so-called friends. What happened to the good-old-fashioned telephone? Why don’t people simply arrange to meet up and go for a drink to keep in touch? Of course, with everyone’s increasingly busy lives, it is possible to argue that posting a tweet via twitter, posting an article on a blog or updating your Facebook profile, allows you to continue a real relationship with your friends, whilst not actually needing to see them every Friday or Saturday night. This is a good thing, right?

Another problem presented by today’s social networks is that they are an enclosed ecosystem, at least to users. Whilst Facebook and LinkedIn, in addition to a whole host of others, have provided APIs for developers to encourage them to interact with their services (this has been particularly successful with Facebook) the same cannot be applied to users. The various social networks, until recently, have been reluctant to allow users to pass data between competing services, afterall, this data is core to the success, or indeed failure, of a site. This is understandable since the networks’ huge valuations depend on the sites maximising revenues and page views, so they need to maintain a tight control. As a result, keen Internet users maintain a plethora of online accounts.

2008 will see a change in how people access social networks.

Google Open SocialThe opening up of social networks, lead by Google with their Open Social API, is set to bring about an evolution in this medium. This change is following the historical standardisation of popular services. First it was email with webmail, which in the early days was restricted to individual ecosystems, for example AOL and CompuServe, then it was instant messaging, with individual services provided by Microsoft, Yahoo!, Google, AOL and Skype.

Further developments include the Data Portability Working Group, whose mission is to put all existing technologies and initiatives in context to create a reference design for end-to-end data portability. In short, allow users to move their data around competing services. Others are pushing OpenID; a plan to create a single, federated online sign-on system that people can use to access many websites.

Data Portability

The opening of social networks is likely to accelerate thanks to the first tentative, yet bold, steps made by webmail; the first social network. As a technology, webmail has become old fashioned, but its younger sybling, the social network will revitalise not only webmail, but online communication and advertising. Through social intelligence, marketers and advertisers will be able to target adverts for items that we are more likely to want. This will not only boost the users online experience, but provide a more targeted revenue stream.

The fight for social networking dominance has been running for several years now, but it shows no sign of letting up.

On the Web, a walled garden is an environment that controls the user’s access to Web content and services. In effect, the walled garden directs the user’s navigation within particular areas, to allow access to a selection of material, or prevent access to other material.

Recent history suggests that open standards will again better the “walled gardens” of the Web.

In 1994, when the previously obscure computer network, developed by the American Department of Defence, first become known to the general public as the “World Wide Web”, or simply The Web, many people first connected to it via AOL and CompuServe. These subscription-based service providers offered not only access to the Internet, but other services such as email, chatrooms, discussion boards and more. It was access to the Web via the Internet that would lead to the undermining of these services, and the opening up of the Web as a platform for individual and creative expression, revenue generation and social interactivity.

Whilst it took some time for the closed communities to venture out into the wilds of the Web, it brought about the standardisation of the services that made up the early web. For instance, POP and SMTP standardised email and as a result it has become the ubiquitous tool of business. Today, of the early pioneers of the Web, only AOL survives, but as an entirely different entity; a web portal supported by advertising.

History appears to be repeating itself. The biggest online phenomena of the past couple of years, the social-networking websites of Facebook and MySpace, are acting very much like the AOL of the mid-1990s. They are closed systems based upon prioprietory standards. You cannot easily move information from one system or another if you so choose. This ties users into one system, or forces them to create profiles on both. A similar comparison can be drawn with the virtual worlds of Second Life and Entropia Universe.

The Web is better when it’s social.

Part of the reason these websites are popular is because they are closed communities, where users can interact with friends and find new friends with which to interact. This community feel has been tested in recent times, with sites such as Facebook being criticised for using their user’s personal data to target advertising. It is innevitable, however, that these systems are proprietory; it is only once these systems immerge and become popular that standards can be developed and implemented.

Open Social API

Just as the Web’s open standards, embodied in the Netscape browser, displaced the online services providers, so the paradigm of open standards awaits the social networking and virtual worlds. Back in the 1990s it was Netscape, but in the 21st Century it falls to Google to defend the open standards of the Web with the Open Social API. Some say there is a large amount of self interest in this move, since Facebook and MySpace have huge communities, which both networks know a huge amount more about than Google and can hence generate billions of dollars of revenue.

The web is more interesting when you can build applications that easily interact with your friends and colleagues. But with the trend towards more social applications also comes a growing list of site-specific APIs that developers must learn. Open Social is an attempt not only to open up the closed communities and allow developers to interact with the different networks, but allow developers to only learn one API. MySpace has signed up to this initiative and, more reluctantly so has Facebook. A curiosity is AOLs recent aquisition of Bebo, another online community popular in Europe. Is AOL simply jumping on the “band-wagon”? Has it learnt its lessons of the past, or is it using knowledge of its past as a guiding principle? Whatever is the answer, Bebo’s inclusion in Open Social will help it continue its competition with other social networking websites.

Adobe AIR LogoSince the Adobe Integrated Runtime (AIR) was released at the end of February, we now have a stable platform on which to build desktop applications with our existing web skills. A number of people have already started and the Adobe AIR Marketplace is filling with AIR applications by the day.

So what is the big deal? The Adobe marketing team state that:

The Adobe AIR runtime lets developers use proven web technologies to build rich Internet applications that deploy to the desktop and run across operating systems. Adobe AIR offers an exciting new way to engage customers with innovative, branded desktop applications, without requiring changes to existing technology, people, or processes.

What AIR applications should you check out?

What is intriguing is that all the tools I have chosen are generally useful tools for the developer or designer, with the exception of twhirl, which is a social-interaction tool. I’m looking forward to when other, less developer-centric tools become freely available. AgileAgenda has taken the lead with this respect, albeit not freely available, as has eBay desktop, but I would like to see examples from the BBC in the form of a desktop BBC iPlayer or maybe a Flickr image browser, del.icio.us bookmark reader, RSS aggregator and a Picnik image editor.

Analytics Reporting Suite

Google Analytics Reporting SuiteThe Analytics Reporting Suite, by Nicolas Lierman, brings Google Analytics to the desktop. It uses it’s own custom API to interact with Google and nearly implements all the features of Analytics.

For website owners this is a must-have application. Like the twhirl AIR application below, it is a fantastic example of what can be achieved with Flex and AIR. Measuring visitor trends and traffic are essential tasks to managing and improving a websites performance. The Analytics Reporting Suite allows you to configure multiple Google’s Analytics accounts and access the web-based suite’s plethora of features via a desktop application. The application displays integrated graphs and animations via a tabbed interface, which allows you switch between a number of reports. These reports can then be saved as a PDF, Excel or XMLdocument, or printed.

You can download and install the application from the About Nico website.

twhirl twitter Client

Twhirl Logotwhirl, by Marco Kaiser, is probably the most popular desktop client for the twitter micro-blogging service. Most of the features available on the twitter website are accessible through twhirl, plus, a lot of usability enhancements have been added to make it easier to manage multiple accounts. This is great for those who want to separate business and personal accounts they may have.

The twhirl application is a great example of how AIR can bring web applications to the desktop; it can dock to the system tray, display message alerts and you can configure the applications opacity when not focused (great if you like Mac and Vista-styled themes). The application allows you to search twitter users, view their timelines, add friends, view followers, delete tweets and much much more. Twhirl automatically fetches your friends’ status updates, direct messages and replies, whilst also colour coding different types of messages and alerting you to messages both audibly and visually.

The twhirl application is skinnable and comes with several built in skins with which you can customise the application. All-in-all twhirl is not only one of the best twitter clients, but AIR applications.

You can download and install the application from the twhirl website.

Kuler Desktop

Adobe Kuler LogoAdobe kuler is the first web-hosted application from Adobe Labs designed both to stand alone and to complement Adobe Creative Suite software. Built using Adobe Flash and ActionScript 3.0, kuler is all about colour: colour for exploration, inspiration, experimentation and sharing. Kuler is clearly targeted at the designer, but anyone interested in colour will benefit from its use.

You can download and install the application from the Adobe Labs website.

WebKut

WebKut LogoWebKut is a web screenshot tool that allows you to capture web pages, or parts of them in a very simple way. It provides you with 3 capture options: the entire page, the current view, or only a selection. This little application proves particularly handy for those presentations or projects that need great visuals from the web.

You can download and install the application from the WebKut website.

RichFLV

RichFLV, by Benjamin Dobler, lets you edit FLV files. The key features include reading FLV metadata, read and edit cuepoints, cut FLV files, convert the sound from an FLV to MP3 format, convert an FLV to an SWF … and much more.

You can download and install the application from the Adobe AIR Marketplace website.

SearchCoders Dashboard

SearchCoders LogoThis Flex-based chat widget is designed with programmers in mind. The code input feature allows developers to chat about code without disrupting the conversation.

You can download and install the application from the SearchCoders website.

Pownce

Pownce LogoMuch like twhirl in look, feel and ease-of-use, but with a slant towards productivity rather than micro-blogging, Pownce is a way to keep in touch and share things with your friends or colleagues. You can send people files, links, events, and messages and then have real conversations with the recipients. This is a great collaboration tool and was one of the first services to really embrace AIR as an application architecture, which could realise their service as a desktop client. Everything that is available via the Pownce website is also available via the client application, except and possibly importantly, the ability for the user to amend their account settings and add friends to your network; this still has to be done via the website.

For a small annual amount, Pownce offers a paid-for service which will eliminate adverts from your profile and allow you to send huge file sizes (100MB) and customise the theme of your Pownce.

Pownce also offers Drupal integration and a mobile application, which works with the iPhone, BlackBerries and many more ‘internet-ready’ mobile devices.

You can download and install the application from the Pownce website.

Google’s mantra of “Don’t Be Evil” is all well and good, but it doesn’t equate to “Always Be Good”!

I, like a vast number of websites, rely on Google Adsense for revenue, which in my case, compensates me for the material costs of running a website but not much more. To me that is OK, assuming Google actually pays up! My account has been set-up for several years, I have added my account details, etcetera etcetera. However, since they introduced the idea of a Pin, everyhting has fallen apart. Three requests for a pin have fallen on deaf ears, emailing the Adsense team has fallen on deaf ears! To add insult to injury, Google now only serve me Public Service Advertisements (PSA), which, unless you really care about hurricane relief whilst reading one of my inspiring blog entries, are not going to receive a single click!

So what can I do? Any suggestions would be gratefully received. For the moment, I will have to repeatedly email the Adsense team and hope for the best.

To paraphrase Plato

Human behaviour flows from three main sources: desire, emotion and knowledge

Desire lines are those well-worn ribbons of dirt that you see cutting across a patch of grass, field or park, often with nearby pavements, particularly those that offer a less direct route, ignored. In winter, desire lines appear spontaneously as trampled down paths in the snow. These paths are never perfectly straight but instead, they meander like a river this way and that, as if to prove that desire itself isn’t uniform or linear and (literally, in this case) straightforward. Desire Lines show that it is human to choose, but it is also human to choose what other people have chosen before. Be it for establishing a pattern, be it for convenience, or be it for not reinventing the wheel. In this way the Desire Lines become well-trodden and pseudo-permanent routes to a particular destination.

Desire Lines are the ultimate unbiased expression of natural human purpose

The term ‘Desire Lines’ originates from the field of urban planning in the early 20th Century.

An optimal way to design pathways in accordance to natural behaviour is not to design them at all

Extending the concept to websites

Broadening the concept of Desire Lines, it is also possible to see other impressions of human desire in websites, many of which can provide a commercial advantage to the particular website in question.

The web phenomenon is simply another way people find entertainment, communicate and interact. Whether the purpose of using the web is for enjoyment or employment, Desire Lines are are a such perfect expression of natural human intention that they become the utlimate design pattern for building successful and intuitive websites. It is easy to envisage users like an army of ants moving in single file towards their goal, not following set paths, but bypassing the planned route directly towards the news article, the video, indeed any product being offered by the website concerned.

Instead of websites providing classic access points via menus and taxonomies, they can employ different access points based upon user preferences and previous browsing history. This is heavily dependent upon capturing user’s details via a cookie or a login system, like at Amazon, but the benefits to the user can be far more rewarding.

So who is doing this?

Understanding the demographics of your users and your site usage is key to Desire Lines. The BBC website is a great example whereby search statics are monitored frequently. They adapt their web user interface and site structure based upon what people are typing into their search box thus making it easier for subsequent users to view the more relevant or interesting stories of the day.

Amazon makes Desire Lines more explicit and beneficial to their users through Listmania, recommendations, Customers Who Bought This Item Also Bought, What Do Customers Ultimately Buy After Viewing Items Like This?, Better Together, Recently Viewed Items, Wish Lists etc. In essence, Amazon does not provide a single standard way of finding an item, but relates each item together through a myriad of historic information from its user community. The user therefore has the option to search for an item, use the catagories, or click on links and begin their own desired user journey, which can be uniquely distinct from other users journeys or a mirror of those journeys.

Current TV takes the concept even further by showing content that is purely based upon what users want, or what they term as viewer-created content. Current slice the schedule into short segments into “pods” — each just a few minutes long. These pods profile interesting people on the rise, intelligence on trends as they spring up around us, and international news from new perspectives. Anyone who wants to contribute can upload a video. Then, everyone in the Current online community helps decide what should be on TV. You can join in at either stage — watch & vote or make video.

Limitations to Desire Lines

A key limitation, or possibly a challenge, to the successful adoption of Desire Lines concerns advertising revenues. For a user it is beneficial to be able to reached the desired destination as quickly and efficiently as possible, but for a website’s advertising revenue it certainly is not. To be simplistic and possibly cynical,