The Internet is not simply a big place it is a huge place; new content is being created all the time. Google, Yahoo and Microsoft each have a finite number of resources, so when faced with the nearly-infinite quantity of content that’s available online, their various crawlers are only able to find and crawl a percentage of that content. Then, of all the content they’ve crawled, they’re only able to index a portion. Of course with the cheapness of storage, the search engines are able to index more and more content each day, but not at the pace the Web is growing.

URLs are like the bridges between your website and a search engine’s crawler: crawlers need to be able to find and cross those bridges (i.e., find and crawl your URLs) in order to get to your site’s content. If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organised and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs.

So, what can you do as a website developer or owner to reduce that labyrinth of URLs and helping crawlers find more of your content faster? Below are a few ideas:

• Remove unnecessary query string details from the URL.
  Parameters in the URL that don’t change the content of the page–like session IDs or list sort orders–can be removed from the URL and put into a cookie. By putting this information in a cookie and 301 redirecting to a clean URL, you retain the information and reduce the number of URLs pointing to that same content.
• Stop infinite pagination in, for example, lists and calendars.
  If you have a calendar with infinite past and future dates or a list with infinite pagination you have what is described as an infinite crawl space, which is a huge burden on crawlers. To resolve the calendar issue, you can add no-follow attributes to links to dynamically created future calendar pages. When creating pagination links, disable previous and next links when the first and last pages are reached and redirect users to an appropriate page if the query string in the URL is hacked (this may be a page not found static page).
• Utilise the robots.txt file to prevent actions the web crawlers can’t or shouldn’t perform.
  Using a robots.txt file, you can disallow crawling of login pages, contact forms, shopping carts, and other pages whose sole functionality is something that a crawler can’t and shouldn’t perform. This lets crawlers spend more of their time crawling content that they can actually do something with.
• Prevent duplicate content.
  An ideal scenario for crawlers is a one-to-one link between content an a URL. Each URL leads to a unique bit of content and each piece of content can be accessed by a unique URL. The closer your site can get to this scenario, the more streamlined your site will be for crawling and indexing. If your CMS makes this difficult to achieve, you can use the canonical tag to indicate a preferred URL for duplicate content.

More information on this topic can be found on the Google Webmaster Central Blog.

Carpe diem on any duplicate content worries: we now support a format that allows you to publicly specify your preferred version of a URL. If your site has identical or vastly similar content that’s accessible through multiple URLs, this format provides you with more control over the URL returned in search results. It also helps to make sure that properties such as link popularity are consolidated to your preferred version.

But what do they mean by canonical? One of the definitions of canonical is reduced to the simplest and most significant form possible without loss of generality.

What this means is that if you have a page–let’s take an e-commerce product page–and the simplest URL that you want it accessible by is:

http://www.site.com/category/product.html

you can add the canonical tag to that specific product. Google, Yahoo and Microsoft use this tag to tell their search engines which URL it should have for the current page.

Now, let’s say that the particular software you use also allows you to access the same product using:

http://www.site.com/company/product.html

and

http://www.site.com/different_category/product.html

Perhaps this one product is in multiple categories. With this tag in place when any of the alternate pages are loaded this tag notifies any search engine that this is really the same product as the page you defined in the canonical tag. So, you are still allowed to have the content available as generally needed (by categories, tags, or some other organisation system) and still avoid having the content duplicated and penalised.

To implement the canonical URL tag in your web application, you simply need to do the following inside the <head> section of the duplicate content URLs:

<link rel="canonical" href="http://www.site.com/category/product.html" />

As Google mention, this tag is a hint that they honour strongly. Google will take your preference into account, in conjunction with other signals, when calculating the most relevant page to display in search results.

For example, the following is WordPress’ default URL configuration for a post:

http://www.domain.com/?p=1635

However, buy using a URL-rewriting available in the Apache webserver, we can achieve a far better result, such as the following:

http://www.domain.com/search-engine-safe-urls

NB. It is also possible to achieve a similar result with an ISAPI rewrite for Microsoft’s IIS webserver, but this topic will not be included in this post.

To get your website working with SES URLs you need to enable both the mod_rewite module and AllowOverride directive in the Apache configuration file.

Uncomment (remove #) from the following to enable the re-write rule:

LoadModule rewrite_module modules/mod_rewrite.so

Change the AllowOverride directive from none to all

<directory />
    Options FollowSymLinks
    AllowOverride all
    Order deny,allow
    Deny from all
</directory>
 
<directory "C:/WebRoot">
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.2/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks
 
    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   Options FileInfo AuthConfig Limit
    #
    AllowOverride All
 
    #
    # Controls who can get stuff from this server.
    #
    Order allow,deny
    Allow from all
</directory>

On Apache webservers, .htaccess (hypertext access) is the default name of directory-level configuration files. An .htaccess file is placed in a particular directory, and the directives in the .htaccess file apply to that directory, and all its subdirectories. It provides the ability to customize configuration for requests to the particular directory. In our case, enabling search engine safe (SES) URLs.

By setting the AllowOverride directive to All in effect defers configuration settings to the .htaccess file.

An example .htaccess file could include the following code to rewrite the URLs:

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php/$1 [L,QSA]

Search engine friendly URLs are implemented with Rewrite engines. The rewrite engine modifies the URL based upon a number of rewrite conditions and rules.

The RewriteBase directive explicitly sets the base URL for per-directory rewrites. The RewriteCond directive defines a rule condition, so in this case handling missing files or directories. Finally, the RewriteRule directive is the real rewriting workhorse. In this example, we’re getting everything in the URI — i.e. not including the protocol (HTTP/S) and domain name — based upon a regular expression. This is then appended to the default file reference — index.php — as a back reference. The [L,QSA] refers to the rule being the last rule and append any query string parameters to the default file. It is important to note that this is all done on the server side, the user will never see the website address changing in the browser’s address bar. Furthermore, simply transposing the index.php filename with your default file name — e.g. index.cfm, default.aspx — will have the same result. Indeed, the above rewrite rules are becoming a de-facto standard for web applications.

To fully understand mod_rewrite rules above, look at the Apache mod_rewrite documentation.

Once you have your SES functionality in place on the webserver, it is then the responsibility of your application framework to understand the URL construction and handle it accordingly. Fortunately, frameworks such as ColdBox and Fusebox for ColdFusion, Zend and Symfony for PHP, all contain functionality to do this, but that is the subject of an entirely different post.

Users of web applications prefer short, neat URLs to raw query string parameters. A concise URL is easy to remember, and less time-consuming to type in. If the URL can be made to relate clearly to the content of the page, then errors are not only less likely to happen, but our good friends the search engine robots are able to draw a stronger assumption of the pages’ relevance and content.

But what makes up a web community; what makes them successful? Below I discuss the four C’s of community: Content, Context, Connectivity and Community.

Content

A current meme when organising or building a website is the catchphrase Content is King. A big shift in the web in recent years has been the way websites are constructed. Today it’s a necessity, and indeed best practice, to separate form from content. In one hand you have the compelling content, whilst in the other you have the presentation, be it in the form of HTML and CSS, Flash or RSS, amongst others.

Quality content is one way in which you can make your website stand out. It is also a great way to attract the people who are needed to form the elusive community that your brand is hoping build. When considering community initiatives, there are three questions to ask: Where will the content come from; for example community driven or syndication? Does it provide indisputable value; does it have a unique selling point (USP)? Can a regular flow of quality content be maintained? Even pre-Web 2.0 initiatives have to focus on keeping the content itself fresh and relevant.

Web accessibility and search engine optimisation are also vital, so having content completely separated from presentation means a number of assistive technologies can make better use of the content, whilst the web robots can also readily consume the information.

Context

Context means understanding how people use your website, where they are in the user-journey and serving them the right experience at the right time. Well-designed applications and functionality have great opportunities to deliver on context.

For example, FriendFeed‘s iPhone version, which is simply a re-worked web interface, is perfectly designed for contextual usage on the go. Similarly, Remember The Milk updates the interface explicitly for mobile and iPhone users, whilst also syndicating the content to applications such as Google Calendar. (It is questionable whether user-agent switching is good practice, but that is a whole new blog post.) Conversely, Delicious makes no attempt at changing the user interface for iPhone or Nokia N95 users since the iPhone and N95 have full web-capabilities through their respective web browsers.

In some instances the context in which the content is displayed will require reduced functionality. For example, the Last.fm mobile site does not allow you to play music, but simply search music listings, view recommendations, events and friend listings, and edit settings. However, through its API, Last.fm is able to offer its data and platform to third party developers to aid the building of new applications and communities, thus changing its context.

Connectivity

Connectivity is the ability of a system, whether that is a web-based community or a device like the iPhone, to connect with little or no modification. In the realm of communities, the ability to easily connect to your peers is the Holy Grail of the application.

Successful communities thrive on fluid, hard-to-measure activities that are, in the purest sense, relationship-based. It’s not all about mass communications — although Twitter and YouTube are both bucking this trend — but more about the micro-interactions. Designing experiences that support thousands of micro-interactions means that the community is able to function, unhindered, almost indefinitely. Facebook lends itself expertly to micro-interactions through the user’s ‘wall’.

Companies are turning to communities as the new customer relationship management (CRM), but this requires people to mind them. Organisations such as 37Signals and WildBit very effectively use Twitter to broadcast service updates and sometimes apologies, whilst the BBC and The Guardian online use it to broadcast links to new content.

Continuity

People often don’t like change, but communities that thrive often do so though evolution to meet the needs of users. Communities need to be flexible to evolve while still providing a valuable and consistent user experience which can be sustained. Too much of a radical change will almost certainly have a detrimental impact upon visits, at least initially.

Building communities is the new marketing for a brand, whether that is through wholely-owned properties or 3rd party social media services such as Twitter, WordPress or Ning. The starting point to any community is finding a niche that is currently underserved and serving that community better than anyone else. But Brands need to know a few things before they head down the community path. The web is saturated with communities. Some are thriving, while others have come and gone. Creating a community is not like your average marketing campaign that you can ditch it is a failure. If the community is successful the four C’s of content, contect, connectivity and continuity will have to be maintained and indeed, developed.

Websites are increasingly the key communication vehicle for a company, its brand and products. Despite this, research from Magus shows that many of the world’s leading organisations don’t have formal brand and technical standards defined to govern their websites. Even those that do are failing to effectively implement or enforce them, achieving full compliance with less than 20% of their own web standards. The websites of these organisations significantly under-perform or damage the brand as a result.

PAS 124 will help to protect the significant investment organisations are making in their web presence and online brands, by establishing best practice for “defining, implementing and managing organisational web standards”. It will provide a clear framework to help organisations apply standards effectively to significantly improve online performance and protect the integrity of their brands.

More detail can be found on the BSI’s press release.