Page Speed analyses the performance of web pages and gives specific suggestions for making them faster. Page Speed is available from any browser. This allows website owners to get immediate access to Page Speed performance suggestions so they can make their pages faster.

The suggestions are divided into High, Medium and Low priority. The test, appropriately, runs in a matter of seconds. This site scored well with a grade of 91 out of 100. Here’s what Google suggests should be done to make up the 9 points:

That’s an analysis of desktop page speed; you can select to also run a mobile speed report.

There are also page speed extensions for Firefox and Chrome, as well as other code and instructions for developers on that page.

Anticipating the need to find video content online, the major search engines (Google, Yahoo and Bing) have created their own video search engines. These spider the web for unique video content and publish it in their video sections.

An interesting and cool result of this is that, instead of posting normal text listings, the search engines publish thumbnail screenshots of the videos on the search engine results pages (SERP). The thumbnail enables users to see what kind of video they can expect to see, before actually viewing it.

Universal search — a system that blends image, video, news, blog, local and traditional search to form a more complete experience and result set — is now integral to all three major search engines. Listings that have video thumbnails have been proven to receive a much higher click-through rate than traditional text listings.

So, how can you get your pages listed with video thumbnails?

1. Embed your video onto your web page using any number of methods. For example, you could use Flash video, Windows media video, Quicktime, Ogg Theora amongst others. As a point to note, you shouldn’t use a third-party video site — such as YouTube, Vimeo and DailyMotion — for this; the video must be unique to your website.
2. Create a thumbnail of the video for indexing.
3. Create four meta tags on the web page that displays the video:

   <meta name="medium" content="video" />
   <meta name="video_type" content="application/x-shockwave-flash" />

   Replace application/x-shockwave-flash with whichever video format you’re using.

   <meta name="video_height" content="280" />
   <meta name="video_width" content="460" />

   Replace the height and width dimensions with those specific to your video.

4. Create a video sitemap and submit it to the search engines. Google and Yahoo have a clear video submission process, whereas Bing does not, so you should submit to Bing in the normal way.

If your page is already in position one for a keyword, that listing will gain a thumbnail once you have optimised it for video. You’ll also get increased time on site and possibly page views, afterwhich you can develop a better conversion-rate strategy.

The decision to utilise a sub-domain, sub-directory or even a microsite is simply an architectural decision, but one that is often compounded with a marketing decision. In general, sub-directories are used to describe what individual pages are about while sub-domains and microsites are used to describe what an entire site is about.

When to use sub-directories

• Sub-directories are useful for organising content into meaningful groupings with descriptive URLs. If you have a meaningful taxonomy, then sub-directories are a good way to make your URLs more search friendly.
• For relatively small websites, i.e. not the BBCs and CNNs of this world, keeping your content in one place will help your website build trust and authority. The more trust and authority a website is able to garner with the search engines, the easier it becomes to rank new and related content.
• For the less technically savvy, sub-directories are easier to manage than sub-domains since they are generally created as part of the website’s page hierarchy. Furthermore, many CMSs don’t support the ability to publish across different domains out-of-the-box.
• Google Webmasters tools allow you to set basic geo-location preferences to sub-directories. The idea is to create new pseudo-website listings that also include sub-directories. For example, the www.domain.com listing could have www.domain.com/uk for the UK, www.domain.com/es for Spain and so on for each geographic location.

When to use sub-domains

• Sub-domains are useful for organising content that is otherwise unrelated. Take Google for example, they have a news product at news.google.com, a maps product at maps.google.com and a email product at mail.google.com. Since they are fundamentally different from each other and Google’s main search product, they aren’t held under the same domain.
• Sub-domains allow you to target regional markets more effectively. Sub domains are easier to market to specific geographical regions. For instance you can assign an IP address to a sub-domain and set a geographical preference in Google’s Webmaster tools for each sub-domain (sub-directories can have their geo-preference set but can’t be assigned an IP in another country, which might also be a signal of geo-location). For example, the www.domain.com listing could have uk.domain.com for the UK, es.domain.com for Spain and so on for each geographic location.
• If you already have a well established domain and want to expand out into other areas not completely related to your main website’s activities then a sub-domain may well be a good option. At the same time, people associate the sub-domain with your main domain’s brand, which means it can be easy to build up momentum on a vertical related to your main site. This is what the web giants Google and Yahoo, Sky and the BBC do successfully, but smaller websites can do the same. You often see this when the third-party functionality is “plugged into” an existing site, such as a payment gateway, events and job boards.

When to use microsites

Like sub-domains, microsites have an important position in the overall debate.

• If you have a new product or service that you potentially want to sell off or brand completely differently from your main business offering, a microsite makes a lot of sense. Microsites are really just a stripped down website, but as the product or service develops, so will the website and associated brand and it will take on trust and authority of its own.
• Microsites offer the ability to completely separate your main brand from the new product, service or promotion. This is useful if you’re pushing a content piece that has little to no association with your site and you don’t want the potential branding confusion or commercial association to hinder link and user growth.
• Microsites can be powerful if you have an exact match domain name for a particular keyword you’re targeting. Google’s preference for and ranking exact-match domains is a very powerful tool to use for SEO.

For a beginner, Google AdWords can be a little daunting to start with, so below I list some tips and tricks that have served and continue to serve me well whenever I setup and manage a campaign.

1. Don’t use broad keyword matching — At least not to start off with. What’s broad matching? “If you include general keyword or keyword phrases-such as tennis shoes-in your keyword list, your adverts will appear when users search for tennis and shoes, in any order, and possibly along with other terms.” With expanded matching it becomes even harder to know when your keywords will show, because Google will pick them algorithmically. Broad match is usually used by experienced advertisers looking to save time (usually with a long list of negative keywords) or by lazy advertisers, who may not have the time or the inclination to target their campaigns. Broad matching can also be the right way to go for parts of certain kinds of campaign, but wading straight into it can be dangerous.
2. Use dynamic titles in your adverts — Dynamic titles are easy, they don’t cost anything and they usually have a good effect on CTR and conversion rates. The phrase that the searcher uses in their search will come up as the the title of your advert. This saves you having to create an individual advert for each keyword and means that your ad will be more targeted. In the title field of your ad simply put {keyword:your default title here}. The default title is in case the search phrase is too long for the title field or if AdWords can’t display the search for some other reason.
3. Use global negatives to prevent advert duplication — If you’re selling something, you don’t want people finding your advert if they’re searching for free stuff. This sounds obvious, but do some searches and you’ll see it happening. Use the word ‘free’ in your campaign global negatives. You can also use this for other words you don’t want to turn up for. Using negative keywords is especially important if you’re using broad matching.
4. Turn off content targeting and search network — Do you know where to look at these adverts? If the answer is a resounding “no”, leave them alone until you feel confident that you know where they’ll be showing and that you can make them work — get comfortable with Google SERPs first. Conversion rates and CTR‘s change dramatically for each search property, not to mention content targeted adverts. Get acquainted with Google before you move on to the others.
5. Test different advert-copy and positions — How much of a difference will being in the first position, as opposed to the second, third or sixth position make, for your net profit? The answer is that it depends on your creative, industry and who else is bidding on your keywords. The bottom line is that you should know. Test your creative in each position and work out where it will be most effective, from an ROI point of view. Remember also that the AdWords ranking algo works on a CPC x CTR basis (it’s actually more complicated than that, but that gives an idea as to why out of two adverts with similar CPC‘s, one will be higher because of a higher CTR.) You can also test your creatives. Write several different adverts and set them to run evenly (Google will run the one which preforms best by default, but you can set them to run evenly in your campaign settings). The creative which gives you the highest ROI is the one you should go with. You should probably run this kind of test for more than just a day.
6. Optimise your destination URLs and landing pages — The landing page is the page which the person who clicked on your advert will see when they come through to your site. Don’t use your homepage as your landing page unless it deals only with selling the product you’re advertising. For example, if you’re selling widgets, optimise a page on your website with information regarding that widget, this could be a product page on your website, or a specific page for the advertising campaign.
7. Track conversion and ROI — Track everything. Google will track impressions, clicks and click through rate. Just because an advert has a high CTR doesn’t mean that it is making you money. Use Google’s own conversion tracking codes and link AdWords with Analytics. If you don’t know how each keyword is preforming, then you won’t be able to optimise your campaigns. Turn off the keywords that aren’t working well and invest more in the keywords which are.
8. Work out your CPA — How much is each sale costing you? Are your Google AdWords listings preforming as well as your Yahoo! listings? Your Bing listings? Your offline marketing? You should know how much you’re paying for each order, sale, download or enquiry on each channel — only then will you be able to set CPA targets to work towards and know which channel is best for you.
9. Don’t enter into bidding wars with your competitors, find a niche — It is easy to get into bidding wars with your competitors. You want to be number one and so does your competitor. The best thing to do is to take a step back, ten deep breaths and consider how important it really is to be in the number one spot. If you can justify it, fine but otherwise: let your competitor be number one, slot into the number two slot and wait. If you have a better product, your click-through rate will get you to the top – and you’ll still be paying the same as you were in the second position. More and more people are using automated bidding software. This software will update at a set interval to keep the ad in the desired position. Doing battle with this kind of software is even more frustrating and less rewarding than trying to outbid a competitor.
10. Set a weekly budget and stick to it — When you’re first starting out this is really important. AdWords is addictive, and like anything else addictive, it can quickly get out of hand. A keyword may convert like nobody’s business one day and die the next. Make sure you know a keyword is working — more importantly, know why it is working — before you invest more.
11. Geo-target your adverts — Geo-target your adverts to the relevant audience. It can be done when you set up a new campaign and in your campaign settings. If you want to show on google.com that’s fine, but remember that you can geo-target to individual countries and even cities (you can target regions in the US, UK, Canada, France, Germany, Italy, Japan, Spain and the Netherlands). The more targeted users feels that your ad is, the more likely they are to click on it.
12. Organise your adverts into logical groups — This will make it easier for you to know what is doing well and what isn’t — especially a year later. If you’re selling widgets have a campaign not just for widgets, but for each brand of widget, and then an ad-group for each widget model.
13. Keep a close eye on what your competitors are doing — Is your competitor doing something better than you? What keywords do you share with your competitors? Can you discern their strategy? Do they use automated bidding software? Do their landing pages have some magic which you should be emulating? This is the sort of information you can find out and the kind of information that you need to know.

If there are inaccuracies in the above list, just give me a shout, otherwise, I’m happy to hear your own tips and experiences.

Google’s addition of a page speed signal to its search rankings algorithm officially links performance with search engine marketing. The loading speed of a web page affects user psychology in a number of ways, and now it can effect its rankings as well.

From the Google Webmasters’ Blog:

Speeding up websites is important — not just to site owners, but to all Internet users. Faster sites create happy users and we’ve seen in our internal studies that when a site responds slowly, visitors spend less time there. But faster sites don’t just improve user experience; recent data shows that improving site speed also reduces operating costs. Like us, our users place a lot of value in speed – that’s why we’ve decided to take site speed into account in our search rankings. We use a variety of sources to determine the speed of a site relative to other sites

If you are a site owner, webmaster or a web author, there are some free tools that you can use to evaluate the speed of your site and therefore improve site performance:

• Page Speed, an open source Firefox/Firebug add-on that evaluates the performance of web pages and gives suggestions for improvement.
• Yahoo! YSlow a Firefox/Firebug Add-on that analyzes web pages and suggests ways to improve their performance, based on a set of rules for high performance web pages.
• WebPageTest shows a waterfall view of your pages’ load performance plus an optimization checklist.
• BrowserMob a tool for website performance monitoring and alerting.
• Chrome Developer Tools that let you edit, debug, and monitor CSS, HTML, and JavaScript live in any web page. You can also use them to optimise web page performance by profiling CPU and memory usage.
• Google’s Webmaster Tools, in the Labs menu, Site Performance shows the speed of your website as experienced by users around the world.

Other tools, including those for web debugging (Charles and Fiddler) and resource optimisation (Smush It and YUI Compressor), can be found on Google Code.

Google’s announcement of adding a page speed signal into its search rankings algorithm will in the long term will force the hand of website owners to increase the performance of their websites. This can only be a good think for user experience. With time and investment, the frustration of slow-loading websites will be banished to the past.

The Internet is not simply a big place it is a huge place; new content is being created all the time. Google, Yahoo and Microsoft each have a finite number of resources, so when faced with the nearly-infinite quantity of content that’s available online, their various crawlers are only able to find and crawl a percentage of that content. Then, of all the content they’ve crawled, they’re only able to index a portion. Of course with the cheapness of storage, the search engines are able to index more and more content each day, but not at the pace the Web is growing.

URLs are like the bridges between your website and a search engine’s crawler: crawlers need to be able to find and cross those bridges (i.e., find and crawl your URLs) in order to get to your site’s content. If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organised and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs.

So, what can you do as a website developer or owner to reduce that labyrinth of URLs and helping crawlers find more of your content faster? Below are a few ideas:

• Remove unnecessary query string details from the URL.
  Parameters in the URL that don’t change the content of the page–like session IDs or list sort orders–can be removed from the URL and put into a cookie. By putting this information in a cookie and 301 redirecting to a clean URL, you retain the information and reduce the number of URLs pointing to that same content.
• Stop infinite pagination in, for example, lists and calendars.
  If you have a calendar with infinite past and future dates or a list with infinite pagination you have what is described as an infinite crawl space, which is a huge burden on crawlers. To resolve the calendar issue, you can add no-follow attributes to links to dynamically created future calendar pages. When creating pagination links, disable previous and next links when the first and last pages are reached and redirect users to an appropriate page if the query string in the URL is hacked (this may be a page not found static page).
• Utilise the robots.txt file to prevent actions the web crawlers can’t or shouldn’t perform.
  Using a robots.txt file, you can disallow crawling of login pages, contact forms, shopping carts, and other pages whose sole functionality is something that a crawler can’t and shouldn’t perform. This lets crawlers spend more of their time crawling content that they can actually do something with.
• Prevent duplicate content.
  An ideal scenario for crawlers is a one-to-one link between content an a URL. Each URL leads to a unique bit of content and each piece of content can be accessed by a unique URL. The closer your site can get to this scenario, the more streamlined your site will be for crawling and indexing. If your CMS makes this difficult to achieve, you can use the canonical tag to indicate a preferred URL for duplicate content.

More information on this topic can be found on the Google Webmaster Central Blog.

Carpe diem on any duplicate content worries: we now support a format that allows you to publicly specify your preferred version of a URL. If your site has identical or vastly similar content that’s accessible through multiple URLs, this format provides you with more control over the URL returned in search results. It also helps to make sure that properties such as link popularity are consolidated to your preferred version.

But what do they mean by canonical? One of the definitions of canonical is reduced to the simplest and most significant form possible without loss of generality.

What this means is that if you have a page–let’s take an e-commerce product page–and the simplest URL that you want it accessible by is:

http://www.site.com/category/product.html

you can add the canonical tag to that specific product. Google, Yahoo and Microsoft use this tag to tell their search engines which URL it should have for the current page.

Now, let’s say that the particular software you use also allows you to access the same product using:

http://www.site.com/company/product.html

and

http://www.site.com/different_category/product.html

Perhaps this one product is in multiple categories. With this tag in place when any of the alternate pages are loaded this tag notifies any search engine that this is really the same product as the page you defined in the canonical tag. So, you are still allowed to have the content available as generally needed (by categories, tags, or some other organisation system) and still avoid having the content duplicated and penalised.

To implement the canonical URL tag in your web application, you simply need to do the following inside the <head> section of the duplicate content URLs:

<link rel="canonical" href="http://www.site.com/category/product.html" />

As Google mention, this tag is a hint that they honour strongly. Google will take your preference into account, in conjunction with other signals, when calculating the most relevant page to display in search results.

For example, the following is WordPress’ default URL configuration for a post:

http://www.domain.com/?p=1635

However, buy using a URL-rewriting available in the Apache webserver, we can achieve a far better result, such as the following:

http://www.domain.com/search-engine-safe-urls

NB. It is also possible to achieve a similar result with an ISAPI rewrite for Microsoft’s IIS webserver, but this topic will not be included in this post.

To get your website working with SES URLs you need to enable both the mod_rewite module and AllowOverride directive in the Apache configuration file.

Uncomment (remove #) from the following to enable the re-write rule:

LoadModule rewrite_module modules/mod_rewrite.so

Change the AllowOverride directive from none to all

<directory />
    Options FollowSymLinks
    AllowOverride all
    Order deny,allow
    Deny from all
</directory>
 
<directory "C:/WebRoot">
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.2/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks
 
    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   Options FileInfo AuthConfig Limit
    #
    AllowOverride All
 
    #
    # Controls who can get stuff from this server.
    #
    Order allow,deny
    Allow from all
</directory>

On Apache webservers, .htaccess (hypertext access) is the default name of directory-level configuration files. An .htaccess file is placed in a particular directory, and the directives in the .htaccess file apply to that directory, and all its subdirectories. It provides the ability to customize configuration for requests to the particular directory. In our case, enabling search engine safe (SES) URLs.

By setting the AllowOverride directive to All in effect defers configuration settings to the .htaccess file.

An example .htaccess file could include the following code to rewrite the URLs:

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php/$1 [L,QSA]

Search engine friendly URLs are implemented with Rewrite engines. The rewrite engine modifies the URL based upon a number of rewrite conditions and rules.

The RewriteBase directive explicitly sets the base URL for per-directory rewrites. The RewriteCond directive defines a rule condition, so in this case handling missing files or directories. Finally, the RewriteRule directive is the real rewriting workhorse. In this example, we’re getting everything in the URI — i.e. not including the protocol (HTTP/S) and domain name — based upon a regular expression. This is then appended to the default file reference — index.php — as a back reference. The [L,QSA] refers to the rule being the last rule and append any query string parameters to the default file. It is important to note that this is all done on the server side, the user will never see the website address changing in the browser’s address bar. Furthermore, simply transposing the index.php filename with your default file name — e.g. index.cfm, default.aspx — will have the same result. Indeed, the above rewrite rules are becoming a de-facto standard for web applications.

To fully understand mod_rewrite rules above, look at the Apache mod_rewrite documentation.

Once you have your SES functionality in place on the webserver, it is then the responsibility of your application framework to understand the URL construction and handle it accordingly. Fortunately, frameworks such as ColdBox and Fusebox for ColdFusion, Zend and Symfony for PHP, all contain functionality to do this, but that is the subject of an entirely different post.

Users of web applications prefer short, neat URLs to raw query string parameters. A concise URL is easy to remember, and less time-consuming to type in. If the URL can be made to relate clearly to the content of the page, then errors are not only less likely to happen, but our good friends the search engine robots are able to draw a stronger assumption of the pages’ relevance and content.

To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file. Conversely and importantly, webmasters can also notify the search engines about the existence and importance of pages with a sitemap.xml file. (Both files are placed in the root directory of the domain.)

Fortunately for the webmaster, the major search engines provide various tools to help manage both Sitemap and Robot files.

To gain an understanding of both ‘protocols’, I’ll discuss them briefly below.

Sitemaps (Inclusion Protocol)

The Sitemaps protocol allows a webmaster to inform search engines about URLs on a website that are available for crawling. A Sitemap is an XML file that lists the URLs for a site. It allows webmasters to include additional information about each URL: when it was last updated, how often it changes, and how important it is in relation to other URLs in the site. This allows search engines to crawl the site more intelligently. Sitemaps are a URL inclusion protocol and complement robots.txt, a URL exclusion protocol.

The webmaster can generate a Sitemap containing all accessible URLs on the site and submit it to search engines. Since Google, MSN, Yahoo!, and Ask use the same protocol now, having a Sitemap would let the biggest search engines have the updated pages information.

Sitemaps supplement and do not replace the existing crawl-based mechanisms that search engines already use to discover URLs. By submitting Sitemaps to a search engine, a webmaster is only helping that engine’s crawlers to do a better job of crawling their site(s). Using this protocol does not guarantee that web pages will be included in search indexes, nor does it influence the way that pages are ranked in search results.

The following is a cut-down version of the sitemap.xml for this website. WordPress, via a plugin, automatically updates this file each time a new post or page is written.

<urlset xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
<url>
<loc>http://www.simonwhatley.co.uk/</loc>
<lastmod>2008-10-08T14:50:16+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>
http://www.simonwhatley.co.uk/big-city-little-people
</loc>
<lastmod>2008-10-08T14:50:16+00:00</lastmod>
<changefreq>monthly</changefreq>
<priority>0.1</priority>
</url>
</urlset>

More information about sitemaps can be found on the Sitemaps.org website.

Robots (Exclusion Protocol)

The robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorise and archive web sites. The standard complements Sitemaps, a robot inclusion standard for websites.

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorisation of the site as a whole.

The protocol, however, is purely advisory. It relies on the cooperation of the web robot, so that marking an area of a site out of bounds with robots.txt does not guarantee privacy. Some web site administrators have tried to use the robots file to make private parts of a website invisible to the rest of the world, but the file is necessarily publicly available and its content is easily checked by anyone with a web browser.

For example, the following tells all crawlers not to enter four directories of a website:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/

Exclusion can also be achieved on a page-level basis using a Meta-tag. This is a tag that would be placed in the HTML head of of a web page. The robots attribute controls whether search engine spiders are allowed to index a page, or not, and whether they should follow links from a page, or not.

A common example could be as follows:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" dir="ltr" lang="en-GB" xml:lang="en">
 <head profile="http://gmpg.org/xfn/11">
	<title>Simon Whatley</title>
	<meta http-equiv="robots" content="index,follow" />
</head>
<body>
</body>
</html>

A word of caution though, Meta tags are not the best option to prevent search engines from indexing content of your website.

More information about Robots.txt files can be found on the Robotstxt.org website.

Webmaster Tools

The top 3 search providers all have their own webmaster tools admin interface. The Google offering is the most advanced, but it’s good practice to use and submit information to all three.

Links to their services are provided below:

• Google Webmasters
• Yahoo! Site Explorer
• Live Search Webmaster Centre

Ask doesn’t have an interface. However, you can still ping their Submission Service using the URL http://submissions.ask.com/ping?sitemap= in conjunction with your sitemap URL.

Further Information

• Sitemap Generator
• Sitemap 3rd Party porgrams and websites
• Webmaster Toolkit