Many questions about website architecture, crawling and indexing, and even ranking issues can be boiled down to one central issue: How easy is it for search engines to crawl your site?
Tags: Canonical, Google, Index, Information retrieval, Microsoft, Robots exclusion standard, search engine, search engine optimisation, search engines, url, URL redirection, Web archiving, web crawlers, Web search engine, webmaster, world wide web, Yahoo
Carpe diem on any duplicate content worries: Google, Yahoo and Microsoft now support a format that allows you to publicly specify your preferred version of a URL. If your site has identical or vastly similar content that’s accessible through multiple URLs, this format provides you with more control over the URL returned in search results. It also helps to make sure that properties such as link popularity are consolidated to your preferred version.
Tags: Canonical, Duplicate content, Google, Microsoft, search engine, search engine optimisation, Search engine optimization, search engines, search results, SEO, Uniform Resource Identifier, url, web application, Yahoo
An increasingly popular technique among websites and in particular, blogs, is the idea of making URLs search engine friendly, or safe, on the premise that doing so will help search engine optimisation. By removing the obscure query string element of a URL and replacing it with keyword rich alternatives, not only makes it more readable for a human being, but also the venerable robots that allow our page content to be found in the first place.
Tags: .htaccess, All, Apache, ColdBox, ColdFusion, Fusebox, HTTP, httpd.conf, ISAPI, Microsoft, mod_rewrite, New Brunswick, None, PHP, search engine, search engine optimisation, search engine robots, search engine safe, url, URL rewriting, USD, web applications
The first step to increasing your site’s visibility on the top search engines such as Google, Yahoo! and MSN is to help their respective robots crawl and index your site. To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file. Conversely and importantly, webmasters can also notify the search engines about the existence and importance of pages with a sitemap.xml file
Tags: Google, Microsoft, optimisation, sitemaps, Tools, webmasters, website, Yahoo