Articles tagged web crawlers

Optimise Your URLs for Web Crawlers and Indexing
Many questions about website architecture, crawling and indexing, and even ranking issues can be boiled down to one central issue: How easy is it for search engines to crawl your site? Read more – ‘Optimise Your URLs for Web Crawlers and Indexing’.
What's In Google Chrome's User-Agent String
With the advent Google Chrome there has been a lot of media coverage regarding the browser’s uptake and how it will compete with Internet Explorer, Firefox and Safari. This is where the User Agent becomes most valuable. Read more – ‘What's In Google Chrome's User-Agent String’.