Comments on: Optimise Your URLs for Web Crawlers and Indexing http://www.simonwhatley.co.uk/optimise-your-urls-for-web-crawlers-and-indexing The opposite of every great idea is another great idea Fri, 03 Sep 2010 14:27:00 +0000 hourly 1 http://wordpress.org/?v=3.0.1 By: Simon http://www.simonwhatley.co.uk/optimise-your-urls-for-web-crawlers-and-indexing/comment-page-1#comment-423 Simon Fri, 09 Oct 2009 09:31:20 +0000 http://www.simonwhatley.co.uk/?p=2011#comment-423 @todd you do have a good point, albeit to a certain degree we should help the "good" crawlers out there and ignore the "bad". Having said that, it is probably better preventing lists and calendars scrolling infinitely by removing links and preventing URL hacking, since the "bad" crawlers could put unneccessary load on your servers. @todd you do have a good point, albeit to a certain degree we should help the “good” crawlers out there and ignore the “bad”.

Having said that, it is probably better preventing lists and calendars scrolling infinitely by removing links and preventing URL hacking, since the “bad” crawlers could put unneccessary load on your servers.

]]> By: Todd Rafferty http://www.simonwhatley.co.uk/optimise-your-urls-for-web-crawlers-and-indexing/comment-page-1#comment-422 Todd Rafferty Thu, 08 Oct 2009 23:20:40 +0000 http://www.simonwhatley.co.uk/?p=2011#comment-422 Regarding 'Stop infinite pagination in, for example, lists and calendars.' - This is assuming the robots respect the rel="nofollow" links - there's a lot of bad robots on the web right now, some that don't respect the robots.txt either. Regarding ‘Stop infinite pagination in, for example, lists and calendars.’ – This is assuming the robots respect the rel=”nofollow” links – there’s a lot of bad robots on the web right now, some that don’t respect the robots.txt either.

]]>