How do I BLOCK search engines? · Blog
Well, there are several options if you're in the boat that you want to block Search Engines from crawling your forums all together.
A good "first step" is to disable SE Friendly URLs, myself I don't like doing this, it's much easier on the eyes and the urls look ugly.
Please note however that this will NOT block search engines, it'll simply make it more difficult for them to crawl your content; so it's not an end-all be-all suggestion.
If the Search Engine respects robots.txt standards, it's even easier (most non-abusive search engines support this), just edit (or add) your robots.txt file (in your web root) and add the following:
Be sure however to update /forum with whatever your forum's location is (from your domain's web root); so if your forum is in http://www.ubbcentral.com/forums/ you'd enter "/forums" (no trailing slash).
Posted on June 22, 2007
A good "first step" is to disable SE Friendly URLs, myself I don't like doing this, it's much easier on the eyes and the urls look ugly.
Please note however that this will NOT block search engines, it'll simply make it more difficult for them to crawl your content; so it's not an end-all be-all suggestion.
If the Search Engine respects robots.txt standards, it's even easier (most non-abusive search engines support this), just edit (or add) your robots.txt file (in your web root) and add the following:
Code
User-agent: *
Disallow: /forumBe sure however to update /forum with whatever your forum's location is (from your domain's web root); so if your forum is in http://www.ubbcentral.com/forums/ you'd enter "/forums" (no trailing slash).
Posted on June 22, 2007
