Search Engine's seem to get "stuck" on my site... · Blog
There are various places in which search engines can get stuck, such as the calendar (as it increases indefinitely); there are some robots.txt entries that you can add such as:
robots.txt
Be sure to edit the path to your forums from your web root; mine for example is /forum but yours may vary.
Several items in this list aren't just bot traps, but places where SE's shouldn't be in the first place as it will just show an access denied error (as a SE is just a guest).
These include:
new user/registration page
Mark all Read
Clear cookies
View privacy
FAQ
search
who's online
show profile (you can remove this setting if you want SE's trolling your member data)
show members
calendar (including show day's events)
Posted on June 22, 2007
robots.txt
Code
User-agent: *
Disallow: /forum/ubbthreads.php?ubb=calendar
Disallow: /forum/ubbthreads.php/ubb/calendar
Disallow: /forum/ubbthreads.php/ubb/calendar.html
Disallow: /forum/ubbthreads.php?ubb=showday
Disallow: /forum/ubbthreads.php/ubb/showday
Disallow: /forum/ubbthreads.php/ubb/showday.html
Disallow: /forum/ubbthreads.php?ubb=showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile
Disallow: /forum/ubbthreads.php/ubb/showprofile.html
Disallow: /forum/ubbthreads.php?ubb=showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers
Disallow: /forum/ubbthreads.php/ubb/showmembers.html
Disallow: /forum/ubbthreads.php?ubb=online
Disallow: /forum/ubbthreads.php/ubb/online
Disallow: /forum/ubbthreads.php/ubb/online.html
Disallow: /forum/ubbthreads.php?ubb=search
Disallow: /forum/ubbthreads.php/ubb/search
Disallow: /forum/ubbthreads.php/ubb/search.html
Disallow: /forum/ubbthreads.php?ubb=faq
Disallow: /forum/ubbthreads.php/ubb/faq
Disallow: /forum/ubbthreads.php/ubb/faq.html
Disallow: /forum/ubbthreads.php?ubb=viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy
Disallow: /forum/ubbthreads.php/ubb/viewprivacy.html
Disallow: /forum/ubbthreads.php?ubb=mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies
Disallow: /forum/ubbthreads.php/ubb/mycookies.html
Disallow: /forum/ubbthreads.php?ubb=markallread
Disallow: /forum/ubbthreads.php/ubb/markallread
Disallow: /forum/ubbthreads.php/ubb/markallread.html
Disallow: /forum/ubbthreads.php?ubb=newuser
Disallow: /forum/ubbthreads.php/ubb/newuser
Disallow: /forum/ubbthreads.php/ubb/newuser.htmlBe sure to edit the path to your forums from your web root; mine for example is /forum but yours may vary.
Several items in this list aren't just bot traps, but places where SE's shouldn't be in the first place as it will just show an access denied error (as a SE is just a guest).
These include:
new user/registration page
Mark all Read
Clear cookies
View privacy
FAQ
search
who's online
show profile (you can remove this setting if you want SE's trolling your member data)
show members
calendar (including show day's events)
Posted on June 22, 2007
