Floris
24th February 2005, 10:29 AM
[Indepth Guide] The proper usage of the robots.txt file
I found this great guide and got permission to publish it on our web site, I think a lot of users could benefit from learning how to use robots.txt in order to prevent overkill in traffic, search bots. To block spammers and hackers, and increase their performance. More vBulletin specific information regarding robots.txt can be found here (http://www.vbulletin.com/forum/showthread.php?t=45828).
Author: Jimmy Whisenhunt
Web site: http://www.vipenterprises.org (http://www.vipenterprises.org/)
The proper usage of the robots.txt file
When optimizing your web site most webmasters don't consider using the robots.txt file. This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder. The robots.txt file goes in the root directory of the website.
Here is a list of variables that you can include in a robots.txt file and there meaning:
User-agent: In this field you can specify a specific robot to describe access policy for or a "*" for all robots more explained in example.
Disallow: In the field you specify the files and folders not to include in the crawl.
The # is to represent comments
Here are some examples of a robots.txt file
User-agent: *
Disallow:
The above would let all spiders index all content.
Here another:
User-agent: *
Disallow: /cgi-bin/
The above would block all spiders from indexing the cgi-bin directory.
User-agent: googlebot
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /stats/
In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.
Jimmy Whisenhunt is the webmaster http://www.vipenterprises.org (http://www.vipenterprises.org) VIP Enterprises
Copyright ©2005 Jimmy Whisenhunt, http://www.vipenterprises.org (http://www.vipenterprises.org/)
You can discuss this indepth guide here (http://www.vBulletin-Fans.com/showthread.php?t=9571).
Floris & Staff
I found this great guide and got permission to publish it on our web site, I think a lot of users could benefit from learning how to use robots.txt in order to prevent overkill in traffic, search bots. To block spammers and hackers, and increase their performance. More vBulletin specific information regarding robots.txt can be found here (http://www.vbulletin.com/forum/showthread.php?t=45828).
Author: Jimmy Whisenhunt
Web site: http://www.vipenterprises.org (http://www.vipenterprises.org/)
The proper usage of the robots.txt file
When optimizing your web site most webmasters don't consider using the robots.txt file. This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder. The robots.txt file goes in the root directory of the website.
Here is a list of variables that you can include in a robots.txt file and there meaning:
User-agent: In this field you can specify a specific robot to describe access policy for or a "*" for all robots more explained in example.
Disallow: In the field you specify the files and folders not to include in the crawl.
The # is to represent comments
Here are some examples of a robots.txt file
User-agent: *
Disallow:
The above would let all spiders index all content.
Here another:
User-agent: *
Disallow: /cgi-bin/
The above would block all spiders from indexing the cgi-bin directory.
User-agent: googlebot
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /stats/
In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.
Jimmy Whisenhunt is the webmaster http://www.vipenterprises.org (http://www.vipenterprises.org) VIP Enterprises
Copyright ©2005 Jimmy Whisenhunt, http://www.vipenterprises.org (http://www.vipenterprises.org/)
You can discuss this indepth guide here (http://www.vBulletin-Fans.com/showthread.php?t=9571).
Floris & Staff