QUOTE(playerseeker @ Dec 1 2009, 10:35 PM)
hi guys..i've did some searching in the net about robot.txt
i found the generator and it generates out there
# robots.txt generated at http://www.mcanerin.com
User-agent: *
Disallow:
Disallow: /cgi-bin/
meaning i allow all the spider go to my website??or i need something like
"user-agent:googlebot
allow:/"
and can some one teach me how to write the proper robot.txt command
something like
User-agent:
Allow:/*
This will do if you want all spiders to index all your pages.i found the generator and it generates out there
# robots.txt generated at http://www.mcanerin.com
User-agent: *
Disallow:
Disallow: /cgi-bin/
meaning i allow all the spider go to my website??or i need something like
"user-agent:googlebot
allow:/"
and can some one teach me how to write the proper robot.txt command
something like
User-agent:
Allow:/*
User-agent: *
Allow: /
Dec 2 2009, 10:54 AM

Quote
0.0141sec
0.61
7 queries
GZIP Disabled