You can find all you need to know about robots at this website http://www.robotstxt.org/wc/robots.html
Here are some basic principles to follow.
User-agent: googlebot This is the name of the robot. There are many different user agents.
User-agent: * The ‘*’ is used to represent all user-agents i.e. any.
disallow: / This function will disallow robots access to any files
disallow: This function will allow all robots access to anything.
disallow: /v3/index.htm This function will disallow robots access to the specific /v3/index.htm directory on the server.
For example this would allow all robots access to everywhere.
User-agent: *
disallow:
|