Scribd
Upload
49 views

Robots - TXT File # # To Ban All Spiders From The Entire Site Uncomment The Next Two Lines: # User-Agent: # Disallow

The document provides instructions on how to use a robots.txt file to control access to a website by web robots and spiders. It gives an example of banning all spiders from an entire site by uncommenting lines that disallow any robot from accessing the site denoted by the forward slash.

Uploaded by

Vassay Khalili
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
49 views

Robots - TXT File # # To Ban All Spiders From The Entire Site Uncomment The Next Two Lines: # User-Agent: # Disallow

The document provides instructions on how to use a robots.txt file to control access to a website by web robots and spiders. It gives an example of banning all spiders from an entire site by uncommenting lines that disallow any robot from accessing the site denoted by the forward slash.

Uploaded by

Vassay Khalili
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 1

# See http://www.robotstxt.org/wc/norobots.

html for documentation on how to use the


robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /

You might also like