Scribd
Upload
16 views

Robots

The document lists URLs and file types that should be disallowed for various web crawlers and bots to access, including sleep, AVI, MP3, DVD, OST files for all bots and JPG, AVI, MPG, MOV, ISO, PDF files specifically for MSN bot.

Uploaded by

Eduardo Correia
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
16 views

Robots

The document lists URLs and file types that should be disallowed for various web crawlers and bots to access, including sleep, AVI, MP3, DVD, OST files for all bots and JPG, AVI, MPG, MOV, ISO, PDF files specifically for MSN bot.

Uploaded by

Eduardo Correia
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 1

User-agent: *

Disallow: /sleep/*
Disallow: *.avi
Disallow: *.mp3
Disallow: *DVD*
Disallow: *DvD*
Disallow: *dvd*
Disallow: *OST*

User-agent: msnbot
Disallow: *.jpg
Disallow: *.JPG
Disallow: *.avi
Disallow: *.mpg
Disallow: *.mov
Disallow: *.iso
Disallow: *.pdf

You might also like