# robots file for web-spiders to disallow indexing certain parts of the site User-agent: * Crawl-delay: 10 Disallow: /res/ Disallow: /files/