Robots.txt for WordPress MU

This is a Robots Exclusion Standard (robots.txt) file which is ideal for WordPress MU.

File: robots.txt

# This rule means it applies to all user-agents
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/

Disallow: error_log
Disallow: index-install.php
Disallow: wp-activate.php
Disallow: wp-atom.php
Disallow: wp-blog-header.php
Disallow: wp-comments-post.php
Disallow: wp-commentsrss2.php
Disallow: wp-config.php
Disallow: wp-cron.php
Disallow: wp-feed.php
Disallow: wp-forum.phps
Disallow: wp-links-opml.php
Disallow: wp-login.php
Disallow: wp-mail.php
Disallow: wp-pass.php
Disallow: wp-rdf.php
Disallow: wp-rss.php
Disallow: wp-rss2.php
Disallow: wp-settings.php
Disallow: wp-trackback.php
Disallow: wpmu-cleanup.php
Disallow: wpmu-settings.php
Disallow: xmlrpc.php

== Alternatively ==

{{Config file|robots.txt|
<pre>
# Disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.tar$
Disallow: /*.tgz$
Disallow: /*.cgi$
Disallow: /*.xhtml$

Special bots

Add this if you site uses Google Adsense:

File: robots.txt

# This is the ad bot for google
User-agent: Mediapartners-Google*
 
# Allow Everything
Disallow: 

See also

• Robots.txt for WordPress MU