Blocking Rogue Bots in IIS7

Some crawlers, bots and spiders choose to ignore a robots.txt file causing high CPU usage and bandwidth for your hosted app.

You can block them using the IIS7 URL Rewrite 2 (download) module.

User-agent blocking can be configured per site (answer on SO) or per server.

For per-site blocking, simply add to the <configuration> section of your web.config:

<system.webServer>
  <rewrite>
    <rules>
      <rule name="Rule1" stopProcessing="true">
        <match url=".*" />
        <conditions>
          <add input="{HTTP_USER_AGENT}" pattern="MyNastySpiderName" />
        </conditions>
        <action type="CustomResponse" statusCode="403"
          statusReason="Forbidden: Access is denied."
          statusDescription="You do not have permission to view this page." />
        </rule>
      </rules>
  </rewrite>
</system.webServer>

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: