Blocking Rogue Bots in IIS7

Some crawlers, bots and spiders choose to ignore a robots.txt file causing high CPU usage and bandwidth for your hosted app.

You can block them using the IIS7 URL Rewrite 2 (download) module.

User-agent blocking can be configured per site (answer on SO) or per server.

For per-site blocking, simply add to the <configuration> section of your web.config:

      <rule name="Rule1" stopProcessing="true">
        <match url=".*" />
          <add input="{HTTP_USER_AGENT}" pattern="MyNastySpiderName" />
        <action type="CustomResponse" statusCode="403"
          statusReason="Forbidden: Access is denied."
          statusDescription="You do not have permission to view this page." />

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: