Googlebot POSTS – using jQuery

After I came up with the idea to log web application hits using jQuery, to my great surprise I found that Googlebot actually performs POSTs implemented as jQuery $.ajax() calls:

2014-01-15 09:46:04 POST /Log - - 
  Mozilla/5.0+(compatible;+Googlebot/2.1;++ - 200 0 0 255


Searching the Interwebs I found other people who observed this behavior, too:

Most importantly, the links in Wikipedia’s Googlebot article analyze the bot’s behavior in more detail:

The articles are about 2 years old, so the bot may now be even more capable than then.

Of course, the simplest solution to prevent bots from POSTing is to add the logger’s URL to robots.txt:

User-agent: *
Disallow: /Log


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.