Googlebot POSTS – using jQuery

After I came up with the idea to log web application hits using jQuery, to my great surprise I found that Googlebot actually performs POSTs implemented as jQuery $.ajax() calls:

2014-01-15 09:46:04 POST /Log - - 66.249.64.45 
  Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 255

Wow!

Searching the Interwebs I found other people who observed this behavior, too:

Most importantly, the links in Wikipedia’s Googlebot article analyze the bot’s behavior in more detail:

The articles are about 2 years old, so the bot may now be even more capable than then.

Of course, the simplest solution to prevent bots from POSTing is to add the logger’s URL to robots.txt:

User-agent: *
Disallow: /Log

 

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 67 other followers

%d bloggers like this: