After I came up with the idea to log web application hits using jQuery, to my great surprise I found that Googlebot actually performs POSTs implemented as jQuery $.ajax() calls:
2014-01-15 09:46:04 POST /Log - - 66.249.64.45 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 255
Wow!
Searching the Interwebs I found other people who observed this behavior, too:
- Does googlebot crawl urls in jQuery $.get() calls and can it be prevented?
- Prevent Googlebot from running a function
- Googlebot doesn’t see jquery generated content
- Stopping googlebot from following links in javascript
Most importantly, the links in Wikipedia’s Googlebot article analyze the bot’s behavior in more detail:
- Googlebot makes POST requests via AJAX
- Googlebot Is Chrome
- Google… Stop Playing; The Jig Is Still Up!
- Googlebot’s Javascript Interpreter: A Diagnostic
The articles are about 2 years old, so the bot may now be even more capable than then.
Of course, the simplest solution to prevent bots from POSTing is to add the logger’s URL to robots.txt:
User-agent: * Disallow: /Log
devioblog 