Googlebot POSTS – using jQuery

After I came up with the idea to log web application hits using jQuery, to my great surprise I found that Googlebot actually performs POSTs implemented as jQuery $.ajax() calls:

2014-01-15 09:46:04 POST /Log - - 66.249.64.45 
  Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 255

Wow!

Searching the Interwebs I found other people who observed this behavior, too:

• Does googlebot crawl urls in jQuery $.get() calls and can it be prevented?
• Prevent Googlebot from running a function
• Googlebot doesn’t see jquery generated content
• Stopping googlebot from following links in javascript

Most importantly, the links in Wikipedia’s Googlebot article analyze the bot’s behavior in more detail:

• Googlebot makes POST requests via AJAX
• Googlebot Is Chrome
• Google… Stop Playing; The Jig Is Still Up!
• Googlebot’s Javascript Interpreter: A Diagnostic

The articles are about 2 years old, so the bot may now be even more capable than then.

Of course, the simplest solution to prevent bots from POSTing is to add the logger’s URL to robots.txt:

User-agent: *
Disallow: /Log

 

Innovation Treadmill

The longer a software product exists and is being actively developed, the more changes are going to affect its development.

The kinds of changes are various, ranging from new technology, new development tools, new development paradigms and trends, to changing requirements and solutions, or complete shift of focus.

I’ll just summarize some of these changes that may apply to my projects (and probably yours as well):

	Old	New
Framework	ASP.Net	ASP.Net MVC
	PostBack	Ajax, In-page update
Client-side scripting	Ajax Control Toolkit	jQuery
	Javascript	TypeScript
Data Access	SqlCommand (aspx, C#)	Data Access Layer
XML Generation	StringBuilder	XSD, xsd, XmlSerializer
Parser	DIY NIH	Antlr
Focus (e.g. VSSlnVis)	Graphical representation of dependencies	Textual analysis of solutions and projects
(e.g. dbscript)	Generate C# constants, Generate CREATE scripts	database versioning, documentation and deployment
Patterns?	Anti-Patterns, God Objects, Spaghetti	Patterns, Lasagna, Ravioli

If your project finds itself in a sort of identity crisis, then don’t worry. Companies and organizations bigger than yours often recognize the need to re-write:

KDE did it, Mozilla did it, phpBB did it, and Google just forked Webkit into Blink to be used in Chrome.

So I came to the conclusion that to increase maintainability and extensibility of the programs and projects that I still update (some more frequently, others less), I need to rework their code.

As I prefer my principle of the least technological requirements, I am still not sure whether to stick with .Net 2 for my WinForms apps, or whether .Net 4 is already acceptable (i.e. does not require the user to separately install specific versions of .Net).

Please leave a comment with your ideas 😉

Viewing Large Text and Log Files

Log files can become quite big, and if you generate scripts of type Data in SSMS, the script files easily reach 1GB or more in size.

But which viewer or editor can you use to open a 20GB text file? Certainly not Notepad, and even Notepad++ struggles and is unresponsive for a long time.

BareTail

BareTail is a log file viewer with a nice “Follow Tail” option which allows you to watch web server and other log files while they are written.

Pros: Line coloring based on text in lines. Updated files are marked in the tabs. Save config to file.

Cons: Pre-UAC application, so saving the configuration fails if it is installed under C:\Program Files\. Cannot copy large texts to clipboard. No Search.

Large Text File Viewer

Large Text File Viewer is a text file viewer. Opens large text files immediately, and loads the file in the background without blocking the UI.

Searching for text displays a dialog with the estimated search time, which is about 1 hour for 20GB or 75M lines.

Support for Unicode, but no UTF-8.

V

The V File Viewer displays files in text and in hex mode which is great for analyzing large binary files, or detecting the encoding of large text files.

What happens to Your Data when the Cloud starts to rain?

“The Cloud” and Cloud Computing in general are the latest hype in IT. However, the news that made the headlines in recent months give cause to worry.

As you put your data into the cloud, how will you regain control over the data again? Who else has access to your data? What happens to your data once the cloud infrastructure fails, and who is responsible (under what terms) to restore data and your access?

Just a couple of news articles on recent outages and privacy failures:

Microsoft Azure (Feb ’12)

Microsoft’s Azure cloud down and out for 8 hours

Amazon (June ’12)

Amazon cloud knocked out by violent storms in Virginia

Instagram and Netflix back online after Amazon cloud outage

Bad generator and bugs take out Amazon cloud

RavenHQ & Amazon EC2 Outage

Salesforce (July ’12)

Salesforce goes titsup, causes CRM outages worldwide

Microsoft Azure (July ’12)

Microsoft Azure goes titsup across Western Europe

Twitter (July ’12)

Never mind Azure: They BROKE Twitter!

Twitter titsup: Our failover was actually just FAIL ALL OVER

Giacom (July ’12)

Cloudy emails up in smoke for FIVE days after fire knackers Giacom

Mat Honan (Aug ’12)

How Apple and Amazon Security Flaws Led to My Epic Hacking

Yes, I was hacked. Hard.

Scribe’s mobe, MacBook pwned after hacker ‘fast-talked Apple support’

Amazon exploited by hacker in scribe’s epic Apple iCloud pwn

Amazon Boosts Security After Journalist Hack

After Epic Hack, Apple Suspends Over-the-Phone AppleID Password Resets

Apple, Amazon, close password door after horse bolts

iCloud et al (Aug ’12)

Ausfälle bei iCloud, FaceTime, iMessages und iTunes Store

Prime Hosting (Aug ’12)

Hundreds of websites go titsup in Prime Hosting disk meltdown

Wikipedia (Aug ’12)

Wikipedia collapses threatening the very fabric of civilisation

Conclusion

I am not alone with my doubts, and other people see issues as well:

Woz: Cloud computing trend is ‘horrendous’

With the cloud, you don’t own anything. You already signed it away. I want to feel that I own things […] A lot of people feel, ‘Oh, everything is really on my computer,’ but I say: the more we transfer everything onto the web, onto the cloud, the less we’re going to have control over it.

There are alternatives though, such as Owncloud and OpenNebula. So why not give them a try?

Owncloud-Apps für Android und iOS

BKA Ransomware spreading

Just a couple of days after I removed the BKA ransomware virus from a friend’s laptop, the laptop got infected again. The result looked the same, a seemingly official warning that law enforcement detected illegal files or images on the PC, and that the machine would be unlocked after payment of a certain amount of money via payment providers.

This time, however, it was not possible to remove the virus so simply, as the symptoms were different:

• no msconfig.dat

• a hellomoto directory under \Users\****\AppData\Roaming\

• the \Users\****\AppData\Local\Microsoft\Windows\ directory contains a directory named 3 or 4 digits with an executable inside

The German-language forums trojaner-board.de and botfrei.de already contained a couple of threads (here, here, here) mentioning these symptoms, but provided no help for removing them, with reformatting and re-installing as the only solution.

Yesterday Heise News reported that the malware is spreading so fast that even the FBI issued a warning about the virus. Surf carefully!

Removing the BKA Trojan

A friend of mine caught the so-called BKA Trojan, and asked me to help him remove it.

This trojan makes using Windows impossible, as it displays an official-looking statement (see sample) if connected to the Internet, and only a white empty desktop if not connected, and does not allow any user action. The only way to revert to normal is supposedly by sending money using PaySafeCard or Ukash.

The warning page (which is full of typos, even in the heading: “Investignation”) lists a couple of possible Internet crimes that have been committed and caused the “computer” to be locked by law enforcement, and unlocking is as easy as sending 100€ via the linked payment providers.

What to do?

We started Windows in command-line safe mode and started msconfig to find suspicious start-up entries, unfortunately without any obvious success.

By cd’ing and dir’ing around we found the date and time the infection took place. The temp directory C:\Users\[username]\AppData\Local\Temp contained an executable with a “funny” name (5628386cos7655422.exe), an HTML file and a couple of images.

Some removal tips mention the Shell setting in the registry, and we had another look using regedit (which can also be called from the win7 command line boot).

Navigating to

HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Winlogon

the Shell key contained the following string:

explorer.exe,C:\Users\[username]\AppData\Roaming\msconfig.dat

This seemed suspicious, as it should only contain “explorer.exe”, and nothing more.

In Explorer, I dragged the file into Notepad (size 47.104 bytes), and found that it contained the MZ and PE headers (wiki, SO), a clear sign that it was not an innocent data file, but an executable.

The next steps were pretty straight-forward: clean the Shell key to read “explorer.exe” only, remove the msconfig.dat, and reboot back to normal.

Surprisingly, the Trojan does not seem to contain any sophisticated survival code (such as copying itself all over the boot disk, planting several hooks in the registry, run a watchdog, etc) – things that can make malware removal a nightmare.

[Insert Programming Language] Bashing

Everybody who has been programming for a while hopefully has found their favorite programming language(s), framework(s) and tools. From my Category Cloud, you can easily find out where I feel most fluent and comfortable, and the “comfort zones” are constantly evolving and/or changing. (I should really add ASP.Net MVC and JavaScript to that list 😉 )

Going hand in hand is the tendency of avoiding other languages, etc., due to lack of knowledge, experience, or because they are considered technically inferior. Recently, I came across a couple of pages bashing PHP, such as on Coding Horror (again!) referring to this blog which analyses the shortcomings of that language.

I guess, while this criticism will have no impact on Real PHP Programmers ™, it should certainly influence people evaluating other languages to avoid it, since the technical reasons NOT to start a PHP project seem overwhelming. (this here deals with “loose comparison“, as documented on PHP.net, and compared to Perl)

So is PHP special? Let’s ask Google:

td>VB.Net bashing

language + “bashing”	hist
PHP bashing	8.500.000
3.200.000
Powershell bashing	3.000.000 *
VisualBasic bashing	2.300.000
Delphi bashing	1.600.000
C# bashing	1.300.000
JavaScript bashing	1.200.000
VBA bashing	600.000
Java bashing	600.000
VB bashing	500.000

Powershell reports 3 million hits, but from the first look that’s mostly due to comparisons of Powershell with Bash and other shells.

Let’s have a look at databases

database + “bashing”	hist
MySQL bashing	3.000.000
Oracle bashing	500.000
SQL Server bashing	300.000

MySQL, SQLite, and Postgres are difficult to compare, again because of references to Bash shell programming.

Superficially judging from the numbers, I think we have a winner, though.

How I survived “S.M.A.R.T. Status: BAD. Backup and Replace”

My Windows 7 started to display the “Windows detected a hard disk problem” error message. While I first ignored it, I finally ran chkdsk /f /r on all disks just to make sure erroneous sectors would not cause the error to show again.

Unfortunately, this did not work out, and the error message became more persistent.

A look into the event log presented more and more warnings (Event ID 52, Source Disk)

The driver has detected that device <device> has predicted that it will fail. Immediately back up your data and replace your hard disk drive. A failure may be imminent.

and errors (Event ID 7, Source Disk)

The device, <device>, has a bad block.

(The Events and Errors Message Center does not even know these event IDs exist.)

Restarting the PC would stop the boot process with the message

S.M.A.R.T. Status: BAD. Backup and Replace
Press F1 to continue

I decided to finally bring myself to fix the situation, and asked teh internets how to do it. (I already fixed a dying drive once, but that was Ubuntu, and this is Windows, and it can get ugly.

The dying disk is a Seagate, so I tried the SeaTools (both Windows and DOS) because they might help resolve the errors, but they didn’t.

Fortunately, the new disk is also a Seagate, a different model but the same size as the old one, so I downloaded Seagate DiscWizard on my laptop to create a bootable CD. Unfortunately, DiscWizard setup requires Seagate disks installed in the PC, which the laptop does not have. Back to square one.

In the end, I mounted the new disk in the PC, installed DiscWizard, and started it. The selection of source and destination disks cause adrenaline levels to jump, but the selection dialog (of both disks) always displays the partitions of each selected disk, so the the likelihood of selecting both drives wrong due to poor UI is greatly reduced (see screenshots in the HowTo).

After disk copying completed, I disconnected the old disk and left the new disk connected on the same cable as during copying, booted, and, voilà, logged into my old new Windows 7.

Vintage Computing Anniversaries

The PC turned 30 last summer, and this week, the Sinclair ZX Spectrum turned 30 as well.

My personal perspective is the other way ’round: ignoring the couple of weeks I “enjoyed” the ZX81 (until the keyboard became unusable), the ZX Spectrum was my first computer that I actively programmed on (read: BASIC and tape recorder). Years later I switched to the Sinclair QL, and finally to the PC world (good-bye SuperBASIC, hello Turbo Pascal). By the way, TP’s 30th birthday is just a few months ahead.

Since then, computers have become bigger and fast enough that you can run a ZX Spectrum emulation in JavaScript in a browser on an operation system that requires a million times more disk space than the original computers had memory available!

Despite all that progress, the past is still with us, and will be so for quite some time 😉

Update: TheRegister has a special on eight-bit classic games featuring mostly the Spectrum.

User Interface Design

Comparing an Apple product, a Google product, and your product…

(When I was searching for this image, I really thought I had seen it on xkcd. It’s actually from stuffthathappens.com which is currently unavailable, so I save it for posterity 😉 )