Googlebot POSTS – using jQuery

January 22, 2014

After I came up with the idea to log web application hits using jQuery, to my great surprise I found that Googlebot actually performs POSTs implemented as jQuery $.ajax() calls:

2014-01-15 09:46:04 POST /Log - - 66.249.64.45 
  Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - 200 0 0 255

Wow!

Searching the Interwebs I found other people who observed this behavior, too:

Most importantly, the links in Wikipedia’s Googlebot article analyze the bot’s behavior in more detail:

The articles are about 2 years old, so the bot may now be even more capable than then.

Of course, the simplest solution to prevent bots from POSTing is to add the logger’s URL to robots.txt:

User-agent: *
Disallow: /Log

 

Advertisements

Innovation Treadmill

April 30, 2013

The longer a software product exists and is being actively developed, the more changes are going to affect its development.

The kinds of changes are various, ranging from new technology, new development tools, new development paradigms and trends, to changing requirements and solutions, or complete shift of focus.

I’ll just summarize some of these changes that may apply to my projects (and probably yours as well):

Old New
Framework ASP.Net ASP.Net MVC
PostBack Ajax, In-page update
Client-side scripting Ajax Control Toolkit jQuery
Javascript TypeScript
Data Access SqlCommand (aspx, C#) Data Access Layer
XML Generation StringBuilder XSD, xsd, XmlSerializer
Parser DIY NIH Antlr
Focus (e.g. VSSlnVis) Graphical representation of dependencies Textual analysis of solutions and projects
(e.g. dbscript) Generate C# constants, Generate CREATE scripts database versioning, documentation and deployment
Patterns? Anti-Patterns, God Objects, Spaghetti Patterns, Lasagna, Ravioli

If your project finds itself in a sort of identity crisis, then don’t worry. Companies and organizations bigger than yours often recognize the need to re-write:

KDE did it, Mozilla did it, phpBB did it, and Google just forked Webkit into Blink to be used in Chrome.

So I came to the conclusion that to increase maintainability and extensibility of the programs and projects that I still update (some more frequently, others less), I need to rework their code.

As I prefer my principle of the least technological requirements, I am still not sure whether to stick with .Net 2 for my WinForms apps, or whether .Net 4 is already acceptable (i.e. does not require the user to separately install specific versions of .Net).

Please leave a comment with your ideas 😉


Viewing Large Text and Log Files

February 20, 2013

Log files can become quite big, and if you generate scripts of type Data in SSMS, the script files easily reach 1GB or more in size.

But which viewer or editor can you use to open a 20GB text file? Certainly not Notepad, and even Notepad++ struggles and is unresponsive for a long time.

BareTail

BareTail is a log file viewer with a nice “Follow Tail” option which allows you to watch web server and other log files while they are written.

Pros: Line coloring based on text in lines. Updated files are marked in the tabs. Save config to file.

Cons: Pre-UAC application, so saving the configuration fails if it is installed under C:\Program Files\. Cannot copy large texts to clipboard. No Search.

Large Text File Viewer

Large Text File Viewer is a text file viewer. Opens large text files immediately, and loads the file in the background without blocking the UI.

Searching for text displays a dialog with the estimated search time, which is about 1 hour for 20GB or 75M lines.

Support for Unicode, but no UTF-8.

V

The V File Viewer displays files in text and in hex mode which is great for analyzing large binary files, or detecting the encoding of large text files.


What happens to Your Data when the Cloud starts to rain?

August 17, 2012

“The Cloud” and Cloud Computing in general are the latest hype in IT. However, the news that made the headlines in recent months give cause to worry.

As you put your data into the cloud, how will you regain control over the data again? Who else has access to your data? What happens to your data once the cloud infrastructure fails, and who is responsible (under what terms) to restore data and your access?

Just a couple of news articles on recent outages and privacy failures:

Microsoft Azure (Feb ’12)

Microsoft’s Azure cloud down and out for 8 hours

Amazon (June ’12)

Amazon cloud knocked out by violent storms in Virginia

Instagram and Netflix back online after Amazon cloud outage

Bad generator and bugs take out Amazon cloud

RavenHQ & Amazon EC2 Outage

Salesforce (July ’12)

Salesforce goes titsup, causes CRM outages worldwide

Microsoft Azure (July ’12)

Microsoft Azure goes titsup across Western Europe

Twitter (July ’12)

Never mind Azure: They BROKE Twitter!

Twitter titsup: Our failover was actually just FAIL ALL OVER

Giacom (July ’12)

Cloudy emails up in smoke for FIVE days after fire knackers Giacom

Mat Honan (Aug ’12)

How Apple and Amazon Security Flaws Led to My Epic Hacking

Yes, I was hacked. Hard.

Scribe’s mobe, MacBook pwned after hacker ‘fast-talked Apple support’

Amazon exploited by hacker in scribe’s epic Apple iCloud pwn

Amazon Boosts Security After Journalist Hack

After Epic Hack, Apple Suspends Over-the-Phone AppleID Password Resets

Apple, Amazon, close password door after horse bolts

iCloud et al (Aug ’12)

Ausfälle bei iCloud, FaceTime, iMessages und iTunes Store

Prime Hosting (Aug ’12)

Hundreds of websites go titsup in Prime Hosting disk meltdown

Wikipedia (Aug ’12)

Wikipedia collapses threatening the very fabric of civilisation

Conclusion

I am not alone with my doubts, and other people see issues as well:

Woz: Cloud computing trend is ‘horrendous’

With the cloud, you don’t own anything. You already signed it away. I want to feel that I own things […] A lot of people feel, ‘Oh, everything is really on my computer,’ but I say: the more we transfer everything onto the web, onto the cloud, the less we’re going to have control over it.

There are alternatives though, such as Owncloud and OpenNebula. So why not give them a try?

Owncloud-Apps für Android und iOS


BKA Ransomware spreading

August 17, 2012

Just a couple of days after I removed the BKA ransomware virus from a friend’s laptop, the laptop got infected again. The result looked the same, a seemingly official warning that law enforcement detected illegal files or images on the PC, and that the machine would be unlocked after payment of a certain amount of money via payment providers.

This time, however, it was not possible to remove the virus so simply, as the symptoms were different:

  • no msconfig.dat
  • a hellomoto directory under \Users\****\AppData\Roaming\
  • the \Users\****\AppData\Local\Microsoft\Windows\ directory contains a directory named 3 or 4 digits with an executable inside

The German-language forums trojaner-board.de and botfrei.de already contained a couple of threads (here, here, here) mentioning these symptoms, but provided no help for removing them, with reformatting and re-installing as the only solution.

Yesterday Heise News reported that the malware is spreading so fast that even the FBI issued a warning about the virus. Surf carefully!


Removing the BKA Trojan

July 26, 2012

A friend of mine caught the so-called BKA Trojan, and asked me to help him remove it.

This trojan makes using Windows impossible, as it displays an official-looking statement (see sample) if connected to the Internet, and only a white empty desktop if not connected, and does not allow any user action. The only way to revert to normal is supposedly by sending money using PaySafeCard or Ukash.

The warning page (which is full of typos, even in the heading: “Investignation”) lists a couple of possible Internet crimes that have been committed and caused the “computer” to be locked by law enforcement, and unlocking is as easy as sending 100€ via the linked payment providers.

What to do?

We started Windows in command-line safe mode and started msconfig to find suspicious start-up entries, unfortunately without any obvious success.

By cd’ing and dir’ing around we found the date and time the infection took place. The temp directory C:\Users\[username]\AppData\Local\Temp contained an executable with a “funny” name (5628386cos7655422.exe), an HTML file and a couple of images.

Some removal tips mention the Shell setting in the registry, and we had another look using regedit (which can also be called from the win7 command line boot).

Navigating to

HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Winlogon

the Shell key contained the following string:

explorer.exe,C:\Users\[username]\AppData\Roaming\msconfig.dat

This seemed suspicious, as it should only contain “explorer.exe”, and nothing more.

In Explorer, I dragged the file into Notepad (size 47.104 bytes), and found that it contained the MZ and PE headers (wiki, SO), a clear sign that it was not an innocent data file, but an executable.

The next steps were pretty straight-forward: clean the Shell key to read “explorer.exe” only, remove the msconfig.dat, and reboot back to normal.

Surprisingly, the Trojan does not seem to contain any sophisticated survival code (such as copying itself all over the boot disk, planting several hooks in the registry, run a watchdog, etc) – things that can make malware removal a nightmare.


[Insert Programming Language] Bashing

July 15, 2012

Everybody who has been programming for a while hopefully has found their favorite programming language(s), framework(s) and tools. From my Category Cloud, you can easily find out where I feel most fluent and comfortable, and the “comfort zones” are constantly evolving and/or changing. (I should really add ASP.Net MVC and JavaScript to that list 😉 )

Going hand in hand is the tendency of avoiding other languages, etc., due to lack of knowledge, experience, or because they are considered technically inferior. Recently, I came across a couple of pages bashing PHP, such as on Coding Horror (again!) referring to this blog which analyses the shortcomings of that language.

I guess, while this criticism will have no impact on Real PHP Programmers ™, it should certainly influence people evaluating other languages to avoid it, since the technical reasons NOT to start a PHP project seem overwhelming. (this here deals with “loose comparison“, as documented on PHP.net, and compared to Perl)

So is PHP special? Let’s ask Google:

td>VB.Net bashing
language + “bashing” hist
PHP bashing 8.500.000
3.200.000
Powershell bashing 3.000.000 *
VisualBasic bashing 2.300.000
Delphi bashing 1.600.000
C# bashing 1.300.000
JavaScript bashing 1.200.000
VBA bashing 600.000
Java bashing 600.000
VB bashing 500.000

Powershell reports 3 million hits, but from the first look that’s mostly due to comparisons of Powershell with Bash and other shells.

Let’s have a look at databases

database + “bashing” hist
MySQL bashing 3.000.000
Oracle bashing 500.000
SQL Server bashing 300.000

MySQL, SQLite, and Postgres are difficult to compare, again because of references to Bash shell programming.

Superficially judging from the numbers, I think we have a winner, though.