Innovation Treadmill

The longer a software product exists and is being actively developed, the more changes are going to affect its development.

The kinds of changes are various, ranging from new technology, new development tools, new development paradigms and trends, to changing requirements and solutions, or complete shift of focus.

I’ll just summarize some of these changes that may apply to my projects (and probably yours as well):

Old New
Framework ASP.Net ASP.Net MVC
PostBack Ajax, In-page update
Client-side scripting Ajax Control Toolkit jQuery
Javascript TypeScript
Data Access SqlCommand (aspx, C#) Data Access Layer
XML Generation StringBuilder XSD, xsd, XmlSerializer
Parser DIY NIH Antlr
Focus (e.g. VSSlnVis) Graphical representation of dependencies Textual analysis of solutions and projects
(e.g. dbscript) Generate C# constants, Generate CREATE scripts database versioning, documentation and deployment
Patterns? Anti-Patterns, God Objects, Spaghetti Patterns, Lasagna, Ravioli

If your project finds itself in a sort of identity crisis, then don’t worry. Companies and organizations bigger than yours often recognize the need to re-write:

KDE did it, Mozilla did it, phpBB did it, and Google just forked Webkit into Blink to be used in Chrome.

So I came to the conclusion that to increase maintainability and extensibility of the programs and projects that I still update (some more frequently, others less), I need to rework their code.

As I prefer my principle of the least technological requirements, I am still not sure whether to stick with .Net 2 for my WinForms apps, or whether .Net 4 is already acceptable (i.e. does not require the user to separately install specific versions of .Net).

Please leave a comment with your ideas 😉

The Evolution of Application Architecture

When I started developing ASP.Net applications (.Net 1.1, around 2004), the world of application development looked simple:

Application connecting to a database and the “universe” (anything it needs to interface to)

(Before that, I developed in mainly Delphi, also in Oracle Forms or ASP/VB, and it always looked like this)

The web application would connect directly to the (SQL Server) database given the support of Visual Studio designers, query builders, etc., and the semi-automatic connection between parameters of database statements and the values of query string parameters, control properties, and so on.

Of course, the “simple” approach had some drawbacks, and I even created a tool called graspx to work around the drawbacks (rather than changing the approach, as I realized later), such as finding the SQL statements retrieving and writing data to figure out database dependencies.

The first step away from the direct application/database connection is to generate a code model based on the database model

Application using a code model representing the database model

Since the code model is derived from the database model, changes in the database model immediately cause the compilation to break if tables or fields are deleted.

However, the application is still monolithic, which restricts the ability to write unit and module tests for each piece of functionality (especially if you chose to develop ASP.Net web sites rather than web applications).

So, let’s separate the business logic from the user interface part (this can be done in classic ASP.Net as well, but it’s trickier. In ASP.Net MVC, you are forced strongly encourage to follow this pattern).

Application using a business layer implementing functionality, communicating with the database via a ORM, and interfacing to other environments using separate libraries.

The business layer may encapsulate various functionality, such as communicating with systems other than the database, handling different data, interfacing to other systems, and so on. We extract the interfacing code to separate assemblies for each system, and route the calls through the business layer assembly.

Usually, the front end application is not the single application accessing the database. You need to add a service that executes asynchronous operations, or write an import or export tool, allow batch processing and so on. These applications do not need the full functionality of the existing business layer (with all its dependencies on other systems), so we create a separate business layer assembly which only provides the data model that application requires:

Application using a business layer implementing functionality, communicating with the database via a ORM, and interfacing to other environments using separate libraries.

This architecture is not even restricted to a certain kind of applications (in my case, web applications with a certain database), you can also apply it to other scenarios:

CRM applications using a business layer implementing functionality, communicating with CRM database via the CRM data classes, and interfacing to other environments using separate libraries.

From my work on CRM projects, I noticed that these projects tend to look like charts 1 and 2. If you start out with the separation of layers and responsibilities, you can easily get a nice architecture, even if it is not obvious from the beginning.

Rover Code

If you were ever curious about the software driving the Mars rovers, I came across a good question on Programmers today.

The JPL LaRS lab provides a couple of documents on coding standards and monitoring, and NODIS has a document on Software Engineering Requirements. The Workshops on Spacecraft Flight Software also offers a couple of slideshow presentations.

I find this topic very interesting (how would you debug a piece of code that is executing on a different planet?), but unfortunately there are not that many related questions (and answers!) on Programmers.

What about the Soviet space program?

Evolving Architecture for Legacy Applications

I started developing database applications before the idea of using data access layers or ORMs became mainstream. If you happened to develop under Delphi or Visual Studio in the early days, your application architecture may look something like this:

There is a database (or any kind of data store, such as files or XML or whatever), and an application that operates on the data.

Any change to the database structure cause a mess in the application code:

  • where was a now-deleted table or field accessed or processed
  • which views and procedures are accessed by the code
  • some business logic implemented as stored procedures, some as application code

Time to separate implicit database access from application code (I’m looking at you, DataGrid, GridView and FormView with your magic DataSources!)

Enter the data access layer, which provides the application code (in my case, C#) a type-safe interface to the database:

This gets rid of the data usage and access problem: changes to the database structure are mirrored into the data access layer, and will cause compiler errors if a table or field has been removed or its data type has changed.

Still, the problem of implementing business functions in the database or in code remains, and the application needs to be aware or where the function is located.

The business layer encapsulated business logic wherever it is implemented, calling either business function in C# (object-oriented) or in the database (taking advantage of set-based operations or database-specific functionality). The application exchanges data with the business layer using the data contract classes.

As I mainly develop web applications, I notice that the object-oriented model occasionally breaks down in the front-end, since HTML is basically all-strings, and you need to parse and re-create your data access objects based on HTML data. Further, if you need to let your users edit hierarchically-structured data, you’ll soon get lost retrieving all your hierarchical business objects.

A user interface functions layer should handle the complexity of translating business objects to and from user interface objects:

Once you use a UI functions layer, it’s up to your aesthetic preferences whether you let the application still access the business data contracts, or pass everything through the UI data contracts. It is one additional step of mapping data, and you have been warned 😉

Notice that, although I have a strong background in MS development tools (Visual Studio, SQL Server), I tried to keep the ideas and diagrams as technology-agnostic as possible. It does not matter whether your business functions reside in a web service (of any kind) or just in a separate library (assembly), or what database (if any) you use.