ExpectOne and ExpectOneOrNull vs. First and FirstOrDefault

November 14, 2009

Linq provides the extension methods First() and FirstOrDefault(), which return the first object in a typed IEnumerable<T> collection, or default<T> if the collection is empty.

Sometimes you don’t only want to use the first element, but require the collection to have at most 1 element.

The methods ExpectOne and ExpectOneOrNull implement the requirement to have exactly 1 (or at most 1) element in an IEnumerable collection:

public static class LinqExtensions
{
    /*
     * ExpectOne: returns exactly one record. if 0 or >1 found then exception
     * ExpectOneOrNull: returns exactly one record or null. if >1 found then exception
     */

    public static TSource ExpectOne(this IEnumerable source)
    {
        TSource result = default(TSource);
        bool b = true;
        foreach (TSource o in source)
        {
            if (b)
                result = o;
            else
                throw new ExpectOneException("too many elements in result set. one element expected.");
            b = false;
        }
        if (b)
            throw new ExpectOneException("no element in result set. one element expected.");
        return result;
    }

    public static TSource ExpectOneOrNull(this IEnumerable source)
    {
        bool b = true;
        TSource result = default(TSource);
        foreach (TSource o in source)
        {
            if (b)
                result = o;
            else
                throw new ExpectOneException("too many elements in result set. one or no element expected.");
            b = false;
        }
        return result;
    }
}

A separate exception class makes it easier for the application code (and developers!) to handle cases where more than 1 record is returned, indicating an error in the query.

If you want to replace for calls to First(<condition>) by ExpectOne(), replace First(<condition>) by Where(<condition>).ExpectOne().

As ExpectOne() contains compiled code, you cannot use these methods in Compiled Queries.

This may seem as a disadvantage. In reality it makes program code simpler because compiled queries are now required to always return an IEnumerable<T>, and the calling code needs to specify the expected result set size of the query.

It may seem desirable to replace all occurrences of First() and FirstOrDefault() by ExpectOne() and ExpectOneOrNull(). There are cases where original Linq methods are still necessary and valid:

Any() to check whether a collection is not empty, i.e. elements exist that match criteria

First() and FirstOrDefault() to find the first element in a sorted collection (e.g. the greatest element less than the original value)

 


Profiling .Net Applications

November 1, 2009

I am working on replacing the current (ugly) SQL parser of dbscript by a nice grammar-based parser as outlined in previous posts. That part of the project has been idle for some months now, but it’s time to integrate a grammer-based parser.

During tests I found that the new parser takes several minutes to parse an SQL file, whereas the old one processed the same file within seconds. Where is this time spent?

Time to profile the code. There are a couple of .Net profilers around, also Microsoft also provides a product called CLR Profiler which can be downloaded here.

When you start the CLRProfiler.exe application, both a console window and a WinForm window open. (This may look unusual, but hey, it’s a developer tool, and it is the functionality that counts)

Click on Start Application to select the executable you want to profile. The application starts up and you perform the tasks to profile just as if you started the application directly. (It just takes a little longer than usual, because the profiler does its work in the background).

When you’re done, close the application. (You may also choose the profiler’s Kill Application button). The profiler will now compute its magic numbers while message “Progress loading (filename).log” is displayed. After computation has finished, a Summary window displays memory and garbage collection statistics.

Click on the Allocated bytes Histogram button, and find the classes with most objects instantiated during the run.

For my purposes, the View Function Graph command was the most useful. It shows the percentage of time spent by each called function relative to the calling function.

It is not obvious from the UI, but each box representing a function can be drilled down by double-clicking it, showing again the percentages of the sub functions. (My guess is that only functions of the same assembly are shown in one graph, and the drill-down is necessary to switch between assemblies).

Anyway, I like that tool, as it showed me in just two (!!) graphs where the code spent its time, and as it turned out, these were just debugging routines (create XML document, retrieve its OuterText property, write to Console).


Get Absolute URL of ASP.Net Application

October 19, 2009

In one of my web projects, I needed to pass the name of a specific URL inside the application to another application. So the first task was: how do I find the absolute URL of an ASP.Net application’s root directory.

I tried ResolveClientUrl, which, according to MSDN, returns

A fully qualified URL to the specified resource suitable for use on the browser

As I was expecting an absolute URL from this description, it turned out, it doesn’t. (There also seems to be a terminological confusion between a Fully Qualified Domain Name, and Absolute and Relative URLs)

Both ResolveUrl and ResolveClientUrl create URLs relative to the page’s URL or the application root, but no absolute URL.

Fortunately, I found this entry on geekpedia, which provided a solution (that I was very close to develop on my own ;) )

public string FullyQualifiedApplicationPath
{
  get
  {
    //Return variable declaration
    string appPath = null;

    //Getting the current context of HTTP request
    HttpContext context = HttpContext.Current;

    //Checking the current context content
    if (context != null)
    {
      //Formatting the fully qualified website url/name
      appPath = string.Format("{0}://{1}{2}{3}",
        context.Request.Url.Scheme,
        context.Request.Url.Host,
        context.Request.Url.Port == 80
          ? string.Empty : ":" + context.Request.Url.Port,
        context.Request.ApplicationPath);
    }
    if (!appPath.EndsWith("/"))
      appPath += "/";
    return appPath;
  }
}

Of course, if you are inside a Page’s context, this reduced version is sufficient

string appPath = string.Format("{0}://{1}{2}{3}",
  context.Request.Url.Scheme,
  context.Request.Url.Host,
  context.Request.Url.Port == 80
    ? string.Empty : ":" + context.Request.Url.Port,
  context.Request.ApplicationPath);
if (!appPath.EndsWith("/"))
  appPath += "/";

Parsing Culture-Invariant Floating-Point Numbers

October 15, 2009

I just came across this non-obvious behavior of double.TryParse:

We develop on German Windows, but the input field for float values should be intelligent to recognize both German (“,”) and international (“.”) notations for the decimal point.

Of course the functions to call to parse a string s into a double value are:

double f; string s = "double value as string";

double.TryParse(s, System.Globalization.NumberStyles.Float,
    System.Globalization.CultureInfo.InvariantCulture, out f)

double.TryParse(s, out f)

But the results were surprising

InvariantCulture default culture
string value return (bool) output (double) return (bool) output (double)
“100,03″ false true 100.03
“100.03″ true 100.03 true 10003.0

My reasoning was, that if current (UI) culture could not parse the string, I let Invariant Culture try to parse it.

It turns out, however, that if the current culture simply removes all characters it does not recognize as valid input string for float numbers, and then parses the string, with the undesired result as seen above.

So the right way seems to try with invariant culture first, then use current culture:

double f;
double? Result;
if (double.TryParse(s, System.Globalization.NumberStyles.Float,
        System.Globalization.CultureInfo.InvariantCulture, out f))
    Result = f;
else if (double.TryParse(s, out f))
    Result = f;
else
    Result = null;

Handling Grants in a Function Hierarchy

September 29, 2009

Applications usually implement some kind of security or grant management which allows a user to perform certain operations or restricts them from invoking them. This post presents a generic way of handling grants in a function hierarchy.

First we need a list of all functions in the application. For illustration, I use a simple shop application with a minimal set of functions. A function can be a general operation, or an operation on a certain database record:

CREATE TABLE #Functions (
	ID	INT NOT NULL PRIMARY KEY,
	Name	NVARCHAR(50),
	ParentFunctionID INT NULL, -- REFERENCES #Functions (ID),
	Parameter1 	NVARCHAR(50),
	-- optionally more parameters
	CargoParameters	NVARCHAR(50),
	ResultType	NVARCHAR(50)
)

SET NOCOUNT ON 

INSERT INTO #Functions (ID, Name, CargoParameters, ResultType)
  VALUES (1, 'CreateProduct', 'ProductDescription pd', 'int')
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1)
  VALUES (10, 'ManageProduct', NULL, 'ProductID')
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1)
  VALUES (11, 'EditProduct', 10, 'ProductID')
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1)
  VALUES (12, 'RemoveProduct', 10, 'ProductID')
INSERT INTO #Functions (ID, Name) VALUES (20, 'GoShopping')
INSERT INTO #Functions (ID, Name, ParentFunctionID)
  VALUES (21, 'ListProducts', 20)
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1, CargoParameters, ResultType)
  VALUES (22, 'AddToCart', 20, 'ProductID', 'int Quantity', 'ShoppingResult')

SELECT * FROM #Functions

This results in the following table:

ID Name Parent FunctionID Parameter1 CargoParameters ResultType
1 CreateProduct NULL NULL ProductDescription pd int
10 ManageProduct NULL ProductID NULL NULL
11 EditProduct 10 ProductID NULL NULL
12 RemoveProduct 10 ProductID NULL NULL
20 GoShopping NULL NULL NULL NULL
21 ListProducts 20 NULL NULL NULL
22 AddToCart 20 ProductID int Quantity ShoppingResult

Once we have the complete list of all functions, we can generate code to implement a number of methods for each function:

IsPossibleToX Is it possible to execute function X now (depending on the information / status of the record
IsUserAllowedToX Does the user have sufficient rights to execute X
ExecuteX Invoke execution of function with parameters and result value

Read the rest of this entry »


Creating a light-weight Data Access Layer in C# (6)

September 9, 2009

Part 1: Retrieve information on tables and columns

Part 2: SELECT record, access record fields

Part 3: SELECT records, INSERT

Part 4: Generating C# classes

Part 5: Generated C# classes and polymorphism

Part 6: UPDATE record

Our DAL object has two collections, with dictNew holding new values, and dictRecord for previously inserted or retrieved values. To implement an Update() method, the Insert() method has to be fixed to save the inserted values:

   dictRecord = dictNew;
   dictNew = null;

Now we are ready to implement UPDATE. The parameters are collected similarly as described in part 3, which describes the INSERT operation:

private void Update(SqlConnection conn)
{
    if (IdentityColumn != null)
    {
        if (!dictRecord.ContainsKey(IdentityColumn))
            throw new Exception("UPDATE " + this.TableName + " without IDENTITY column set");

        StringBuilder sb = new StringBuilder();
        sb.Append("UPDATE " + TableName + " SET ");

        bool bFirst = true;
        foreach (string s in dictNew.Keys)
        {
            if (!bFirst)
                sb.Append(", ");
            sb.Append(s + " = @" + s);
            bFirst = false;
        }

        sb.Append(" WHERE " + IdentityColumn + " = @" + IdentityColumn);

        SqlCommand cmd = new SqlCommand();
        cmd.Connection = conn;

        foreach (string s in dictNew.Keys)
        {
            if (dictNew[s] == null)
                cmd.Parameters.AddWithValue("@" + s, DBNull.Value);
            else
                cmd.Parameters.AddWithValue("@" + s, dictNew[s]);
        }

        cmd.Parameters.AddWithValue("@" + IdentityColumn, dictRecord[IdentityColumn]);
        cmd.CommandText = sb.ToString();

        foreach (string s in dictNew.Keys)
            if (dictRecord.ContainsKey(s))
                dictRecord[s] = dictNew[s];
            else
                dictRecord.Add(s, dictNew[s]);

        dictNew = null;
    }
    else
        throw new Exception("UPDATE " + this.TableName + " without IDENTITY column is not implemented.");
}

If the table does not have an IDENTITY column, the WHERE condition for the UPDATE is unknown and an exception is raised. If the identity column of the existing record is not set, an exception is raised, too.


Reading LONG Oracle columns in .Net

August 24, 2009

If you use ODP.Net to access an Oracle database from .Net, and select LONG text columns, the DataReader’s GetString() method will return an empty string.

To work around this behavior, you need to set the InitialLONGFetchSize to a non-zero value, and use the OracleDataReader’s GetOracleString() method, and use ToString() to convert the result into a .Net string.

Another possibility is to set InitialLONGFetchSize to -1, so that GetString() works as expected.

The linked documentation also explains how to access LONG RAW data.


Creating a light-weight Data Access Layer in C# (5)

August 4, 2009

Part 1: Retrieve information on tables and columns

Part 2: SELECT record, access record fields

Part 3: SELECT records, INSERT

Part 4: Generating C# classes

Part 5: Generated C# classes and polymorphism

The code presented in the previous articles generates a C# class definition (based on dbscript’s Object table) like this:

public partial class Object : DAL.BaseTable
{
  public Object() { }
  public Object(IDataReader dr) : base(dr) { }
  protected override string TableName { get { return "Object"; } }
  public int OID
  { get { return GetColumnValue<int>(Columns.OID); } }
  public int PV_OID
  {
    get { return GetColumnValue<int>(Columns.PV_OID); }
    set { SetColumnValue(Columns.PV_OID, value); }
  }
  ....
  protected override string IdentityColumn { get { return "OID"; } }
  public static class Columns
  {
    public static string OID = "OID";
    public static string PV_OID = "PV_OID";
    ....
  }
  public static List<Object> GetRecords(IDataReader dr)
  { return GetRecords<Object>(dr); }
  public static List<Object> GetRecords(SqlConnection conn, string sSelect)
  { return GetRecords<Object>(conn, sSelect); }
}

Note that the class is generated as a partial class, so you are free to extend it in your own code.

In the dbscript database, all information about schema objects is stored in the Object table, and of course, not all columns of this table are used by the different object types, such as tables, views, etc. If we take into account different database engines (a concept introduced in dbscript 0.96), then tables, views, etc in MSSQL have different properties than their counterparts in Oracle or PostgreSQL.

Time to create some class which are derived from DAL.BaseTable to inherit the DAL mechanisms, but reference the generated DAL.Object columns:

public class SchemaObject : dbscriptlib.DAL.BaseTable
{
  protected override string TableName { get { return "Object"; } }
  protected override string IdentityColumn { get { return "OID"; } }
  public int OID
  { get { return GetColumnValue<int>(DAL.Object.Columns.OID); } }
  public string OwnerName
  {
    get { return GetColumnValue<string>(DAL.Object.Columns.OwnerName); }
    set { SetColumnValue(DAL.Object.Columns.OwnerName, value); }
  }
  public string ID
  {
    get { return GetColumnValue<string>(DAL.Object.Columns.ID); }
    set { SetColumnValue(DAL.Object.Columns.ID, value); }
  }
  ....
}
public class Table : SchemaObject
{
  public string FileGroupName
  {
    get { return GetColumnValue<string>(DAL.Object.Columns.FileGroupName); }
    set { SetColumnValue(DAL.Object.Columns.FileGroupName, value); }
  }
  ....
}

Note that the FileGroupName is defined only for tables, rather than for all DAL classes. In Oracle, there is no FileGroupName, but a TablespaceName, which we map to the original column using this definition:

public class OracleTable : SchemaObject
{
  public string TablespaceName
  {
    get { return GetColumnValue<string>(DAL.Object.Columns.FileGroupName); }
    set { SetColumnValue(DAL.Object.Columns.FileGroupName, value); }
  }
  ....
}

With this simple mechanism, a sort of polymorphic table can easily be implemented, and only the accessing classes “know” how about the mapping of the table columns.


Creating a light-weight Data Access Layer in C# (4)

July 30, 2009

Part 1: Retrieve information on tables and columns

Part 2: SELECT record, access record fields

Part 3: SELECT records, INSERT

As we have the base class of our DAL almost complete, it’s time to generate the actual table access classes.

CREATE PROCEDURE Generate_DAL AS
PRINT
'using System;
using System.Data;
using System.Data.SqlClient;
using System.Collections.Generic;
/*
 Class Definition for Data Access Layer
 automatically generated by Generate_DAL
 ' + CONVERT(VARCHAR, GETDATE(), 120) + '
*/
namespace DAL
{'
DECLARE @OID INT, @ID VARCHAR(50)
DECLARE cTable CURSOR LOCAL READ_ONLY FORWARD_ONLY FOR

(insert the table SELECT statement here). First we generate the access class constructors:

OPEN cTable
FETCH cTable INTO @OID, @ID
WHILE @@FETCH_STATUS=0 BEGIN
  IF UPPER(@ID) = 'COLUMNS' SET @ID = SUBSTRING(@ID, 1, LEN(@ID)-1)
  PRINT
'    public partial class ' + @ID + ' : DAL.BaseTable
 {
 public ' + @ID + '() { }
 public ' + @ID + '(IDataReader dr)  : base(dr)  { }
 protected override string TableName { get { return "' + @ID + '"; } }'

DECLARE @colID VARCHAR(50), @colDataType VARCHAR(50), @colLength INT
DECLARE @colNullable BIT, @colIdentity BIT, @colComputed BIT
DECLARE @colNetType VARCHAR(50), @colIdentityName VARCHAR(50)
SET @colIdentityName = NULL
DECLARE cCol CURSOR LOCAL READ_ONLY FORWARD_ONLY FOR

(insert the column SELECT statement here). We need to convert SQL data types to C# data types, and translate NULLable to Nullable<>:

OPEN cCol
FETCH cCol INTO @colID, @colDataType, @colLength, @colNullable, @colIdentity, @colComputed
WHILE @@FETCH_STATUS=0 BEGIN
  IF @colNullable = 1
    SELECT @colNetType = CASE @colDataType
      WHEN 'int' THEN 'int?'
      WHEN 'bit' THEN 'bool?'
      WHEN 'nvarchar' THEN 'string'
      WHEN 'varchar' THEN 'string'
      WHEN 'ntext' THEN 'string'
      WHEN 'text' THEN 'string'
      WHEN 'datetime' THEN 'DateTime?'
      ELSE @colDataType
    END
  ELSE
    SELECT @colNetType = CASE @colDataType
      WHEN 'int' THEN 'int'
      WHEN 'bit' THEN 'bool'
      WHEN 'nvarchar' THEN 'string'
      WHEN 'varchar' THEN 'string'
      WHEN 'ntext' THEN 'string'
      WHEN 'text' THEN 'string'
      WHEN 'datetime' THEN 'DateTime'
      ELSE @colDataType
    END

  IF @colIdentity = 1 SET @colIdentityName = @colID
  PRINT ' public ' + @colNetType + ' ' + @colID + '{
get { return GetColumnValue<' + @colNetType + '>(Columns.' + @colID + '); }'
  IF @colIdentity = 0 AND @colComputed = 0
    PRINT ' set { SetColumnValue(Columns.' + @colID + ', value); }'
  PRINT ' }'
  FETCH cCol INTO @colID, @colDataType, @colLength, @colNullable, @colIdentity, @colComputed
END
CLOSE cCol

IF @colIdentityName IS NOT NULL
  PRINT '        protected override string IdentityColumn { get { return "' + @colIdentityName + '"; } }'
PRINT '        public static class Columns {'
OPEN cCol
FETCH cCol INTO @colID, @colDataType, @colLength, @colNullable, @colIdentity, @colComputed
WHILE @@FETCH_STATUS=0 BEGIN
  PRINT    '            public static string ' + @colID + ' = "' + @colID + '";';
  FETCH cCol INTO @colID, @colDataType, @colLength, @colNullable, @colIdentity, @colComputed
END
CLOSE cCol
DEALLOCATE cCol
PRINT    '        }'
PRINT    '        public List<' + @ID + '> GetRecords(IDataReader dr) {
    return GetRecords<' + @ID + '>(dr);
  }
public List<' + @ID + '> GetRecords(SqlConnection conn, string sSelect) {
    return GetRecords<' + @ID + '>(conn, sSelect);
  }
}'

FETCH cTable INTO @OID, @ID
END
PRINT    '}'

The resulting class definition has typed getter/setter properties for each column of the base table, and a static string array containing the names of the table columns.

Thus changes the data model (column type, column name, dropped table or column) are immediately reflected in the generated code, and automatically raise to compilation warnings or errors in your code which depends on the model.



Creating a light-weight Data Access Layer in C# (3)

July 29, 2009

Part 1: Retrieve information on tables and columns

Part 2: SELECT record, access record fields

To select multiple records and create their DAL objects, we pass an IDataReader object or a combination of connection and SELECT statement to a static method GetRecords() and receive a typed List<> of records:

protected static List<T> GetRecords<T>(IDataReader dr)
    where T : BaseTable, new()
{
  List<T> li = new List<T>();
  while (dr.Read())
  {
    T record = new T();
    record.GetRecord(dr);
    li.Add(record);
  }
  dr.Close();
  return li;
}

protected static List<T> GetRecords<T>(SqlConnection conn, string sSelect)
    where T : BaseTable, new()
{
  SqlCommand cmd = new SqlCommand(sSelect, conn);
  return GetRecords<T>(cmd.ExecuteReader());
}

The Save() method writes all values of dictNew as INSERT or UPDATE statement:

public void Save(SqlConnection conn)
{
  if (dictNew == null)
    return;

  if (dictRecord == null)
    Insert(conn);
  else
    Update(conn);
}

This is a sample implementation of the Insert() command. Note that in the simplest version, the values of dictNew are INSERTed, and only the IDENTITY column (if defined) is retrieved after the insert.

In more complex scenarios, you would want to retrieve all the values of the newly generated record to handle defaults and computed columns. If the table is lacking an IDENTITY column, the record needs to be selected using the columns of a UNIQUE constraint. These scenarios are not covered right now.

private void Insert(SqlConnection conn)
{
  StringBuilder sb = new StringBuilder();
  sb.Append("INSERT INTO " + TableName + " (");

  bool bFirst = true;
  foreach (string s in dictNew.Keys)
  {
    if (!bFirst)
      sb.Append(", ");
    sb.Append(s);
    bFirst = false;
  }

  sb.Append(") VALUES (");
  bFirst = true;
  foreach (string s in dictNew.Keys)
  {
    if (!bFirst)
      sb.Append(", ");
    sb.Append("@" + s);
    bFirst = false;
  }
  sb.Append(")");

  SqlCommand cmd = new SqlCommand();
  cmd.Connection = conn;

  foreach (string s in dictNew.Keys)
  {
    if (dictNew[s] == null)
      cmd.Parameters.AddWithValue("@" + s, DBNull.Value);
    else
      cmd.Parameters.AddWithValue("@" + s, dictNew[s]);
  }

  if (IdentityColumn != null)
  {
    sb.Append("; SELECT " + IdentityColumn + " FROM " + TableName +
      " WHERE " + IdentityColumn + " = SCOPE_IDENTITY()");
    cmd.CommandText = sb.ToString();
    object oIdentity = cmd.ExecuteScalar();
    dictNew.Add(IdentityColumn, oIdentity);
  }
  else
  {
    cmd.CommandText = sb.ToString();
    cmd.ExecuteNonQuery();
  }
}