Handling Grants in a Function Hierarchy

September 29, 2009

Applications usually implement some kind of security or grant management which allows a user to perform certain operations or restricts them from invoking them. This post presents a generic way of handling grants in a function hierarchy.

First we need a list of all functions in the application. For illustration, I use a simple shop application with a minimal set of functions. A function can be a general operation, or an operation on a certain database record:

CREATE TABLE #Functions (
	ID	INT NOT NULL PRIMARY KEY,
	Name	NVARCHAR(50),
	ParentFunctionID INT NULL, -- REFERENCES #Functions (ID),
	Parameter1 	NVARCHAR(50),
	-- optionally more parameters
	CargoParameters	NVARCHAR(50),
	ResultType	NVARCHAR(50)
)

SET NOCOUNT ON 

INSERT INTO #Functions (ID, Name, CargoParameters, ResultType)
  VALUES (1, 'CreateProduct', 'ProductDescription pd', 'int')
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1)
  VALUES (10, 'ManageProduct', NULL, 'ProductID')
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1)
  VALUES (11, 'EditProduct', 10, 'ProductID')
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1)
  VALUES (12, 'RemoveProduct', 10, 'ProductID')
INSERT INTO #Functions (ID, Name) VALUES (20, 'GoShopping')
INSERT INTO #Functions (ID, Name, ParentFunctionID)
  VALUES (21, 'ListProducts', 20)
INSERT INTO #Functions (ID, Name, ParentFunctionID, Parameter1, CargoParameters, ResultType)
  VALUES (22, 'AddToCart', 20, 'ProductID', 'int Quantity', 'ShoppingResult')

SELECT * FROM #Functions

This results in the following table:

ID Name Parent FunctionID Parameter1 CargoParameters ResultType
1 CreateProduct NULL NULL ProductDescription pd int
10 ManageProduct NULL ProductID NULL NULL
11 EditProduct 10 ProductID NULL NULL
12 RemoveProduct 10 ProductID NULL NULL
20 GoShopping NULL NULL NULL NULL
21 ListProducts 20 NULL NULL NULL
22 AddToCart 20 ProductID int Quantity ShoppingResult

Once we have the complete list of all functions, we can generate code to implement a number of methods for each function:

IsPossibleToX Is it possible to execute function X now (depending on the information / status of the record
IsUserAllowedToX Does the user have sufficient rights to execute X
ExecuteX Invoke execution of function with parameters and result value

Read the rest of this entry »


Creating Dia Data Diagrams from Database Schema

September 27, 2009

As I stated in earlier posts, you can use dbscript to generate PNG Data Diagrams (MS SQL Server, Oracle, PostgreSQL).

The generated PNG files are intended to give you an overview of the data model. The great drawback is that the information is graphical only, as it contains no information on the original data.

Upcoming version 0.99 will also include the capability to generate Dia files representing a database schema. Dia is an open-source diagram creation program which runs on Windows and Linux.

The advantages of generating in Dia format are that it allows users / developers to modify the generated diagrams, and that general Dia functionality can be used as stated on their website:

[It] can export diagrams to a number of formats, including EPS, SVG, XFIG, WMF and PNG, and can print diagrams (including ones that span multiple pages).

There is one minor issue in generating and opening Dia files: since Dia automatically sizes Table shapes according to their contents, it is not possible to predict what the table shape size will turn out to be (see FAQ). Thus when you open a generated Dia file for the first time, the foreign key connectors will look disconnected from the tables, even though they are not (in the data).

You need to manually enforce connector layout following these steps:

  • Select All (Ctrl-A)
  • Move selected objects (Cursor-Left, then Cursor-Right)

which will layout the foreign key connectors as expected.

Visit the gallery to view PNG images generated by Dia from diagrams created by dbscript.

Download Dia files with database models of

You need to install Dia to view the contents of these files.


Checking Table Relations in MS SQL Server

September 25, 2009

To find isolated tables (that is, tables without foreign key constraints referencing to or from another table) in your data model, use this query:

SELECT  name
FROM    sys.objects t
LEFT OUTER JOIN sys.foreign_key_columns AS FKC1
    ON t.object_id = FKC1.parent_object_id
LEFT OUTER JOIN sys.foreign_key_columns AS FKC2
    ON t.object_id = FKC2.referenced_object_id
WHERE (t.type = 'U')
GROUP BY t.name
HAVING (COUNT(FKC1.constraint_object_id) = 0)
AND    (COUNT(FKC2.constraint_object_id) = 0)

If you have a naming convention for foreign key columns, such as FooID or Bar_ID, you can check this requirement:

SELECT Detail.name, DCol1.name AS Column1, Master.name
FROM sys.objects AS Detail
INNER JOIN sys.columns AS DCol1 ON Detail.object_id = DCol1.object_id
INNER JOIN sys.foreign_key_columns AS FKC1
    ON DCol1.column_id = FKC1.parent_column_id
    AND DCol1.object_id = FKC1.parent_object_id
INNER JOIN sys.objects AS Master ON FKC1.referenced_object_id = Master.object_id
WHERE DCol1.name NOT LIKE N'%[_]ID'
AND   DCol1.name NOT LIKE N'%[_]ID[_]%'

To check the opposite, i.e. a column with final “ID” is not part of a foreign key:

SELECT Detail.name, DCol1.name
FROM sys.objects AS Detail
INNER JOIN sys.columns AS DCol1 ON Detail.object_id = DCol1.object_id
LEFT OUTER JOIN sys.foreign_key_columns AS FKC1
    ON  DCol1.column_id = FKC1.parent_column_id
    AND DCol1.object_id = FKC1.parent_object_id
WHERE (DCol1.name LIKE N'%[_]ID' or DCol1.name LIKE N'%[_]ID[_]%')
AND   (FKC1.constraint_object_id IS NULL)
AND   (Detail.type = 'U')

To check the requirement, that the foreign key of a column with a certain name must always reference a certain table:

SELECT Detail.name, Master.name
FROM sys.objects AS Detail
INNER JOIN sys.columns AS DCol1 ON Detail.object_id = DCol1.object_id
INNER JOIN sys.types on DCol1.system_type_id = sys.types.system_type_id
LEFT OUTER JOIN	sys.foreign_key_columns AS FKC1
    INNER JOIN	sys.objects AS Master
        ON FKC1.referenced_object_id = Master.object_id
    ON DCol1.column_id = FKC1.parent_column_id
    AND DCol1.object_id = FKC1.parent_object_id
WHERE Detail.schema_id =1
AND   DCol1.name = 'STATUS'
AND   ISNULL(Master.Name, '') <> 'StatusValues'

And finally, to check whether all foreign key relations can make use of an index, use:

SELECT t.name, c.name, r.name, ic.index_column_id
FROM sys.objects t
INNER JOIN sys.columns c ON t.object_id = c.object_id
INNER JOIN sys.foreign_key_columns fkc
    ON fkc.parent_object_id = c.object_id
    AND fkc.parent_column_id = c.column_id
INNER JOIN sys.objects fk ON fkc.constraint_object_id = fk.object_id
INNER JOIN sys.objects r ON fkc.referenced_object_id = r.object_id
LEFT OUTER JOIN sys.index_columns ic
        INNER JOIN sys.indexes i
            ON  ic.object_id = i.object_id AND ic.index_id = i.index_id
    ON c.object_id = ic.object_id AND c.column_id = ic.column_id
LEFT OUTER JOIN sys.index_columns ic1
    ON  c.object_id = ic1.object_id
    AND c.column_id = ic1.column_id
    AND ic1.index_column_id = 1
WHERE t.schema_id = 1
AND   ISNULL(ic.index_column_id, -1) != 1
AND   ic1.index_column_id IS NULL

Stack Overflow Yearling

September 24, 2009

You’ve earned the “Yearling” badge. See your profile.


Finding Table Names not referenced by TSQL Code

September 19, 2009

A question on Stack Overflow titled TSQL script to find tables not being used by stored procedures, views, functions, etc? reminded me of a recent post on searching the TSQL source code catalog view of MS SQL Server.

This query would list all occurrences of all table names in T-SQL code, regardless whether the table name really refers to that table, or is just a substring of whatever identifier:

select t.name, sys.objects.name foundin, sys.objects.type_desc
from sys.objects t
left outer join
	sys.sql_modules
	inner join sys.objects on sys.objects.object_id = sys.sql_modules.object_id
on sys.sql_modules.definition like '%' + t.name + '%'
where t.type = 'U'
order by t.name, type_desc, foundin

To list unreferenced table names, we add an IS NULL condition on the modules/objects:

select t.name, sys.objects.name foundin, sys.objects.type_desc
from sys.objects t
left outer join
	sys.sql_modules
	inner join sys.objects on sys.objects.object_id = sys.sql_modules.object_id
on sys.sql_modules.definition like '%' + t.name + '%'
where t.type = 'U'
and sys.objects.name is null
order by t.name, type_desc, foundin

Doing the same check on views is a bit trickier, since the view definition contains the view name itself in the code. Therefore the OUTER JOIN condition needs to be adjusted to exclude the view’s own source code from the match:

select t.name, sys.objects.name foundin, sys.objects.type_desc
from sys.objects t
left outer join
	sys.sql_modules
	inner join sys.objects on sys.objects.object_id = sys.sql_modules.object_id
on sys.sql_modules.definition like '%' + t.name + '%'
	and sys.objects.name != t.name
where t.type = 'V'
order by t.name, type_desc, foundin

Command-line XSLT processor with PowerShell

September 15, 2009

There are already a lot of XSLT processors out there, such as MSXSL, but without downloading and installing an application you can create your own processor using a couple of PowerShell lines and the System.Xml.Xsl namespace of .Net:

param ($xml, $xsl, $output)

if (-not $xml -or -not $xsl -or -not $output)
{
	Write-Host "& .\xslt.ps1 [-xml] xml-input [-xsl] xsl-input [-output] transform-output"
	exit;
}

trap [Exception]
{
	Write-Host $_.Exception;
}

$xslt = New-Object System.Xml.Xsl.XslCompiledTransform;
$xslt.Load($xsl);
$xslt.Transform($xml, $output);

Write-Host "generated" $output;

What does this code do:

  • Declare command-line parameters $xml, $xsl, $output
  • Check parameters are passed to the script
  • Set a trap to display detailed error message in case an exception is raised
  • Load the XSLT file
  • Transform XML and write result to output file

Finding cycles in directed graphs using TSQL

September 13, 2009

I recently noticed that the Create Data Diagram function of dbscript creates rather distorted diagrams if there are cycles of more than 2 tables in table relations. However, the algorithm handles pig’s ears and reciprocal relations correctly.

So the question was, how do I detect cycles in table relations?

Regarding a directed graph, tables are equivalent to nodes, foreign key relations are equivalent to directed edges, thus the question can be rephrased as:

How do I detect cycles in a directed graph, given the graph is stored in 2 tables with this structure:

CREATE TABLE #Node (
	ID INT PRIMARY KEY,
	Name NVARCHAR(128)
)

CREATE TABLE #Edge (
	ID INT IDENTITY(1,1) PRIMARY KEY,
	NodeIDFrom INT,
	NodeIDTo INT
)

Let’s populate the tables with the following values:

INSERT INTO #Node (ID, Name) VALUES (1, 'A')
INSERT INTO #Node (ID, Name) VALUES (2, 'B1')
INSERT INTO #Node (ID, Name) VALUES (3, 'B2')
INSERT INTO #Node (ID, Name) VALUES (4, 'C1')
INSERT INTO #Node (ID, Name) VALUES (5, 'C2')
INSERT INTO #Node (ID, Name) VALUES (6, 'C3')
INSERT INTO #Node (ID, Name) VALUES (7, 'D')
INSERT INTO #Node (ID, Name) VALUES (8, 'E')

-- Bx loop
INSERT INTO #Edge(NodeIDFrom, NodeIDTo) VALUES (2, 3)
INSERT INTO #Edge(NodeIDFrom, NodeIDTo) VALUES (3, 2)
-- Cx loop
INSERT INTO #Edge(NodeIDFrom, NodeIDTo) VALUES (4, 5)
INSERT INTO #Edge(NodeIDFrom, NodeIDTo) VALUES (5, 6)
INSERT INTO #Edge(NodeIDFrom, NodeIDTo) VALUES (6, 4)
-- D, E, C1, B1
INSERT INTO #Edge(NodeIDFrom, NodeIDTo) VALUES (7, 8 )
INSERT INTO #Edge(NodeIDFrom, NodeIDTo) VALUES (8, 4)
INSERT INTO #Edge(NodeIDFrom, NodeIDTo) VALUES (4, 2)

i.e. node A is not connected to any other, the Bx nodes form a cycle (length 2), the Cx nodes form a cycle (length 3), and there is a path D-E-C1-B1.

This algorithm performs the following operations:

In table #Graph, create a list of all paths starting from all nodes, until the most recently reached nodes match the starting nodes. List creation ends when no new edges can be added to the list of paths (@@ROWCOUNT = 0):

CREATE TABLE #Graph(
	ID INT IDENTITY(1,1) NOT NULL PRIMARY KEY,
	Seq INT,
	NodeID INT,
	EdgeID INT,
	NodeIDNext INT,
	NodeIDRoot int,
	IsCycle BIT DEFAULT (0)
)

INSERT INTO #Graph (Seq, NodeID, NodeIDRoot, EdgeID, NodeIDNext)
SELECT DISTINCT 1, #Node.ID, #Node.ID, #Edge.ID, #Edge.NodeIDTo
FROM #Node
INNER JOIN #Edge ON #Node.ID = #Edge.NodeIDFrom

PRINT @@ROWCOUNT

DECLARE @Seq INT, @RC INT
SET @Seq = 1
SET @RC = 1

WHILE @RC > 0 BEGIN

	UPDATE 	#Graph
	SET	IsCycle = 1
	WHERE	NodeIDRoot = NodeIDNext
	AND	Seq = @Seq

	INSERT INTO #Graph (Seq, NodeID, NodeIDRoot, EdgeID, NodeIDNext)
	SELECT DISTINCT @Seq + 1, #Edge.NodeIDFrom, #Graph.NodeIDRoot,
		#Edge.ID, #Edge.NodeIDTo
	FROM 	#Graph
	INNER JOIN #Edge ON #Graph.NodeIDNext = #Edge.NodeIDFrom
	LEFT OUTER JOIN #Graph ex ON ex.NodeIDRoot = #Graph.NodeIDRoot
		AND ex.EdgeID = #Edge.ID
	WHERE 	#Graph.IsCycle = 0
	AND 	#Graph.Seq = @Seq
	AND 	ex.ID IS NULL

	SET @RC = @@ROWCOUNT

	PRINT CONVERT(VARCHAR, @Seq) + ': ' + CONVERT(VARCHAR, @RC)
	SET @Seq = @Seq + 1
END

SELECT 	#Graph.NodeIDRoot, #Graph.Seq, #Graph.NodeID,
		#Graph.NodeIDNext, #Graph.EdgeID, #Node.Name, Node2.Name
FROM 	#Graph
INNER JOIN #Node ON #Graph.NodeID = #Node.ID
INNER JOIN #Node Node2 ON #Graph.NodeIDNext = Node2.ID
WHERE 	IsCycle = 1
ORDER BY #Graph.NodeIDRoot, #Graph.Seq

The IsCycle attribute marks paths that reached its starting node (NodeIDRoot = NodeIDNext).

Next, we collect all final steps that led to cycle detection:

CREATE TABLE #Cycles(
	ID INT IDENTITY(1,1) PRIMARY KEY,
	CycleID INT,
	NodeIDRoot INT,
	Seq INT,
	NodeID INT,
	NodeIDNext INT,
	NodeName NVARCHAR(128),
	EdgeID INT
)

INSERT INTO #Cycles (NodeIDRoot, Seq, NodeID, NodeIDNext, NodeName, EdgeID)
SELECT NodeIDRoot, Seq, NodeID, NodeIDNext, #Node.Name, EdgeID
FROM 	#Graph
INNER JOIN #Node ON #Graph.NodeIDNext = #Node.ID
WHERE 	IsCycle = 1
ORDER BY #Graph.NodeIDRoot, #Graph.Seq

UPDATE 	#Cycles SET CycleID = ID

SELECT * FROM #Cycles
ORDER BY CycleID, Seq

SET @RC = @@ROWCOUNT

From this information we reconstruct the path taken from the starting nodes to the end nodes:

WHILE @RC > 0 BEGIN
	INSERT INTO #Cycles (CycleID, NodeIDRoot, Seq, NodeID, NodeIDNext, NodeName, EdgeID)
	SELECT DISTINCT #Cycles.CycleID, #Cycles.NodeIDRoot, #Graph.Seq,
		#Graph.NodeID, #Graph.NodeIDNext, #Node.Name, #Graph.EdgeID
	FROM #Cycles
	INNER JOIN #Graph ON #Cycles.NodeIDRoot = #Graph.NodeIDRoot
		AND #Cycles.Seq -1 = #Graph.Seq
		AND #Cycles.NodeID = #Graph.NodeIDNext
	INNER JOIN #Node ON #Graph.NodeIDNext = #Node.ID
	LEFT OUTER JOIN #Cycles ex ON #Cycles.CycleID = ex.CycleID
		AND ex.NodeIDRoot = #Cycles.NodeIDRoot
		AND ex.Seq = #Graph.Seq
		AND ex.NodeID = #Graph.NodeID
		AND ex.NodeIDNext = #Graph.NodeIDNext
		AND ex.EdgeID = #Graph.EdgeID
	WHERE ex.ID IS NULL
	ORDER BY #Cycles.NodeIDRoot, #Graph.Seq

	SET @RC = @@ROWCOUNT
END

SELECT * FROM #Cycles
ORDER BY CycleID, Seq

The last query lists the cycles:
Cycle 1: B2 – B1
Cycle 2: B1 – B2
Cycle 3: C2 – C3 – C1
Cycle 4: C3 – C1 – C2
Cycle 5: C1 – C2 – C3

Finally, we clean up the temporary tables

DROP TABLE #Edge
DROP TABLE #Node
DROP TABLE #Graph
DROP TABLE #Cycles

Blogging for Two Years

September 13, 2009

This blog is now running for 2 years, and sometimes I feel that there is more to write about than I find time to actually type.

During the last 12 months, I released various versions of dbscript, my database scripting / versioning / comparison / documentation tool, and added support for Oracle and PostgreSQL databases beside the previous MSSQL-only functionality.

automssqlbackup is a PowerShell script which creates backups of MSSQL databases and rotates the generated and/or zipped backup files, and has also experienced a couple of updates since its first release.

Just the numbers, please: 78,000 views, 164 posts, 1,300 downloads in 2 years.

Interestingly, my older posts on building and publishing Visual Studio projects from the command line, and Selenium tests are still those generating most views.

But I need to note that the newer articles on code generation, documentation generation, and data access layers are increasingly attracting interest.


Version 0.98 of dbscript Released

September 9, 2009

The latest version 0.98 of dbscript supports PostgreSQL databases in its documentation generation capabilities.

After importing the database dictionary (via direct connection using ADO.Net and Npgsql) can document a PostgreSQL database in all currently supported documentation format:

MediaWiki

Data Diagram (PNG)

HTML

ScrewTurn wiki

Integration support for PostgreSQL had some consequences: More and more functionality is handled separately for each database engine.

Database import was obviously the first one, since the data access classes (SqlConnection, SqlCommand) in .Net are different for every database library. Same goes for the database dictionary, which is best retrieve from the native system catalogs.

For import and upload, data access classes have been introduced to distinguish the different object types and their properties of each database engine. I mention work on the data access classes in a series of articles already.

In version 0.98, XML generation and object script generation are implemented separately. This results in XSL style sheets being now related to certain a database engine.

For Oracle, XML and object script generation have been updated, and the XSL style sheets have been adjusted to Oracle-specific objects and properties. The results were documented earlier.

The latest version of dbscript is available for download here.

Please leave comments and feedback.


Creating a light-weight Data Access Layer in C# (6)

September 9, 2009

Part 1: Retrieve information on tables and columns

Part 2: SELECT record, access record fields

Part 3: SELECT records, INSERT

Part 4: Generating C# classes

Part 5: Generated C# classes and polymorphism

Part 6: UPDATE record

Our DAL object has two collections, with dictNew holding new values, and dictRecord for previously inserted or retrieved values. To implement an Update() method, the Insert() method has to be fixed to save the inserted values:

   dictRecord = dictNew;
   dictNew = null;

Now we are ready to implement UPDATE. The parameters are collected similarly as described in part 3, which describes the INSERT operation:

private void Update(SqlConnection conn)
{
    if (IdentityColumn != null)
    {
        if (!dictRecord.ContainsKey(IdentityColumn))
            throw new Exception("UPDATE " + this.TableName + " without IDENTITY column set");

        StringBuilder sb = new StringBuilder();
        sb.Append("UPDATE " + TableName + " SET ");

        bool bFirst = true;
        foreach (string s in dictNew.Keys)
        {
            if (!bFirst)
                sb.Append(", ");
            sb.Append(s + " = @" + s);
            bFirst = false;
        }

        sb.Append(" WHERE " + IdentityColumn + " = @" + IdentityColumn);

        SqlCommand cmd = new SqlCommand();
        cmd.Connection = conn;

        foreach (string s in dictNew.Keys)
        {
            if (dictNew[s] == null)
                cmd.Parameters.AddWithValue("@" + s, DBNull.Value);
            else
                cmd.Parameters.AddWithValue("@" + s, dictNew[s]);
        }

        cmd.Parameters.AddWithValue("@" + IdentityColumn, dictRecord[IdentityColumn]);
        cmd.CommandText = sb.ToString();

        foreach (string s in dictNew.Keys)
            if (dictRecord.ContainsKey(s))
                dictRecord[s] = dictNew[s];
            else
                dictRecord.Add(s, dictNew[s]);

        dictNew = null;
    }
    else
        throw new Exception("UPDATE " + this.TableName + " without IDENTITY column is not implemented.");
}

If the table does not have an IDENTITY column, the WHERE condition for the UPDATE is unknown and an exception is raised. If the identity column of the existing record is not set, an exception is raised, too.