Fatal Error 682 when accessing MS CRM 2011

Developing a CRM 2011 application, I unexpectedly received the SqlException

Warning: Fatal error 682 occurred at [date, time]. Note the error and time, and contact your system administrator.

The complete trace entry in the MS CRM log file looks like this:

[2012-06-12 11:40:43.387] Process: w3wp |Organization:00000000-0000-0000-0000-000000000000 |Thread: 15 |Category: Platform.Sql |User: 00000000-0000-0000-0000-000000000000 |Level: Error | DynamicMetadataCacheLoader.LoadMetadataContainerFromDatabase
>Exception while loading Metadata Cache from Database with LoadMasks = All, OData using new connection and new transaction. Exception: System.Data.SqlClient.SqlException (0x80131904): Warnung: Schwerwiegender Fehler 682 um Jun 12 2012 11:40AM. Notieren Sie den Fehler und den Zeitpunkt, und wenden Sie sich an den Systemadministrator.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.HasMoreRows()
at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
at Microsoft.Crm.CrmDataReader.Read()
at Microsoft.Crm.Metadata.MultiOrgSharableMetadataCacheLoader.LoadDescriptions(Boolean isSystemDataLoaded, CrmDbConnection connection, IDbTransaction transaction, MetadataContainer systemContainer, MetadataContainer aggregateContainer, TableFillPropertiesMultiOrg properties, CounterList counter)
at Microsoft.Crm.Metadata.MultiOrgSharableMetadataCacheLoader.LoadContainerData(Boolean isSystemDataLoaded, Boolean isSystemLanguageDataLoaded, CrmDbConnection connection, IDbTransaction transaction, MetadataContainer systemContainer, MetadataContainer aggregateContainer, List`1 fillProperties, CounterList counter)
at Microsoft.Crm.Metadata.MultiOrgSharableMetadataCacheLoader.BuildMetadataContainerFromDatabase(LoadMasks masks, CrmDbConnection connection, IDbTransaction transaction, Guid organizationId, CounterList counter)
at Microsoft.Crm.Metadata.DynamicMetadataCacheLoader.LoadMetadataContainerFromDatabase(LoadMasks masks, CrmDbConnection connection, CrmTransaction transaction, Guid organizationId, CounterList counter)
[2012-06-12 11:40:43.398] Process: w3wp |Organization:00000000-0000-0000-0000-000000000000 |Thread: 15 |Category: Platform.Sdk |User: 00000000-0000-0000-0000-000000000000 |Level: Error | ServiceModelTraceRedirector.TraceData
><TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error"><TraceIdentifier>http://msdn.microsoft.com/de-DE/library/System.ServiceModel.Diagnostics.TraceHandledException.aspx</TraceIdentifier><Description>Handling an exception.</Description><AppDomain>/LM/W3SVC/2/ROOT-1-129839675491242152</AppDomain><Exception><ExceptionType>System.Data.SqlClient.SqlException, System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>Warnung: Schwerwiegender Fehler 682 um Jun 12 2012 11:40AM. Notieren Sie den Fehler und den Zeitpunkt, und wenden Sie sich an den Systemadministrator.</Message><StackTrace> at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
> at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
> at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
> at System.Data.SqlClient.SqlDataReader.HasMoreRows()
> at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
> at Microsoft.Crm.CrmDataReader.Read()
> at Microsoft.Crm.Metadata.MultiOrgSharableMetadataCacheLoader.LoadDescriptions(Boolean isSystemDataLoaded, CrmDbConnection connection, IDbTransaction transaction, MetadataContainer systemContainer, MetadataContainer aggregateContainer, TableFillPropertiesMultiOrg properties, CounterList counter)
> at Microsoft.Crm.Metadata.MultiOrgSharableMetadataCacheLoader.LoadContainerData(Boolean isSystemDataLoaded, Boolean isSystemLanguageDataLoaded, CrmDbConnection connection, IDbTransaction transaction, MetadataContainer systemContainer, MetadataContainer aggregateContainer, List`1 fillProperties, CounterList counter)
> at Microsoft.Crm.Metadata.MultiOrgSharableMetadataCacheLoader.BuildMetadataContainerFromDatabase(LoadMasks masks, CrmDbConnection connection, IDbTransaction transaction, Guid organizationId, CounterList counter)
> at Microsoft.Crm.Metadata.DynamicMetadataCacheLoader.LoadMetadataContainerFromDatabase(LoadMasks masks, CrmDbConnection connection, CrmTransaction transaction, Guid organizationId, CounterList counter)
> at Microsoft.Crm.Metadata.DynamicMetadataCacheLoader.LoadCacheFromDatabaseInternal(LoadMasks masks, CrmDbConnection connection, CrmTransaction transaction, IOrganizationContext context, CounterList counter)
> at Microsoft.Crm.Metadata.DynamicMetadataCacheLoader.LoadCacheFromDatabase(LoadMasks masks, IOrganizationContext context, CounterList counter)
> at Microsoft.Crm.Metadata.DynamicMetadataCacheFactory.LoadMetadataCache(LoadMethod method, CacheType type, IOrganizationContext context)
> at Microsoft.Crm.Metadata.MetadataCache.LoadCache(IOrganizationContext context, Boolean fileOnlyIfExists)
> at Microsoft.Crm.Metadata.MetadataCache.GetInstance(IOrganizationContext context)
> at Microsoft.Crm.BusinessEntities.BusinessEntityMoniker..ctor(Guid id, String entityName, IOrganizationContext orgContext)
> at Microsoft.Crm.Caching.UserDataCacheLoader.LoadCacheData(Guid key, ExecutionContext context)
> at Microsoft.Crm.Caching.ObjectModelCacheLoader`2.LoadCacheData(TKey key, IOrganizationContext context)
> at Microsoft.Crm.Caching.CrmMultiOrgCache`2.CreateEntry(TKey key, IOrganizationContext context)
> at Microsoft.Crm.Caching.CrmMultiOrgCache`2.LookupEntry(TKey key, IOrganizationContext context)
> at Microsoft.Crm.Authentication.UserManagementFactory.UpdateAccessTime(Guid userId, UserAuthenticationAccessType accessType, DateTime accessTime, IOrganizationContext context)
> at Microsoft.Crm.Authentication.UserManagementFactory.UpdateAccessTime(Guid userId, Guid organizationId, UserAuthenticationAccessType accessType, DateTime accessTime)
> at Microsoft.Crm.Authentication.WindowsIdentityAuthorizationManager.Authenticate(OperationContext operationContext)
> at Microsoft.Crm.Authentication.WindowsIdentityAuthorizationManager.CheckAccessCore(OperationContext operationContext)
> at System.ServiceModel.Dispatcher.AuthorizationBehavior.Authorize(MessageRpc&amp;amp; rpc)
> at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc&amp;amp; rpc)
> at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)</StackTrace><ExceptionString>System.Data.SqlClient.SqlException (0x80131904): Warnung: Schwerwiegender Fehler 682 um Jun 12 2012 11:40AM. Notieren Sie den Fehler und den Zeitpunkt, und wenden Sie sich an den Systemadministrator.
> at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
> at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
> at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
> at System.Data.SqlClient.SqlDataReader.HasMoreRows()
> at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
> at Microsoft.Crm.CrmDataReader.Read()
> at Microsoft.Crm.Metadata.MultiOrgSharableMetadataCacheLoader.LoadDescriptions(Boolean isSystemDataLoaded, CrmDbConnection connection, IDbTransaction transaction, MetadataContainer systemContainer, MetadataContainer aggregateContainer, TableFillPropertiesMultiOrg properties, CounterList counter)
> at Microsoft.Crm.Metadata.MultiOrgSharableMetadataCacheLoader.LoadContainerData(Boolean isSystemDataLoaded, Boolean isSystemLanguageDataLoaded, CrmDbConnection connection, IDbTransaction transaction, MetadataContainer systemContainer, MetadataContainer aggregateContainer, List`1 fillProperties, CounterList counter)
> at Microsoft.Crm.Metadata.MultiOrgSharableMetadataCacheLoader.BuildMetadataContainerFromDatabase(LoadMasks masks, CrmDbConnection connection, IDbTransaction transaction, Guid organizationId, CounterList counter)
> at Microsoft.Crm.Metadata.DynamicMetadataCacheLoader.LoadMetadataContainerFromDatabase(LoadMasks masks, CrmDbConnection connection, CrmTransaction transaction, Guid organizationId, CounterList counter)
> at Microsoft.Crm.Metadata.DynamicMetadataCacheLoader.LoadCacheFromDatabaseInternal(LoadMasks masks, CrmDbConnection connection, CrmTransaction transaction, IOrganizationContext context, CounterList counter)
> at Microsoft.Crm.Metadata.DynamicMetadataCacheLoader.LoadCacheFromDatabase(LoadMasks masks, IOrganizationContext context, CounterList counter)
> at Microsoft.Crm.Metadata.DynamicMetadataCacheFactory.LoadMetadataCache(LoadMethod method, CacheType type, IOrganizationContext context)
> at Microsoft.Crm.Metadata.MetadataCache.LoadCache(IOrganizationContext context, Boolean fileOnlyIfExists)
> at Microsoft.Crm.Metadata.MetadataCache.GetInstance(IOrganizationContext context)
> at Microsoft.Crm.BusinessEntities.BusinessEntityMoniker..ctor(Guid id, String entityName, IOrganizationContext orgContext)
> at Microsoft.Crm.Caching.UserDataCacheLoader.LoadCacheData(Guid key, ExecutionContext context)
> at Microsoft.Crm.Caching.ObjectModelCacheLoader`2.LoadCacheData(TKey key, IOrganizationContext context)
> at Microsoft.Crm.Caching.CrmMultiOrgCache`2.CreateEntry(TKey key, IOrganizationContext context)
> at Microsoft.Crm.Caching.CrmMultiOrgCache`2.LookupEntry(TKey key, IOrganizationContext context)
> at Microsoft.Crm.Authentication.UserManagementFactory.UpdateAccessTime(Guid userId, UserAuthenticationAccessType accessType, DateTime accessTime, IOrganizationContext context)
> at Microsoft.Crm.Authentication.UserManagementFactory.UpdateAccessTime(Guid userId, Guid organizationId, UserAuthenticationAccessType accessType, DateTime accessTime)
> at Microsoft.Crm.Authentication.WindowsIdentityAuthorizationManager.Authenticate(OperationContext operationContext)
> at Microsoft.Crm.Authentication.WindowsIdentityAuthorizationManager.CheckAccessCore(OperationContext operationContext)
> at System.ServiceModel.Dispatcher.AuthorizationBehavior.Authorize(MessageRpc&amp;amp; rpc)
> at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc&amp;amp; rpc)
> at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)</ExceptionString><DataItems><Data><Key>HelpLink.ProdName</Key><Value>Microsoft SQL Server</Value></Data><Data><Key>HelpLink.EvtSrc</Key><Value>MSSQLServer</Value></Data><Data><Key>HelpLink.EvtID</Key><Value>21</Value></Data><Data><Key>HelpLink.BaseHelpUrl</Key><Value>http://go.microsoft.com/fwlink</Value></Data><Data><Key>HelpLink.LinkId</Key><Value>20476</Value></Data></DataItems></Exception></TraceRecord>

I originally thought that something was wrong with the restored CRM database, when I found a similar bug report. The comments clearly indicated that there was a problem with (another) CRM database, rather than CRM itself.

Go to the Services control panel and stop the services accessing the MSCRM databases

• Microsoft CRM Async Service (MSCRMAsyncService)
• Microsoft Dynamics CRM Unzip Service (MSCRMUnzipService)
• MS CRM Sandbox Service (MSCRMSandboxService)

and the corresponding IIS webs or the IIS service in IIS Manager.

In SSMS, execute the statement

EXEC sp_who

to find out whether there are open connections to the database causing the error.

Next, run

DBCC CHECKDB
WITH NO_INFOMSGS

to check the database for errors and consistency. Some of the results (message codes) I got were (original in German, English versions as found on the internet):

Meldung 8925, Ebene 16, Status 2, Zeile 1
Tabellenfehler: Objektquerverknüpfung: Seite (1:366305), Slot 0, in Objekt-ID 229575856, Index-ID 1, Partitions-ID 72057609057206272, Zuordnungseinheits-ID 72057594044350464 (LOB data-Typ) verweist auf Seite (1:366560), Slot 0, in Objekt-ID 229575856, Index-ID 1, Partitions-ID 2377900618270900224, Zuordnungseinheits-ID 72057594044350464 (LOB data-Typ).

Msg 8925, Level 16, State 1, Line 1
Table error: Cross object linkage: Page (1:5193082), slot 89, in object ID 1575168857, index ID 6, refers to page (1:5193939), slot 0, in object ID 1575168857, index ID 1.

Meldung 8928, Ebene 16, Status 1, Zeile 1
Objekt-ID 162815642, Index-ID 1, Partitions-ID 72057609051242496, Zuordnungseinheits-ID 72057594068992000 (LOB data-Typ): Seite (1:599672) konnte nicht verarbeitet werden. Einzelheiten finden Sie in anderen Fehlermeldungen.

Msg 8928, Level 16, State 1, Line 2 Object ID 645577338, index ID 0: Page (1:168576) could not be processed. See other errors for details.

Meldung 8929, Ebene 16, Status 1, Zeile 1
Objekt-ID 162815642, Index-ID 1, Partitions-ID 72057609051242496, Zuordnungseinheits-ID 72057609471328256 (In-row data-Typ): Es wurden Fehler in Daten außerhalb von Zeilen gefunden mit der ID 1288110080, im Besitz von data, Datensatz identifiziert durch RID = (1:1036135:0).

Msg 8929, Level 16, State 1, Line 1
Object ID 2: Errors found in text ID 852426752 owned by data record identified by RID = (1:110:19) id = 1925581898 and indid = 2.

Meldung 8939, Ebene 16, Status 98, Zeile 1
Tabellenfehler: Objekt-ID 162815642, Index-ID 1, Partitions-ID 72057609051242496, Zuordnungseinheits-ID 72057594068992000 (LOB data-Typ), Seite (1:599672). Fehler bei Test (IS_OFF (BUF_IOERR, pBUF->bstat)). Die Werte sind 12716041 und -4.

Msg 8939, Level 16, State 1, Line 2 Table error: Object ID 1797637899, index ID 0, page (1:168576). Test (m_headerVersion == HEADER_7_0) failed. Values are 0 and 1.

Meldung 8961, Ebene 16, Status 1, Zeile 1
Tabellenfehler: Objekt-ID 229575856, Index-ID 1, Partitions-ID 72057609057206272, Zuordnungseinheits-ID 72057594044350464 (Unknown-Typ). Der Datenknoten außerhalb von Zeilen auf Seite (1:366501), Slot 0, Text-ID 35185262002176 stimmt nicht mit seinem Verweis von Seite (1:366305), Slot 0 überein.

Msg 8961, Level 16, State 1, Line 1
Table error: Object ID 434100587, index ID 1, partition ID 72057594146521088, alloc unit ID 71804568277286912 (type LOB data). The off-row data node at page (1:2487), slot 2, text ID 341442560 does not match its reference from page (1:2487), slot 0.

Meldung 8965, Ebene 16, Status 1, Zeile 1
Tabellenfehler: Objekt-ID 162815642, Index-ID 1, Partitions-ID 72057609051242496, Zuordnungseinheits-ID 72057594068992000 (LOB data-Typ). Auf den Datenknoten außerhalb von Zeilen auf Seite (1:599672), Slot 0, Text-ID 1288110080 wird von Seite (1:1036135), Slot 0 verwiesen, er wurde jedoch im Scan nicht betrachtet.

Msg 8965, Level 16, State 1, Line 1
Table error: Object ID 2. The text, ntext, or image node at page (1:438), slot 3, text ID 852426752 is referenced by page (1:71), slot 4, but was not seen in the scan.

DBCC finished with the message

Von CHECKDB wurden 0 Zuordnungsfehler und 10 Konsistenzfehler in der My_MSCRM-Datenbank gefunden.
repair_allow_data_loss ist die minimale Reparaturstufe für die Fehler, die mit DBCC CHECKDB (My_MSCRM) gefunden wurden.

CHECKDB found 0 allocation errors and 29 consistency errors in database ‘My_MSCRM’.
repair_allow_data_loss is the minimum repair level for the errors found by DBCC CHECKDB (My_MSCRM).

This final message tells you to run DBCC with the REPAIR_ALLOW_DATA_LOSS option. This option can only be applied if the database runs in single-user mode:

ALTER DATABASE MY_MSCRM 
SET SINGLE_USER
WITH ROLLBACK IMMEDIATE;
DBCC CHECKDB ('MY_MSCRM', REPAIR_ALLOW_DATA_LOSS)
WITH NO_INFOMSGS;

If this command still lists unrecoverable errors, use the REPAIR_REBUILD option:

DBCC CHECKDB ('MY_MSCRM', REPAIR_REBUILD)
WITH NO_INFOMSGS

After completion, reset the database to multi-user mode again:

ALTER DATABASE MY_MSCRM SET MULTI_USER 
WITH ROLLBACK IMMEDIATE

Start the CRM services again, as well as the IIS web sites.