SMSMSE Cluster Resource Failed

I have 3 Exchange Servers 2003 running on 2 Active and 1 Passive configurations. When I failed over all the cluster resources from an Active node to the Passive node, the SMSMSE cluster resource fails. All the cluster resources will failed over back to the original Active node.

A check on the Exchange Server and found the following events entries in the Windows System Event log:

Event ID 1069 - Cluster resource 'SMSMSE ' in Resource Group 'EVS2' failed.

Event ID 10016 - The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID {3AE2B8E5-7555-4E0E-B56C-A804C743D1CB} to the user KANDTI\ClusterAdmin SID (S-1-5-21-3050210616-160733423-812606249-1103). This security permission can be modified using the Component Services administrative tool.

The following events entries also appear in the Windows Application Event log:

Event ID 371 - Failed to save settings to the shared storage location (G:\SMSMSE\SHARED).

Event ID 370 - Failed to retrieve settings from the shared storage location (G:\SMSMSE\SHARED).

The following steps were carried out to resolve the problem:

1. On the Domain Controller, click Start > Programs > Administrative Tools > Active Directory Users and Computers.

2. Open Users.

3. Right-click SMSMSE Admins.

4. Click Properties.

5. On the Members tab, add the Cluster Service account ClusterAdmin.

6. On the Windows taskbar, click Start > Programs > Administrative Tools > Cluster Administrator.

7. Right-click one of the nodes and click Stop Cluster Service.

8. When the cluster service stops, start it on the same node.

9. Repeat steps 6 - 8 for each cluster node.

It was later found out that my colleague who is the secondary Exchange Servers administrator removed the ClusterAdmin account from the SMSMSE Admins group as he was questioned by the auditor.

Installation Of The Microsoft .NET Framework 3.5 SP1 – KB951847

An installation of the Microsoft .NET Framework 3.5 SP1 (full package) – KB951847 had failed on a passive node Exchange 2003 server. Having spent a number of days looking at it, I finally managed to resolve it today and get the Exchange server to the latest patch status.

Initially I thought the installation failure is due to the corruption of either the MSI software update registration or the .NET Framework installed on the server.

I tried the following methods mentioned in the Microsoft KB articles (i.e. KB922377, KB976982) but were not successful :

1. Manually remove the .NET Framework, and then reinstall the .NET Framework

2. Remove the .NET Framework by using the Installation Cleanup Utility and then reinstall the .NET Framework

3. Fix the MSI software update registration corruption issue by using Microsoft Fix it 50123

A detailed look at the Windows Application Event log after each installation of the Microsoft .NET Framework 3.5 SP1 revealed the following entries:

Event ID 10005 - Product: Microsoft .NET Framework 2.0 Service Pack 2 -- The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2318. The arguments are: C:\WINDOWS\PCHEALTH\ERRORREP\QSIGNOFF\30E8DF.cab, ,

Event ID 1023 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework CLR' could not be installed. Error code 1603. Additional information is available in the log file C:\DOCUME~1\kandti01\LOCALS~1\Temp\dd_NET_Framework20_Setup5306.txt.

Event ID 1023 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework CA' could not be installed. Error code 1603. Additional information is available in the log file C:\DOCUME~1\kandti01\LOCALS~1\Temp\dd_NET_Framework20_Setup5306.txt.

Event ID 1023 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework CRT' could not be installed. Error code 1603. Additional information is available in the log file C:\DOCUME~1\kandti01\LOCALS~1\Temp\dd_NET_Framework20_Setup5306.txt.

Event ID 1023 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework PreXP' could not be installed. Error code 1603. Additional information is available in the log file C:\DOCUME~1\kandti01\LOCALS~1\Temp\dd_NET_Framework20_Setup5306.txt.

Event ID 1023 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update 'Dr. Watson' could not be installed. Error code 1603. Additional information is available in the log file C:\DOCUME~1\kandti01\LOCALS~1\Temp\dd_NET_Framework20_Setup5306.txt.

Event ID 1023 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework 1' could not be installed. Error code 1603. Additional information is available in the log file C:\DOCUME~1\kandti01\LOCALS~1\Temp\dd_NET_Framework20_Setup5306.txt.

Event ID 1023 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework 2' could not be installed. Error code 1603. Additional information is available in the log file C:\DOCUME~1\kandti01\LOCALS~1\Temp\dd_NET_Framework20_Setup5306.txt.

Event ID 1023 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework ASP .NET' could not be installed. Error code 1603. Additional information is available in the log file C:\DOCUME~1\kandti01\LOCALS~1\Temp\dd_NET_Framework20_Setup5306.txt.

Event ID 1023 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework WinForms' could not be installed. Error code 1603. Additional information is available in the log file C:\DOCUME~1\kandti01\LOCALS~1\Temp\dd_NET_Framework20_Setup5306.txt.

In the directory C:\Windows\PCHealth\ErrorRep\Qsignoff, there is indeed a file 30E8DF.cab. 

Further investigation into the permission for the Qsignoff folder, I found out that the group EVERYONE is being assigned with DENY permission to Traverse Folder/ Execute Files and the permissions apply onto the folder itself, subfolders and files. This could be the cause for the Event ID 10005 to occur.

The DENY permission was removed and the 3 files in the directory C:\Windows\PCHealth\ErrorRep\Qsignoff were deleted successfully.

After which the installation of the Microsoft .NET Framework 3.5 SP1 was carried out successfully with the following entries in the Windows Application Event log:

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework CLR' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework CA' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework CRT' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework PreXP' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update 'Dr. Watson' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework 1' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework 2' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework ASP .NET' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 - Update '.NET Framework WinForms' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 2.0 Service Pack 2 -- Installation completed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'WPF2_32' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'WCF' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'WPF_1' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'WF' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'WPF_Other' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'WF_32' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'WPF2' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'XPS' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'WCS' installed successfully.

Event ID 1022 - Product: Microsoft .NET Framework 3.0 Service Pack 2 - Update 'WPF_Other_32' installed successfully.

Event ID 11707 - Product: Microsoft .NET Framework 3.0 Service Pack 2 -- Installation completed successfully.

Event ID 11707 - Product: Microsoft .NET Framework 3.5 SP1 -- Installation completed successfully.

The Exchange server was restarted and applied with the Microsoft .NET Framework 3.5 Family Update - KB959209. This update addresses a set of known application compatibility issues and consists of 3 files (i.e. NDP20SP2-KB958481-x86.exe, NDP30SP2-KB958483-x86.exe and P35SP1-KB958484-x86.exe). A reboot was performed for the Exchange server and the server was subsequently patched with those outstanding .NET Framework updates pushed down via the WSUS server.

The MBSA scan was conducted on the Exchange server after all the updates were installed successfully. The patch status report shows that the Exchange server has been updated till the latest patch status.
 

Event ID 68 Unable To Initialize Scan Engine. The Virus Definitions May Be Missing Or Corrupt

Our Helpdesk reported that there are users complaining that they were unable to send out or receive emails. The system administrator performed a check on the Exchange Servers and found the following Symantec Mail Security for Microsoft Exchange events entries in the Windows Application Event log:

Event ID 110 - The process SAVFMSESp.exe failed to start (0xC009008A).

Event ID 168 - The process SAVFMSESp.exe was restarted.

Event ID 68 - Unable to initialize scan engine. The virus definitions may be missing or corrupt. Perform a LiveUpdate to retrieve the latest virus definitions.

Event ID 167 - The process SAVFMSESp.exe terminated unexpectedly.

The following steps were carried out to resolve the problem:

1. Stop the Symantec Mail Security for Microsoft Exchange service.

2. Open usage.dat file from the directory C:\Program Files\Common Files\Symantec Shared\VirusDefs\ and confirm that it's missing SAVFMSE_SP_x=1 entries, where x is a number from 1 to 9.

Example of how an invalid usage.dat file looks:
[20121024.017]
SMSMSE=1
[20121025.021]
DEFWATCH_10=1
NAVCORP_70=1
NAVCORP_70_2=1

Example of how a valid usage.dat file looks:
[20120424.005]
DEFWATCH_10=1
NAVCORP_70=1
SSS_MICROSOFT_EXCHANGE_30=1
SAVFMSE_SP_3=1
SAVFMSE_SP_1=1
SAVFMSE_SP_2=1

3. Remove VirusDefs0000000x folders from the directory C:\Program Files\Common Files\Symantec Shared\definitions\Antivirus (where 'x' is a number).

4. Remove files from the directory C:\Program Files\Common Files\Symantec Shared\definitions\Antivirus\incoming.

5. Copy all files from the latest definition folder from the directory C:\Program Files\Common Files\Symantec Shared\VirusDefs\ to the directory C:\Program Files\Common Files\Symantec Shared\definitions\Antivirus\incoming.

6. Restart the SMSMSE service.

7. When the service starts up, SMSMSE will read new definition files from incoming folder and create new virus definition folder (i.e. VirusDefs0000000x)under definitions folder.