At a recent consulting engagement I was tasked with installing SharePoint 2010, Enterprise Edition. This was supposed to be a routine installation, with a standard setup to get the organization a baseline implementation of SharePoint 2010. During pre-setup meetings there were no red flags regarding infrastructure or other potential factors that could affect the installation process.
In distant past implementations the one thorny issue has been the User Profile Service Application (UPSA). Of course this is old news. And since the release of Service Pack 1 and the subsequent CU’s the USPA ‘sticking points’ were supposed to be a thing of the past – supposed to is the operative phrase here…
The installation for SharePoint 2010 was in fact routine. SharePoint was installed with SP 1 and the August 2011 CU. The setup went without incident, and the user profile service application provisioned as expected. A connection to the company Active Directory went without incident, and selection of Organization Unit containers within the AD were made. A full import of user profiles was performed, and the timer job for incremental imports was configured. People search worked wonderfully. All Was Well.
Review and Monitoring
A couple of days after the installation I went back to review event logs and the ULS. I was expecting to see some of the usual suspects in the logs, such as the 10016 error concerning DCOM launch privileges for the IIS WAMREG service (which was subsequently fixed). I was going to correct any deficiencies found in the error and warning logs.
What I was greeted with was a host of warnings and errors. What really caught my attention was the 234 warning about the FIM service, as well as the 5555 errors. Several warnings were present, similar to the following:
Event ID: 234
ILM Certificate could not be created: Cert step 2 could not be created: C:\Program Files\Microsoft Office Servers\14.0\Tools\MakeCert.exe -pe -sr LocalMachine -ss My -a sha1 -n CN=”ForefrontIdentityManager” -sky exchange -pe -in “ForefrontIdentityManager” -ir localmachine -is root
This issue and its resolution is well documented in the blog Clever Workarounds. Although this does not call for a reinstallation, the other errors were suspect. These included such things as Event ID 1004 (component detection failure) and Event ID 5555 (user profile application not available exception).
In addition, I was getting a number of Event ID 1004 warnings and 5555 errors. All of this made me consider as to whether it might be time to reinstall the UPSA. As this was an initial install, I wanted to make sure that for some reason the UPSA did not install correctly.
I decided to do a reinstallation, as it would be quick, and it would allow me to closely monitor the activity and see if there were any irregularities during the process. I decided to use the Microsoft SharePoint Escalation Team guidance as the basis for checking the process and progress to make sure that everything was correct.
So I removed the working UPSA and started the process of rebuilding it.
After removal, I did the following:
- Made sure that the following services were disabled
- Forefront Identity Manager Synchronization Service
- Forefront Identity Manager Service
- Made sure that the the Forefront Identity Management Certificates were removed from the following certificated stores
- Trusted Root Certification Authorities
- Trusted People
- Removed the application pool used for the UPSA with PowerShell
A Re-Installation Gone Bad
So the game was on – reinstall with the ULS set to verbose. Closely monitoring the installation.
This was supposed to be somewhat routine, but also detailed enough to make sure that nothing really egregious was happening. So I recreated the service application according to specification, creating a new service application pool with the appropriate account – the account granted replicating permissions on AD.
The logs began to roll, and what I saw was a lot of the following:
UserProfileApplicationProxy.InitializePropertyCache: Microsoft.SharePoint.SPEndpointAddressNotFoundException: There are no addresses available for this application.
at Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock)
at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock)
at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.RefreshProperties(Guid applicationID)
at Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValueNow(K key)
at Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValue(K key, Boolean asynchronous)
Failure retrieving application ID for User Profile Application Proxy 'User Profile and My Sites Service': System.NullReferenceException: Object reference not set to an instance of an object.
This before any attempt was made to provision either the User Profile or User Profile Synchronization Service.
Just to prove that this was indeed a problem, I attempted to start both the User Profile and User Profile Synchronization Services. It was just a guess that the synchronization service would get stuck – it did. And there was no evidence in the logs that it was even starting the provisioning process. Just more errors as shown above.
A Finer Look
So I went through the above exercise a number of times, as a sanity check to make sure that the process was sound. Each time I got the same results. Errors following the creation of the UPSA.
OK, so what gives.
I really studied the logs, focusing on the UPSA creation. I found the following line:
Really? Putting the My Site Host site collection in place is part of the standard build, and I know that the My Site Host site collection was created, and that it existed. None of the previously cited errors were in the logs during the initial installation of the UPSA.
Although I have not been able to isolate and say for sure what happened, it appears that the the My Site Host was corrupted at some point during the removal of the UPSA. I have not found this error in any of the other logs, though I did a light search as I was on a schedule.
So I did the following:
- Removed the UPSA
- Made sure that the FIM services were disabled, which they were
- Made sure the certificate stores did not have any Forefront certificates for SharePoint
- Deleted the My Site Host site collection
- Created the My Site Host site collection with the My Site Host template
- Rebooted the server
My next attempt to create the UPSA was clean, and the ULS had no errors related to the profile service. I started the User Profile Service, followed by the User Profile Synchronization Service. Both ramped up cleanly, with no delays or errors. I created a connection to Active Directory and did a profile import.
Everything worked as it should. And, after a couple more configuration changes, I was able to remove many of the warnings and errors that started me down the path in the first place.
In the end the UPSA worked. But, as has happened to many in the past, there was a requirement to pay a pound of flesh to the piece of code that demands so much…