Those working regularly with SharePoint Server 2010 know the frustration with working with the User Profile Service (UPS). There are tons of blog articles are out there discussing the challenges. Let me add this one to the pile that frustrated me this week.
One of the common problems when working with UPS is when you try to view, add or edit synchronization connections. A error reported is the dreaded “The query returns nothing”. Before this week, I’ve seen two main reasons for this: 1) the FIM service is not running. 2) the synchronization service was changed to an account other than the farm account. This week I had a third reason, and it was the toughest yet.
The farm was a basic two-server farm. One WFE/app server, and one SQL server. Prior to my getting involved in the project, the UPS service was provisioned and a connection to AD was successfully added and a few thousand profiles were imported. Several days later, it stopped working and “The query returns nothing” was showing when trying to view the connections. Having confirmed that it wasn’t any of the usual reasons listed above, I started doing some deeper troubleshooting. I noticed that when trying to view the connections, the Windows application log would show an HTTP 407 error (proxy authentication required). I would also see this same 407 error in Central Admin if I tried to add a new connection to UPS. This caused me to do some serious head scratching. Why on earth am I getting a proxy server error? I could get to Central Admin, so I knew my logon’s proxy settings were working, but I double checked in IE anyway. They were correct. All communication should be on this one WFE/app server anyway—when viewing connections, nothing should be trying to connect to a proxy server. What gives?
We then used remote desktop to log onto the server using the same logon account. (Note: the account we were using was a regular user account that was a SharePoint farm administrator—we did not use the farm account to logon). We started Central Admin, tested it, and got the same problem. For the next hour or so, we started double checking everything, ULS logs, MIISClient.exe, you name it. I couldn’t find any good reason for this. I really wanted to run Fiddler, but that was not an option. Frustrating. So, I took a step back and reflected on what I knew about the architecture, and here are two of the key things that helped:
As described in the most authoritative resources on the UPS
"Stuck on Starting": Common Issues with SharePoint Server 2010 User Profile Synchronization
Rational Guide to Implementing SharePoint Server 2010 User Profile Synchronization
Synchronize user and group profiles is SharePoint Server 2013
you must grant the farm account the “allow log on locally” user right on the server that is configured to run the UPS synchronization service. (The UPS synch service runs under the context of the farm account). This is done to ensure the that it can provision the Forefront Identity Management components when the service is started. And, in fact, when starting the UPS synch service, the farm account does log on locally. This is evidenced by a local Windows profile that is created. (BTW, do not confuse this Windows Profile with a SharePoint User Profile – totally different concepts but the same term. You can see the Windows profiles here: Control Panel –> System –> Advanced System Settings –> User Profile, Settings button.)
When you are working with with UPS synchronization connections (displaying, creating, etc.), communication between the UPS service app and Forefront Identity Manager (FIM) occurs. Specifically, the UPS service app calls into an HTTP service provided by the FIM Service on port 5725. (Note: this service is not run within IIS but the Microsoft.ResourceManagement.Service.exe, the process name for the FIM Service).
It was about this time I tried to put these two important bullets together in light of the problem: When viewing the list of connections, the farm account calls into the FIM service to retrieve connection details. This was not working and throwing a 407 error. These next two questions came to mind:
1. Is it possible that the farm account is using the proxy settings in its local Windows profile when calling into the FIM service?
2. If yes, what are the proxy settings for the farm account in this Windows profile?
To test, we log into the server using the farm account. Not advised, but sometimes you do what you gotta do. The proxy server settings were configured correctly as per the domain policy, but we decided to turn off the proxy server off and try again from the remote browser. It worked! The connection was now displaying correctly. We flip the proxy server back on and sure enough, we get the “The query returns nothing” problem from the remote browser. We decided to keep the proxy server off, since the farm account will not be connecting to any external resources that require a proxy server. Problem solved.
Lesson learned: When the UPS service app calls into the FIM, it is using the settings of its local Windows profile on the server which is running the UPS synch service. And if these settings are not configured correctly (e.g. because of a GPO), it will cause the communication to fail. In our case, since it had been working, our only guess is that a policy change occurred at the domain level that affected the Windows profile.