Problems with Free/Busy from O365 to on-premises users

I was recently working on a customer solution with Exchange 2013 configured in a hybrid setup with Office 365.  I was having issues with free/busy requests from Exchange Online mailboxes to on-premises mailboxes.

Here is a description of the problem and the eventual resolution which took a few days and late nights to resolve:

Scenario

With a recent deployment of Exchange Server 2013 CU6 in a hybrid configuration with Office 365 user that had mailboxes on O365 couldn’t query for free/busy of on-premises users.  Users with on-premises mailboxes could query the free/busy status of O365 mailboxes.  The free/busy test for Office 365 in the Exchange Remote Connectivity Analyser (http://exrca.com ) showed the test for free/busy queries from O365 to on-premises as working successfully.

Environment

·        A multi-domain Active Directory forest with all DCs running Windows Server 2012R2

·        A single Exchange Server 2013 CU6 server with Mailbox/CAS roles installed, running on Windows Server 2012R2

·        DirSync and ADFS running on Windows Server 2012R2

·        Exchange Hybrid configured by running the hybrid configuration wizard on the Exchange Server 2013 CU6 server

·        Clients running Windows 8.1 and Outlook 2013 SP1

·        Clients running IE11 and connecting to OWA on the O365 portal

Symptoms

·       OWA and Outlook clients on O365 could not query free/busy for on-premises mailboxes.  The hash markings were displayed with the error “The recipient’s server could not be contacted”

·         The Exchange Remote Connectivity Analyser could retrieve free/busy from O365 to on-premises users

·       In the IIS logs on the CAS server (c:\inetpub\logs\logfiles\w3svc1) you could see successful connections from the ExRCA, but when actual users queried the free/busy status there was an IIS Internal Error 500 logged

Troubleshooting steps

The following troubleshooting steps were performed:

·         Ensure that the Organisation Relationship on-prem had the correct domains for the O365 users (get-organizationrelationship | fl DomainNames).  They should be the vanity domain and the xxx.onmicrosoft.com

·         Ensure that the Organisation Relationship on-prem had the correct free/busy settigns (get-organizationrelationship | fl free*).  They should be set to allow free/busy with limited details

·         Repeat the above 2 tests from the O365 side and ensure that the correct domains and settings are in place to allow on-premises to contact O365

·         Check the federationTrust settings by running get-federationtrust on both on-premises and O365

·         Check web-services on-premises and ensure free/busy is working for on-premises-on-premises

·         Re-run the hybrid wizard in case something wasn’t set correctly

At this stage no further progress could be made so a call was logged with Office 365 support.  They connected remotely to our solution and confirmed that all the above settings were correct.  They also checked that the O365 service itself could issue queries into on-premises.  The call was then escalated within Microsoft performed the following:

·         Run the free/busy query from an Outlook client with Advanced Troubleshooting logging enabled and captured the ETL trace files (placed in %temp%\Outlook Logging)

·         Install the ExTra utility on the Exchange Server.  This was previously built into Exchange Server in 2010 and prior but has since been removed and can only be obtained from Microsoft as part of a troubleshooting call.  The was configured to capture a huge range of tracing information

At this point Microsoft took away the logs and did some overnight analysis.  It then became apparent what the problem was:

·       They could see in their server trace files the error “ExceptionMapper.TryGetServiceError called for exception: Microsoft.Exchange.Services.Core.Types.ServiceAccessDeniedException: The application is missing a linked account for RBAC roles, or the linked account has no RBAC role assignments, or the calling users account is logon disabled.”

·       Running the command Get-ManagementRoleAssignment – RoleAssignee “Exchange Online-ApplicationAccount” | fl name,effectiveusername.  This showed that the account that the hybrid servers used to connect was “Exchange Online-ApplicationAccount” located in the root domain

·       Running “get-partnerapplication | fl linkedaccount” returned a blank result

Basically the above means that the RBAC roles to allow O365 to query free/busy on-premises had been setup but the partner application had not been told to use the correct account.  This should have been configured by the Hybrid Configuration Wizard but this hadn’t happened even though the wizard had been run more than once.  It was resolved by:

·       Running ‘“get-partnerapplication |set-partnerapplication –linkedaccount “<rootdomainFQDN>/users/Exchange Online-ApplicationAccount” ‘

·       Force multiple forest-wide AD synchronisations using the repadmin /syncall command

·       Restart all Exchange Services.  This configuration change will not take effect until this has been done

After performing the above the problem was resolved and free/busy worked as it should.

Learning Points

The above problem has led to the following learnings:

·       Prior to Exchange Server CU6 O365 performed queries to on-premises using user-impersonation

·       In an Exchange Server 2013 CU6 hybrid deployment O365 now uses OAuth to connect back to on-premises

·       The Remote Connectivity Analyser does not currently check using the new authentication mechanism.  This means that although you get a successful result it is using a different mechanism and therefore cannot actually tell you if your solution is working

·       Even after multiple runs of the HCW the above setting was never set.  This could be a bug in CU6 but has not been confirmed

·       Sometimes you are unable to resolve a problem yourself as you do not have access to the level of logging that Microsoft do and may therefore sometimes require their assistance

·       It is definitely worth having a Premier Support contract.  Without this the resolution of the above issue could have taken an extended period of time.  Even with Premier Support the large amount of log analysis meant that resolution took 4 days