Microsoft is reporting: Users may be unable to access or use some Microsoft 365 services and features Sunday 24th November 2024 01:20:00

Root Cause There was a backend internal Microsoft 365 service that was being decommissioned. During the decommissioning workflow, a process issue meant that the traffic for this service was not disabled as expected, and the decommissioning process continued. This meant that once the service had been removed, traffic intended for this service was still generated by other services that had interactions with it.

After the service was removed, these requests originally went to the expected default endpoint for processing, however, as the service no longer existed, the logic of this default endpoint was to redirect requests to another endpoint as its backup. This ‘backup’ endpoint is the primary endpoint for the Exchange Online service and some other Microsoft 365 services and features. The requests are then managed by the front-end components in this endpoint (referred to as Client Access Front End (CAFE) for the rest of this document). As with the previous endpoint, because the backend service no longer existed, the CAFE routing service was unable to direct the request to this service. As part of the CAFE routing service's more thorough attempts to resolve to the backend service, a synchronous call was made inside an asynchronous code path, causing threads to be held for an extended duration while attempting the lookup.

The requests made to this service are triggered in response to user activity, so as peak weekday traffic began, the significant increase in the volume of requests going down the thorough resolution path exhausted the available threads in the CAFE routing service, causing the service to stall and all further requests to the routing service to fail, causing impact to other requests.

Some products and services, such as Outlook on the web, are architectured in a way that has more reliance on the CAFE routing service, and this is why they experienced higher levels of impact compared with other products, such as the Outlook desktop client. Engineering Actions Summary After monitoring alerting identified an issue with the Exchange Online service, we triggered a high priority event and started to investigate. We quickly identified timeout errors on some CAFE components and performed targeted restarts, however, the issue continued to occur on different components and returned on previously impacted components. The action of targeted restarts was performed throughout the duration of the issue to provide some relief.

We identified a recent change causing increased memory usage in other CAFE services, and believed this was the root cause. We reverted the change and performed restarts so it would take effect and release memory, however, after monitoring and testing we confirmed that the issue was still occurring.

After further analysis and root cause investigations, we identified that a different update related to the decommission of a backend service was the underlying cause of the issue. We implemented a patch that stopped traffic to this background service so that no new CAFE routing service instances would become impacted. We then performed scans to identify all the unhealthy CAFE servers and then restarted them to remediate impact. Additionally, we took some unhealthy CAFE servers out of active service to route requests to healthy components and expedite recovery.