Past Incidents

3/20/2025 10:41 EDT: GMS VPN - VPN service degraded. Users getting 'Server Communication Error' upon trying to connect.

3/19/2025 9:41 EDT: DSI Lincoln - Network outages. Due to inclement weather in area.

3/13/25 - 6:58 EST - Network outage, Spectrum issue, Spectrum Ticket ECT-84211832

Site came back online at 8:45 EST with two one minute outages afterwards.

Total outage time - 1 hour and 47 minutes

03/12/25 10:31 AM EDT Cherokee Memphis - Network Outage: Yard reported network was down and power was verified with yard. Cycled power on ISR and ISP equipment and site came back online 11:34 AM EDT

3/10/2025 14:08 EDT: RBM Chattanooga - Network Outage.

3/5/25 - 7:40 EST Network Outage Spectrum outage - back up and online at 8:33 EST

3/4/2025 10:00AM EDT COW Ft Worth North: Network outage: Power outage due to high winds taking down nearby power poles. ETR tonight or early morning

2/26/25 - 6:29 EST - Network outage - Gator - Bonita Springs has an outage, power verified, restarted equipment and Comcast modem. Local contact if Alex McDowell 2393625518 Comcast outage verified - Network restored at 8:41 EST for a total outage of 2 hours and 12 minutes

2/21/25 - 4:45 EST - Network Outage, yard going up and down until 2/21/25 - 6:16 EST, suspected ISP issue. Total outage time 1 hour and 31 mintues.

2/20/2025 06:00AM EDT - JB Perris network and phones down. Frontier ticket 3783584

2/18/2025 12:56 EDT OneCloud - Vendor reporting outage. Issue causing phone to go to Voicemail and state location is closed.

2/14/25 - 6:56 CST - Network outage for 6 minutes until 7:02 CST. Suspected ISP blip.

All Deployments: Admin Panel Logins and Telephony Authentication

New incident: Investigating

We are currently investigating an issue causing Admin Panel login and SMS/Phone Call delivery failures across all deployments.

02/12/25 12:08 PM EST Chaparral Materials Albuquerque Sto - Network Outage due to provider outage (Comcast). Due to be resolved 15:00 EST

2/11/2025 15:09 EDT TMG Evans: Network outage.

2/6/25 - 7:15 CST - Network Outage, appears to have been a blip at the Dallas datacenter, restored at 7:33 CST for a total outage time of 17 minutes.

Issue ID: EX1021490 Affected services: Exchange Online Status: Service degradation Issue type: Advisory Start time: Mar 2, 2025, 7:14 PM EST

More info Users may be able to click "continue" when prompted and navigate back to re-entering their password to attempt to resolve impact. Additionally, some users may be able to bypass the prompt to sign-in by clicking "edit settings" and then entering into the app.

Scope of impact Any user serviced by the affected infrastructure may experience issues with accessing email messages on the iOS native mail app.

Current status Mar 3, 2025, 6:58 AM EST We've so far been unable to identify any Microsoft service updates which we believe are contributing towards impact. However, we've observed through analysis of telemetry that an accumulation of authentication token errors associated with a third-party application may be causing the issue. We're continuing our review of available diagnostic data to confirm the root cause. Next update by: Monday, March 3, 2025 at 9:00 AM EST

History of updates Mar 3, 2025, 4:54 AM EST We're progressing with our review of recent scheduled service updates and diagnostic monitoring to isolate the root cause, before we begin assessing potential mitigation options. Mar 3, 2025, 2:37 AM EST We're continuing to assess and analyze our service logs, as well as monitor the service health and check for any changes that were made recently to the service, to understand the root cause of impact. Mar 3, 2025, 12:55 AM EST The review of the failures and service logs is ongoing. We're continuing our analysis to help understand the root cause of impact. Mar 2, 2025, 10:56 PM EST We identified some failures present in our service logs and we're continuing to review these, along with additional support case details, to help identify the cause of impact and develop our next troubleshooting steps. Mar 2, 2025, 8:53 PM EST We're continuing to review our service logs, along with provided screenshots, to better understand the cause of impact. Mar 2, 2025, 7:49 PM EST We're reviewing our service logs to aid in understanding the cause of impact and to develop our next steps. Mar 2, 2025, 7:22 PM EST We're investigating a potential issue with Exchange Online and checking for impact to your organization. We'll provide an update within 30 minutes

01/30/25 14:22 EST ASI Southfield - Network Outage for approx 7 minutes. Network restored on its own 14:29 EST

1/29/2025 18:01 GTS Missoula - Network Outage.

1/23/25 10:54 PST - Network Outage, verified power, restarted equipment - local contact is Carlos Navarrete - 5034772585 Carlos said that the Comcast ISP seems to be working as his phone was able to connect to the xfinitywifi