Some systems are experiencing issues

About This Site

This page offers performance, outage and maintenance information for all of GMS services and applications. Check back here any time to view the current status of the services listed below. For additional information or to report a problem, send an email to pchelp@gms.com and include as much information as you can along with your contact info.

Past Incidents

Monday 14th October 2024

Power Issue GMS Suwanee Datacenter - Planned Maintenance - UPS System Replacement: PDU Substation 1A Cut Over

10/15/2024 08:00 am - 11:59 pm EDT: QTS Critical Operations Technicians (COTs) and third-party contractors at the Suwanee, GA data center will perform important power switchovers to connect backup power to PDU Substation 1A. This will maintain N+1 redundancy while they work on updating the UPS System. During this process, there will be brief interruptions to power from the affected substation, which will temporarily put customer loads on a single power source (no backup). There is a chance for High impact to GMS equipment during this change at the Datacenter.

Friday 11th October 2024

GTS Poulsbo - Network outage

10/11/2024 8:00AM EST - GTS Poulsbo phones and network are down. ISP Lumen has a fiber outage in the area being worked on, no eta. Lumen Ticket #: 30382365

  • 2024-10-11 17:48:31 GMT - Crews in the field are accessing a manhole for further testing to narrow down the location of the damage.

    2024-10-11 17:04:11 GMT - Efforts to pinpoint the damage are ongoing, as fiber test results are compared with fiber records.

    2024-10-11 16:30:59 GMT - Technicians have arrived in the area of the suspected damage; however, no physical signs of damage are apparent. Additional personnel are en route to a nearby facility for further testing to triangulate the fault.

    2024-10-11 15:51:39 GMT - Lumen personnel are en route to the suspected fault site to locate and inspect the physical damage. Fiber crews are continuing to mobilize; an ETA will be provided once available.

    2024-10-11 15:47:55 GMT - Lumen personnel are en route to the suspected fault site to locate and inspect the physical damage. Fiber crews are continuing to mobilize; an ETA will be provided once available.

    2024-10-11 15:11:37 GMT - Fiber testing has identified a fault approximately 6 miles from the first test site and 1.5 miles from the second test site in Tacoma. Fiber crews are being mobilized. An ETA is pending.

    • Thursday 10th October 2024

    Carrier (ISP) Cherokee - Jackson - Network Outage

    10/9/24 22:10 CDT - Network Outage - Cherokee Jackson, power verified, local contact is Mark Zipp 731-499-3575 - Spectrum issue which they had a tech fix. Back online 10/10/24 6:25 CDT. Total outage time 8 hours and 15 minutes.

    Tuesday 8th October 2024

    Evalon/Converge Credit Card Machine service outage.

    10/8/2024 13:37 EDT - Evalon/Converge Credit Card Machine service outage.

  • 10/8/2024 15:53 EDT - Evalon/Converge Credit Card Machine services are online.

    • Trend/WebUI Outage: Due to datacenter power outage.

    10/8/2024 8:54 EDT - Trend/WebUI Outage: Due to a brief power event at our datacenter, a load balancer at the front of the Trend ERP briefly failed to come back online. We have personnel headed to the datacenter to physically check the equipment, however the equipment is showing online right now and the ERP is coming back online at present

  • 10/8/2024 9:06 EDT - Trend/WebUI restored: Datacenter back online and trend load back to normal.

    • Friday 4th October 2024

    Okta Smartsheets Outage

    09/04/2024 9:25 EDT: We are currently experiencing an outage with Smartsheets, affecting the ability to access Smartsheet's company wide. GMS IT is aware of the issue and is working to resolve the problem as quickly as possible.

    Power Issue GAT Sarasota - Network Outage

    10/3/24 17:38 EDT - Network Outage due to a power surge and tripped breaker. Resetting the breaker got everything back up and running at 10/4/24 6:44 EDT. Total outage time 13 hours and 6 minutes.

    Thursday 3rd October 2024

    Cisco Email Security IPv6 from Microsoft causing Remote Email systems alignment issues

    We have been made aware that Microsoft is already processing email through IPv6 which for the security side is handled slightly different than IPv4 (current standard). As a result, remote email systems can't properly process emails and end up stuck in their security stack/tools. We are working with our vendor, Cisco, to come up to par and align/match what Microsoft is doing.

    Tuesday 1st October 2024

    Syncplicity Mass role update in Syncplicity

    We are aware that some users have experienced disruptions with access to Syncplicity. This appears to have been caused by an automated process that unexpectedly adjusted user access.

    Please rest assured that we are actively investigating the issue alongside our third-party vendors to understand the cause and resolve it as quickly as possible. Our goal is to restore full access to everyone impacted, and we'll provide updates as soon as we have more information.

    Monday 30th September 2024

    Carrier (ISP) CMS Statesboro - Network Outage

    9/27/24 4:11 EDT - Network Outage - Power verified on Monday 9/30/24 morning - Equipment restarted, local contact is Roger Wentz 912-486-4397 - Damage visible outside the building to the fiber coming in. AT&T Ticket number CS1042334 From AT&T. AN ACCESS SUPPLIER PROBLEM HAS BEEN DETECTED. THE TICKET HAS BEEN REFERRED TO THE ACCESS SUPPLIER FOR FURTHER ANALYSIS. AT&T DIAGNOSE HAS DETECTED THAT THE PINGS BETWEEN THE PROVIDER EDGE AND CUSTOMER EDGE ROUTERS ARE UNSUCCESSFUL. FURTHER TROUBLESHOOTING WILL BE PERFORMED BY AT&T. PLEASE CHECK TICKET FOR ADDITIONAL STATUS. Cradlepoint is being shipped to the site as a temporary solution. CP#12 - Device IMEI 350884581087817

  • Sent a cradlepoint #12 for them to use until AT&T fixes the issue.

    • Tuesday 24th September 2024

    Cisco Email Security Emails to USG.com bouncing back

    We are aware of an issue where emails sent to USG.com are bouncing back. After investigating, we’ve found that USG’s email filtering systems are modifying the emails during their security scans, which changes the email’s source address to their system’s IP (148.163.140.163). This is causing the emails to be blocked.

    We have reached out to USG and offered to work together on resolving the issue. Since we don’t manage their security systems or have access to their environment, we’re dependent on their team to address this on their end.

    We will keep you informed as soon as we have updates. Thank you for your understanding and patience.

  • Monitoring shows no further incidents. Safe to close.

  • Got confirmation that this issue has been resolved by cbarron@usg.com Charles Barron @ USG.

    • Thursday 19th September 2024

    WBM Oakland phones working one direction. Telware ticket: 7698

    9/19/2024 12:29 EDT - WBM Oakland phones working one direction. Telware ticket: 7698

  • 9/23/2024 10:50 EDT WBM Oakland phones working one direction. Telware ticket: 7698

  • 9/20/2024 14:41 EDT - WBM Oakland phones repaired per Telware ticket: 7698.

    • Wednesday 18th September 2024

    Carrier (ISP) Cherokee Jonesboro - ISP outage in the area, network and phones down. No ETR.

    09/18/2024 10:00AM EST - Cherokee Jonesboro lost network and phones, issue caused by ISP Optimum circuit outage in the area. No ETR.

  • 09/18/2024 7:29PM EST - ISP Optimum circuit outage in the area was resolved by the ISP

    • Tuesday 17th September 2024

    TAM Bloomington - Network offline: due to power outage.

    9/17/2024 12:25 EDT - TAM Bloomington: Network offline due to area power outage.

  • 9/17/2024 13:08 EDT - TAM Bloomington: Network restored after power restoration.

    • Monday 16th September 2024

    Cowtown Dallas: Network outage. Due to downed fiber line.

    9/16/24 00:59 - Cowtown Dallas: Network outage. Due to downed fiber line.

  • 9/16/24 13:08 - Cowtown Dallas: Network restored. Due to downed fiber line. Secondary Spectrum provider back online.

    • Friday 13th September 2024

    CBSI - Dover - Network Outage

    9/12/24 22:21 EDT - Network Outage - Came back up at 9/13/24 5:56 EDT Issue was a surge protector that they are replacing. Ordered an APC UPS for them. Total outage time - 7 hours and 35 minutes

    Thursday 12th September 2024

    Carrier (ISP) GTS Puyallup 9/12/2024 1:00PM EST - Network and phones down. Comcast is experiencing an unexpcted outage in the Puyallup, WA area due to a power outage. No etr currently.

    GTS Puyallup 9/12/2024 1:00PM EST - Network and phones down. Comcast is experiencing an unexpcted outage in the Puyallup, WA area due to a power outage. No etr currently.

  • GTS Puyallup 9/12/2024 1:00PM EST - Power restored to area, site is back online.

    • O365 Exchange Microsoft is investigating issues with users accessing O365 Services

    Unable to access one or more Microsoft 365 services

    Issue ID: MO888473 Affected services: Microsoft 365 suite Status: Investigating Issue type: Advisory Start time: Sep 12, 2024 at 8:27 AM EDT

    User impact Users may be unable to access one or more Microsoft 365 services.

    Scope of impact Impact is specific to a subset of users who are accessing the service from the US.

    Current status Sep 12, 2024 at 8:28 AM EDT We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

  • Users may have been unable to access one or more Microsoft 365 services

    Issue ID: MO888473 Affected services: Microsoft 365 suite Status: Service restored Issue type: Incident Start time: Sep 12, 2024 at 7:46 AM EDT End time: Sep 12, 2024 at 9:14 AM EDT

    User impact Users may be unable to access one or more Microsoft 365 services.

    More info Impacted Microsoft 365 services and applications included, but were not limited to:

    • Microsoft Teams
    • Exchange Online
    • SharePoint Online

    Scope of impact Impact was specific to a subset of users who are attempting to access the service from the North America region.

    Root cause A third-party Internet Service Provider (ISP) made a change within their managed-environment which resulted in impact.

    Next steps

    • We're working with the third-party ISP to better understand the issue, and develop improved resiliencies where possible.

    This is the final update for this incident.

    Current status Sep 12, 2024 at 10:31 AM EDT After an extended period of monitoring our network telemetry, we've confirmed the issue is now resolved.

  • Users may be unable to access one or more Microsoft 365 services

    Issue ID: MO888473 Affected services: Microsoft 365 suite Status: Service degradation Issue type: Incident Start time: Sep 12, 2024 at 8:27 AM EDT

    More info Impacted Microsoft 365 services and applications include, but aren't limited to:

    • Microsoft Teams
    • Exchange Online
    • SharePoint Online

    Scope of impact Impact is specific to a subset of users who are attempting to access the service from the North America region.

    Current status Sep 12, 2024 at 9:50 AM EDT We've worked with the third-party Internet Service Provider (ISP) and confirmed that a change within their managed-environment resulted in impact. The ISP has reverted the change and we're now seeing signs of recovery. We're continuing to monitor network telemetry data to ensure a full recovery. Next update by: Thursday, September 12, 2024 at 11:00 AM EDT

  • Users may be unable to access one or more Microsoft 365 services

    Issue ID: MO888473 Affected services: Microsoft 365 suite Status: Investigating Issue type: Incident Start time: Sep 12, 2024 at 8:27 AM EDT

    More info Impacted Microsoft 365 services and applications include, but aren't limited to:

    • Microsoft Teams
    • Exchange Online
    • SharePoint Online

    Scope of impact Impact is specific to a subset of users who are attempting to access the service from the North America region.

    Current status Sep 12, 2024 at 8:54 AM EDT We're reviewing network telemetry logs to identify the root cause of the issue and develop our remediation plan. Next update by: Thursday, September 12, 2024 at 10:00 AM EDT

    History of updates Sep 12, 2024 at 8:28 AM EDT We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

    • Wednesday 11th September 2024

    Carrier (ISP) CMI Tuscaloosa - Network Outage

    9/11/24 3:37 EDT - Network Outage - Sam Kelley is the local contact - 256-990-9279, Networking team has reached out to AT&T

  • 9/11/24 17:13 EDT CMI Tuscaloosa - Network restored. Due to ISP IP configuration issue.

    • Comsense: Experiencing an issue with our Cloud Environment.

    9/11/2024 9:39 EST Comsense: Experiencing an issue with our Cloud Environment.

  • 9/11/2024, 11:22 EDT - Comsense: Repaired known issue with Comsense cloud service.

    • Carrier (ISP) OVS Holland - Network Outage

    9/11/24 00:13 EDT - Network Outage - site came back online at 6:35 EDT without any intervention. Appears to be ISP related. I'll have the networking guys look at the logs. Total outage time was 6 hours and 22 minutes

    Sunday 8th September 2024

    CHE Memphis: Network outage.

    9/8/2024 23:38 - CHE Memphis: Network outage.

  • 9/9/2024 09:05 - CHE Memphis: Network restored. Restarted provider equipment.

    • Thursday 5th September 2024

    Carrier (ISP) CBSI - New Castle - Network Outage

    9/5/24 4:14 EDT - Network outage - Checked the YIAB and power was out to the switch. Had him restart the UPS. Local contact Cesil Saldivar - (443) 797-4194 Restarted the Comcast equipment. Restarted the Cienna as well. Damon needed to reload the config on the router. reference number for comcast coax cr159956177 Ticket CR159957679 Comcast sent a tech out and switched out the modem. Back online at 9/5/24 10:17 EDT for a total outage time of 6 hours and 3 minutes

    Wednesday 4th September 2024

    NEG Milford: Network Outage.

    9/4/2024 12:00 NEG Milford: Network Outage.

  • 9/4/2024 14:07 NEG Milford: Network restored. ISP/ATT is in outage CS0996180. Set location on secondary cradle point.

    • Tuesday 3rd September 2024

    WBM Santa Maria: Power Outage

    09/03/24 14:41 EDT WBM Santa Maria - Power outage / PG&E estimates power will be restored around 17:30 EDT

  • 09/03/24 15:46 EDT power restored

    • Power Issue CBSI - Dover - Network Outage

    9/1/24 8:03 EDT - Network Outage, Power outage needed to restart the UPS and things came back up at 8/3/24 7:24 EDT Total outage time 1 day, 23 hours, 21 minutes

    May need to replace the UPS.

    Friday 30th August 2024

    Kamco Manhatten: Network Outage

    8/30/2024 13:35 - Kamco Manhattan: Network Outage. Spectrum maintenance.

  • 8/30/2024 17:12 - Kamco Manhattan: Network Restored. Spectrum maintenance.

    • Thursday 29th August 2024

    Carrier (ISP) Colonial Fayetteville - Network Outage

    8/24/24 12:30 EDT Outage - Restarted networking equipment, ours & ISP's. Appears to be ISP related (Metronet with a Spectrum backup). Local contact is Andy Hinman - 910-929-8132 dispatch scheduled for Fayetteville. currently 12 to 3 ticket number ett860760

    Site back online 8/26/24 13:36 EDT - ISP repaired the line. Total outage time

    Wednesday 28th August 2024

    Carrier (ISP) TMG West Union - Network Outage

    8/28/24 7:17 EDT Network Outage, verified power and restarted our equipment and the ISP's, contact number for Jason is 8646081252. AT&T has been dispatched

    Tuesday 27th August 2024

    Power Issue CMI - Madison/Huntsville - Power Outage, network down

    8/27/24 9:01 EDT Power Outage took the network down. Power restored at 10:06 EDT for a total outage time of one hour and 5 minutes.

    GAT Sarasota CS - Network Outage

    8/27/2024 6:50 EST GAT Sarasota CS - Network Outage

  • 8/27/2024 7:02 EST GAT Sarasota CS - Network restored. ISP outage.

    • Thursday 22nd August 2024

    Power Issue GAT Clearwater - Power Outage

    8-22-24 10:55 EDT: GAT Clearwater - Power Outage. The Clearwater location currently has no power due to a power transmitter going down, ETR is unknown at this time.

  • 8-22-24 20:03 EDT Power restored; all networking functions operational.

    • Tuesday 20th August 2024

    CHP Farmington - Network Outage

    8-20-24 11:35 EDT CHP Farmington - Network Outage. IT teams investigating, will update as more information is known.

  • 8-20-24 14:14 EDT - Networking restored. Amiri remoted into ISR, updated configuration and had me reboot switch.

    • KAM Bronx - Network Outage.

    8/20/2024 03:27 EDT KAM Bronx - Network Outage.

  • 8/20/2024 14:56 EDT KAM Bronx - Network restored: Primary, wrong ip configuration on ISR; Secondary, bill unpaid (service cancelled).

    • Saturday 17th August 2024

    CBS Beltsville - Network Outage

    8/17/2024 18:29 EDT CBS Beltsville - Network Outage

  • 8/20/2024 8:38 EDT CBS Beltsville - Network back online, Comcast circuit hardware issue.

  • 8/19/2024 9:17 EDT CBS Beltsville - Service ticket started with Comcast. Ticket: si068067136.

    • Friday 16th August 2024

    DSI Lincoln - Network and phones down. Spectrum outage in the area due to 2 fiber cuts. Spectrum ticket ECT83458264

    8/16/2024 13:47 EDT - DSI Lincoln - Network and phones down. Spectrum outage in the area due to 2 fiber cuts. Spectrum ticket ECT83458264

  • 8-16-24 20:47 EDT - Spectrum repaired fiber cuts. Networking restored.