Some systems are experiencing issues

About This Site

This page offers performance, outage and maintenance information for all of GMS services and applications. Check back here any time to view the current status of the services listed below. For additional information or to report a problem, send an email to pchelp@gms.com and include as much information as you can along with your contact info.

Past Incidents

Wednesday 20th November 2024

GTS Seattle - Site lost network and phones. Investigating

11/20/2024 03:00AM EST - GTS Seattle lost network and phones. Investigating cause.

GTS Woodinville - Site lost network and phones. Investigating

11/20/2024 03:00AM EST - GTS Woodinville lost network and phones. Investigating cause.

GTS Kirkland - Site lost network and phones. Investigating

11/20/2024 03:00AM EST - GTS Kirkland lost network and phones. Investigating cause.

Monday 18th November 2024

CMS Ridgeland: Network outage. Provider outage in area.

11/18/2024 11:18 EDT - CMS Ridgeland: Network outage. Provider outage in area.

  • 11/19/2024 01:54 EDT - CMS Ridgeland: Network restored. Provider outage in area.

  • 11/18/2024 14:50 EDT - CMS Ridgeland: Network outage. Provider outage in area.

    • AMES 3219 - Network outage. Provider outage in area.

    11/18/2024 11:18 AMES 3219 - Network outage. Provider outage in area.

  • 11/18/2024 13:54 AMES 3219 - Network restored. Provider outage in area.

    • Friday 15th November 2024

    Capture OnTheGo: Server degraded.

    11/15/2024 9:04 EDT - Capture OnTheGo: Server degraded.

  • 11/15/2024 16:04 EDT - Capture OnTheGo: Server restored after settings adjusted by Security Team.

    • Thursday 7th November 2024

    Carrier (ISP) VIP Nashville - Network Outage

    11/6/24 21:40 CDT - VIP Nashville has a network outage, had them reboot equipment, local contact Armando Carrasco 615-670-1470

    Monday 4th November 2024

    Okta Dayforce not loading

    Dayforce is extremely slow to load

  • 11/4/2024 16:20 EST Dayforce website running normal after vendor outage and performance issues.

    • TAM Cedar: Network Outage.

    11/4/2024 7:56 EDT - TAM Cedar: Network Outage.

  • 11/4/2024 11:46 EDT - TAM Cedar: Network restored. Line pole down in area causing outage.

    • Friday 1st November 2024

    Local Network Ames network degraded

    The Ames datacenter network is in a significant degraded state. The server vlan .148 is unreachable from other vlans. This is impacting several systems including Citrix. Networking team is working on the issue.

  • Network team made adjustments and Ames datacenter network is operational again.

    • Thursday 31st October 2024

    ASI-Southfield Network Outage.

    11/31/2024 12:31 EDT ASI-Southfield Network Outage.

  • 11/31/2024 16:26 EDT ASI-Southfield Network restored. Local ISP outage.

    • Tuesday 29th October 2024

    Carrier (ISP) Tannerbolt - Brooklyn - Network Outage

    10/28/24 17:14 EDT - Network Outage Power verified local contact Maleek St Omer 347 307 0543, restarted all equipment, suspected Verizon issue, after talking with Maleek he said there was a large electrical fire in the area which may be the cause.

    Monday 28th October 2024

    Carrier (ISP) GTS Seattle - Network Outage

    10/27/24 7:00AM MDT - GTS Seattle has a network outage, power to the equipment has been verified and recycled. Lumen ticket 30492874 in progress, no eta.

  • 10/28/2024 2:30PM EST - Power restored to Cisco Meraki, site is back online.

    • Carrier (ISP) CHP - Farmington - Network Outage

    10/27/24 4:36 MDT - Network Outage - Verified power and ISR has green lights. Restarted all equipment including Comcast. Local contact is Matt Olson - 505-252-2756 Having networking team check with the ISP.

  • Damon was able to work with the provider and get them back up. Back up at 10.28.24 12:20 MDT.

    • Sunday 27th October 2024

    CMI Macon - Network Outage.

    10/27/2024 13:51 EDT: CMI Macon - Network Outage.

  • 10/30/2024 08:14 EDT: CMI Macon - Network restored. Outage due to Cisco router failure. Replacement installed.

    • Thursday 24th October 2024

    Infor Web WEBUI / TREND Outage

    10/24/10:30PM The trend database server is experiencing issues and is being assessed. Trend web services may be impacted.

  • 10/24/2024 11:07PM EST - IT Team had to perform an immediate Emergency reboot of the Trend database server due to a security application crashing and locking up the server. Trend is back online.

  • Trend's production database server, PGS01, segmentation faults, repeatedly Ampdaemon faulting we are prepping for immediate Trend reboot

    • Carrier (ISP) CBMI - Harrisonburg - Network Issues

    10/22/24 1:54 EDT - Harrisonburg started having intermittent outages that last about an hour and then reconnect. Damon engaged Verizon and opened a ticket 202410222124. After numerous escalations and shipping out a cradlepoint as a stopgap the connection stablized at 10/23/24 20:54 EDT.

    GMS WIDE: The Kronos site is currently undergoing maintenance. Services will be restored as soon as possible.

    10/24/2024 6:00 EDT GMS WIDE: The Kronos site is currently undergoing maintenance. Services will be restored as soon as possible.

  • 10/25/2024 6:00 EDT GMS WIDE: The Kronos site maintenance is complete.

    • Wednesday 23rd October 2024

    TAM Iowa City - Network outage.

    10/23/2024 11:32 EDT: TAM Iowa City - Network outage.

  • 10/24/2024 13:02 EDT - TAM Iowa City - Network outage restored. Local provider outage in area.

  • Iowa City is still down, but they are going to dispatch today, since for whatever reason they weren't able to figure out the issue yesterday until like 10 o clock. "An outside tech will arrive at your location to isolate the troubles at that time. The next update you will receive on this repair will be when the tech fixes it and calls in to confirm..."

  • 10/23/2024 12:47 EDT - TAM Iowa City - Network outage. Ticket 30459343 started with provider.

    • Tuesday 22nd October 2024

    Carrier (ISP) GAT - Brooksville - Network Outage

    10/21/24 16:00 EDT Network Outage after Spectrum replace some equipment. The Spectrum tech unplugged their whole network stack and didn't plug it back in when he was done. It's been plugged back in and we're letting things boot back up. Amiri is checking on the ISP IP info. Network back up 10/22/24 11:01 EDT for a total outage time of 19 hours and 1 minute.

    Monday 14th October 2024

    Power Issue GMS Suwanee Datacenter - Planned Maintenance - UPS System Replacement: PDU Substation 1A Cut Over

    10/15/2024 08:00 am - 11:59 pm EDT: QTS Critical Operations Technicians (COTs) and third-party contractors at the Suwanee, GA data center will perform important power switchovers to connect backup power to PDU Substation 1A. This will maintain N+1 redundancy while they work on updating the UPS System. During this process, there will be brief interruptions to power from the affected substation, which will temporarily put customer loads on a single power source (no backup). There is a chance for High impact to GMS equipment during this change at the Datacenter.

    Friday 11th October 2024

    GTS Poulsbo - Network outage

    10/11/2024 8:00AM EST - GTS Poulsbo phones and network are down. ISP Lumen has a fiber outage in the area being worked on, no eta. Lumen Ticket #: 30382365

  • 2024-10-11 17:48:31 GMT - Crews in the field are accessing a manhole for further testing to narrow down the location of the damage.

    2024-10-11 17:04:11 GMT - Efforts to pinpoint the damage are ongoing, as fiber test results are compared with fiber records.

    2024-10-11 16:30:59 GMT - Technicians have arrived in the area of the suspected damage; however, no physical signs of damage are apparent. Additional personnel are en route to a nearby facility for further testing to triangulate the fault.

    2024-10-11 15:51:39 GMT - Lumen personnel are en route to the suspected fault site to locate and inspect the physical damage. Fiber crews are continuing to mobilize; an ETA will be provided once available.

    2024-10-11 15:47:55 GMT - Lumen personnel are en route to the suspected fault site to locate and inspect the physical damage. Fiber crews are continuing to mobilize; an ETA will be provided once available.

    2024-10-11 15:11:37 GMT - Fiber testing has identified a fault approximately 6 miles from the first test site and 1.5 miles from the second test site in Tacoma. Fiber crews are being mobilized. An ETA is pending.

    • Thursday 10th October 2024

    Carrier (ISP) Cherokee - Jackson - Network Outage

    10/9/24 22:10 CDT - Network Outage - Cherokee Jackson, power verified, local contact is Mark Zipp 731-499-3575 - Spectrum issue which they had a tech fix. Back online 10/10/24 6:25 CDT. Total outage time 8 hours and 15 minutes.

    Tuesday 8th October 2024

    Evalon/Converge Credit Card Machine service outage.

    10/8/2024 13:37 EDT - Evalon/Converge Credit Card Machine service outage.

  • 10/8/2024 15:53 EDT - Evalon/Converge Credit Card Machine services are online.

    • Trend/WebUI Outage: Due to datacenter power outage.

    10/8/2024 8:54 EDT - Trend/WebUI Outage: Due to a brief power event at our datacenter, a load balancer at the front of the Trend ERP briefly failed to come back online. We have personnel headed to the datacenter to physically check the equipment, however the equipment is showing online right now and the ERP is coming back online at present

  • 10/8/2024 9:06 EDT - Trend/WebUI restored: Datacenter back online and trend load back to normal.

    • Friday 4th October 2024

    Okta Smartsheets Outage

    09/04/2024 9:25 EDT: We are currently experiencing an outage with Smartsheets, affecting the ability to access Smartsheet's company wide. GMS IT is aware of the issue and is working to resolve the problem as quickly as possible.

    Power Issue GAT Sarasota - Network Outage

    10/3/24 17:38 EDT - Network Outage due to a power surge and tripped breaker. Resetting the breaker got everything back up and running at 10/4/24 6:44 EDT. Total outage time 13 hours and 6 minutes.

    Thursday 3rd October 2024

    Cisco Email Security IPv6 from Microsoft causing Remote Email systems alignment issues

    We have been made aware that Microsoft is already processing email through IPv6 which for the security side is handled slightly different than IPv4 (current standard). As a result, remote email systems can't properly process emails and end up stuck in their security stack/tools. We are working with our vendor, Cisco, to come up to par and align/match what Microsoft is doing.

    Tuesday 1st October 2024

    Syncplicity Mass role update in Syncplicity

    We are aware that some users have experienced disruptions with access to Syncplicity. This appears to have been caused by an automated process that unexpectedly adjusted user access.

    Please rest assured that we are actively investigating the issue alongside our third-party vendors to understand the cause and resolve it as quickly as possible. Our goal is to restore full access to everyone impacted, and we'll provide updates as soon as we have more information.

    Monday 30th September 2024

    Carrier (ISP) CMS Statesboro - Network Outage

    9/27/24 4:11 EDT - Network Outage - Power verified on Monday 9/30/24 morning - Equipment restarted, local contact is Roger Wentz 912-486-4397 - Damage visible outside the building to the fiber coming in. AT&T Ticket number CS1042334 From AT&T. AN ACCESS SUPPLIER PROBLEM HAS BEEN DETECTED. THE TICKET HAS BEEN REFERRED TO THE ACCESS SUPPLIER FOR FURTHER ANALYSIS. AT&T DIAGNOSE HAS DETECTED THAT THE PINGS BETWEEN THE PROVIDER EDGE AND CUSTOMER EDGE ROUTERS ARE UNSUCCESSFUL. FURTHER TROUBLESHOOTING WILL BE PERFORMED BY AT&T. PLEASE CHECK TICKET FOR ADDITIONAL STATUS. Cradlepoint is being shipped to the site as a temporary solution. CP#12 - Device IMEI 350884581087817

  • Sent a cradlepoint #12 for them to use until AT&T fixes the issue.

    • Tuesday 24th September 2024

    Cisco Email Security Emails to USG.com bouncing back

    We are aware of an issue where emails sent to USG.com are bouncing back. After investigating, we’ve found that USG’s email filtering systems are modifying the emails during their security scans, which changes the email’s source address to their system’s IP (148.163.140.163). This is causing the emails to be blocked.

    We have reached out to USG and offered to work together on resolving the issue. Since we don’t manage their security systems or have access to their environment, we’re dependent on their team to address this on their end.

    We will keep you informed as soon as we have updates. Thank you for your understanding and patience.

  • Monitoring shows no further incidents. Safe to close.

  • Got confirmation that this issue has been resolved by cbarron@usg.com Charles Barron @ USG.