Some systems are experiencing issues

About This Site

This page offers performance, outage and maintenance information for all of GMS services and applications. Check back here any time to view the current status of the services listed below. For additional information or to report a problem, send an email to pchelp@gms.com and include as much information as you can along with your contact info.

Past Incidents

Wednesday 11th September 2024

Carrier (ISP) CMI Tuscaloosa - Network Outage

9/11/24 3:37 EDT - Network Outage - Sam Kelley is the local contact - 256-990-9279, Networking team has reached out to AT&T

  • 9/11/24 17:13 EDT CMI Tuscaloosa - Network restored. Due to ISP IP configuration issue.

    • Comsense: Experiencing an issue with our Cloud Environment.

    9/11/2024 9:39 EST Comsense: Experiencing an issue with our Cloud Environment.

  • 9/11/2024, 11:22 EDT - Comsense: Repaired known issue with Comsense cloud service.

    • Carrier (ISP) OVS Holland - Network Outage

    9/11/24 00:13 EDT - Network Outage - site came back online at 6:35 EDT without any intervention. Appears to be ISP related. I'll have the networking guys look at the logs. Total outage time was 6 hours and 22 minutes

    Sunday 8th September 2024

    CHE Memphis: Network outage.

    9/8/2024 23:38 - CHE Memphis: Network outage.

  • 9/9/2024 09:05 - CHE Memphis: Network restored. Restarted provider equipment.

    • Thursday 5th September 2024

    Carrier (ISP) CBSI - New Castle - Network Outage

    9/5/24 4:14 EDT - Network outage - Checked the YIAB and power was out to the switch. Had him restart the UPS. Local contact Cesil Saldivar - (443) 797-4194 Restarted the Comcast equipment. Restarted the Cienna as well. Damon needed to reload the config on the router. reference number for comcast coax cr159956177 Ticket CR159957679 Comcast sent a tech out and switched out the modem. Back online at 9/5/24 10:17 EDT for a total outage time of 6 hours and 3 minutes

    Wednesday 4th September 2024

    NEG Milford: Network Outage.

    9/4/2024 12:00 NEG Milford: Network Outage.

  • 9/4/2024 14:07 NEG Milford: Network restored. ISP/ATT is in outage CS0996180. Set location on secondary cradle point.

    • Tuesday 3rd September 2024

    WBM Santa Maria: Power Outage

    09/03/24 14:41 EDT WBM Santa Maria - Power outage / PG&E estimates power will be restored around 17:30 EDT

  • 09/03/24 15:46 EDT power restored

    • Power Issue CBSI - Dover - Network Outage

    9/1/24 8:03 EDT - Network Outage, Power outage needed to restart the UPS and things came back up at 8/3/24 7:24 EDT Total outage time 1 day, 23 hours, 21 minutes

    May need to replace the UPS.

    Friday 30th August 2024

    Kamco Manhatten: Network Outage

    8/30/2024 13:35 - Kamco Manhattan: Network Outage. Spectrum maintenance.

  • 8/30/2024 17:12 - Kamco Manhattan: Network Restored. Spectrum maintenance.

    • Thursday 29th August 2024

    Carrier (ISP) Colonial Fayetteville - Network Outage

    8/24/24 12:30 EDT Outage - Restarted networking equipment, ours & ISP's. Appears to be ISP related (Metronet with a Spectrum backup). Local contact is Andy Hinman - 910-929-8132 dispatch scheduled for Fayetteville. currently 12 to 3 ticket number ett860760

    Site back online 8/26/24 13:36 EDT - ISP repaired the line. Total outage time

    Wednesday 28th August 2024

    Carrier (ISP) TMG West Union - Network Outage

    8/28/24 7:17 EDT Network Outage, verified power and restarted our equipment and the ISP's, contact number for Jason is 8646081252. AT&T has been dispatched

    Tuesday 27th August 2024

    Power Issue CMI - Madison/Huntsville - Power Outage, network down

    8/27/24 9:01 EDT Power Outage took the network down. Power restored at 10:06 EDT for a total outage time of one hour and 5 minutes.

    GAT Sarasota CS - Network Outage

    8/27/2024 6:50 EST GAT Sarasota CS - Network Outage

  • 8/27/2024 7:02 EST GAT Sarasota CS - Network restored. ISP outage.

    • Thursday 22nd August 2024

    Power Issue GAT Clearwater - Power Outage

    8-22-24 10:55 EDT: GAT Clearwater - Power Outage. The Clearwater location currently has no power due to a power transmitter going down, ETR is unknown at this time.

  • 8-22-24 20:03 EDT Power restored; all networking functions operational.

    • Tuesday 20th August 2024

    CHP Farmington - Network Outage

    8-20-24 11:35 EDT CHP Farmington - Network Outage. IT teams investigating, will update as more information is known.

  • 8-20-24 14:14 EDT - Networking restored. Amiri remoted into ISR, updated configuration and had me reboot switch.

    • KAM Bronx - Network Outage.

    8/20/2024 03:27 EDT KAM Bronx - Network Outage.

  • 8/20/2024 14:56 EDT KAM Bronx - Network restored: Primary, wrong ip configuration on ISR; Secondary, bill unpaid (service cancelled).

    • Saturday 17th August 2024

    CBS Beltsville - Network Outage

    8/17/2024 18:29 EDT CBS Beltsville - Network Outage

  • 8/20/2024 8:38 EDT CBS Beltsville - Network back online, Comcast circuit hardware issue.

  • 8/19/2024 9:17 EDT CBS Beltsville - Service ticket started with Comcast. Ticket: si068067136.

    • Friday 16th August 2024

    DSI Lincoln - Network and phones down. Spectrum outage in the area due to 2 fiber cuts. Spectrum ticket ECT83458264

    8/16/2024 13:47 EDT - DSI Lincoln - Network and phones down. Spectrum outage in the area due to 2 fiber cuts. Spectrum ticket ECT83458264

  • 8-16-24 20:47 EDT - Spectrum repaired fiber cuts. Networking restored.

    • Thursday 15th August 2024

    Power Issue Commonwealth Materials - Norfolk - Network Outage

    8/15/24 1:35 EDT - Network outage, Tyler restarted the network equipment and they restored at 7:00 EDT for a total outage time of 5 hours and 25 minutes.

    Monday 12th August 2024

    Kamco - AS400 printing down for remote sites

    AS400 printing was resolved around 9:20am EST after tracking down a loose network cable that caused the connectivity between sites to be dropped.

    Friday 9th August 2024

    Power Issue CSB Dover - Accidental Power Outage / Network Outage

    8-9-24 13:12 CSB Dover - Accidental Power Outage / Network Outage. UPS was turned off. Restored USP and all the gear came back online. at 14:40 EDT.

    GMS Wide - Users unable to Scan to Email

    8-9-24 09:05 EDT GMS Wide - Users unable to Scan to Email. Network issue in the datacenter, networking team is currently investigating. Will update Status page when more information is known.

  • 8-9-24 10:59 EDT Full restart of the FTDs resolved issue with SMTP Servers. Scan to Email services have restored.

    • Wednesday 7th August 2024

    GTS Billings - Network Outage

    8/6/2024 22:00 - GTS Billings lost power after an overnight wind storm. Investigating.

  • 8-9-24 17:45 Complete power restored. Networking functioning properly.

  • 8-7-24 10:00 EDT GTS Billings update, only half the building has power after checking the breakers. They have an electrician coming out to check it out. 8/7/24 10:00 Per Bill Barnekow

    • Tuesday 6th August 2024

    Power Issue GTS Bend Power Outage

    8-6-24 10:29 EDT GTS Bend Power Outage / Network Outage. Will update Status when power restores.

  • 8-6-24 13:58 EDT Power and networking restored.

    • Canada Beanfield Datacenter availability Degraded

    8-6-24 09:26 EDT Nuvollo's VMWare platform is experiencing a SAN issue. Currently 19 servers are affected. Nuvollo team is working with HPE to restore services.

  • 8-8-24 22:30 EDT We experienced a crash of our new HPE Alletra all flash SAN yesterday. This caused the hosts to lose connectivity with this SAN (1 of 4 in that DC, but this SAN has many VMs for GMS Canada).

    We have received the HPE analysis, and will proceed with an emergency maintenance tonight as recommended by HPE.

    We certainly strive for 100% availability, but this is IT and we sometimes have unpredictable issues.

    The new Dynamic Business Central architecture is planned with SQL HA across 2 datacenters and load balancing across the 2 DCs for the front end too, so this event would not have affected anyone with the coming architecture.

    Notes from HPE:

    We have an update from the Engineering team on this. This is matching the footprints of a known issue seen on the current OS version This is iSCSI specific and happened during normal use of the array. A fix is available in the next OS version 10.4.0. In order to avoid this issue in future request you to please have the system upgraded to 10.4.0.

    • Monday 5th August 2024

    PHD St. Charles - Network Outage

    8-5-24 18:35 EDT PHD St. Charles - Network Outage. Root cause unknown as site is closed for the day. Will update in the AM when more information is known.

  • 8-5-24 20:36 Network restored. Root cause is unknown. Verified with yard all networking functioning properly.

    • COL Acoustical - Network Outage

    8-5-24 08:05 EDT COL Acoustical - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-5-24 08:21 EDT - Network Restored. Power cycle of equipment restored service.

    • CHP Durango - Network Outage

    8-5-24 06:56 EDT CHP Durango - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-7-24 11:29 EDT - Network Restored - Lumen to replaced defective ISP equipment on site.

  • 8-5-24 06:56 EDT CHP Durango - Ticket escalated with Lumen to replace defective ISP equipment on site.

    • Carrier (ISP) PMW Steamboat Springs - Network Outage

    8-5-24 06:56 EDT PWM Steamboat Springs - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-5-24 09:43 EDT Lumen corrected issue; Network has been restored.

    • Carrier (ISP) PMW Silverthorne - Network Outage

    8-5-24 06:56 EDT PMW Silverthorne - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-5-24 12:46 EDT Lumen corrected issue; Network has been restored.

    • Carrier (ISP) PWM Casper - Network Outage

    8-5-24 06:56 EDT PWM Casper - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-5-24 09:43 EDT Lumen corrected issue; Network has been restored.

    • Carrier (ISP) PIO Berthoud Network Outage

    8-5-24 06:56 EDT PWE Berthoud Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-5-24 09:43 EDT Lumen corrected issue, Network has been restored.

    • Carrier (ISP) PWE Acoustical Network Outage

    8-5-24 06:56 EDT PWE Acoustical - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-6-24 08:50 Yard purchased console cable; Damon remoted into ISR to resolve networking. Network has been restored and functioning properly.

  • 8-5-24 13:47 - Per Lumen issue is related to HVAC issues at the Denver datacenter for the outage.

    • Carrier (ISP) PWE Grand Junction - Network Outage

    8-5-24 06:56 EDT PWE Grand Junction - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-5-24 09:41 EDT temp restore from ISP (Lumen). Went back down at 10:10 EDT and restored at 10:49 EDT.

    • OHV Trafford - Network Outage

    8-3-24 17:22 EDT OHV Trafford - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-6-24 09:22 EDT - Replacement ISR received at yard (current one had bad power supply). Damon remoted in, finalized configuration. Network has been restored.

    • GTS Missoula - Network Outage

    8-5-24 06:56 EDT GTS Missoula - Network Outage. IT Teams are investigating and will update status as more information is known. Temp restore from 10:47 - 11:10. Network outage again 11:10 - 11:49

  • 8-5-24 11:10 EDT Secondary Outage. Network Restored at 11:49 EDT.

  • 8-5-24 11:49 EDT EDT Lumen corrected issue; Network has been restored. Temp restore from 10:47 - 11:10. Network outage again 11:10 - 11:49

    • GTS Billings Network Outage

    8-5-24 06:56 EDT GTS Billings - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-5-24 09:43 EDT Lumen corrected issue; Network has been restored.

    • CBSI New Castle - Network Outage

    8-2-24 21:12 EDT CBSI New Castle - Network Outage. IT Teams are investigating and will update status as more information is known.

  • 8-5-24 14:25 EDT New Castle a comcast port broke and we had to move ports to restore service. Networking functioning properly.

    • Friday 2nd August 2024

    Power Issue LSM La Feria - Power Outage

    8-2-24 13:01 EDT LSM La Feria - Power Outage. Local power company is aware, no ETR at this time.

  • 8-2-24 17:54 EDT Power and Network functions have been restored.

    • Thursday 1st August 2024

    Postmark Postmark Email issues for CBSI

    Postmark outage for CBSI/Kempf users. This incident has been identified and Postmark support has been engaged.

  • Postmark is resuming the service

    • GAT Sarasota - Network Outage

    8-1-24 08:00 EDT GAT Sarasota - Network Outage. Network equipment restart corrected the issue. Network restored on 8-1-24 at 08:16 EDT

    Friday 26th July 2024

    VIP Nashville - Network Outage

    7-25-24 22:07 EDT VIP Nashville - Network Outage, cause unknown. Network restored on 7-25-24 at 22:28 EDT No users affected as outage occurred during off hours.

    Thursday 25th July 2024

    Postmark Postmark Service Issue: Outbound sending. Outbound messages are delayed.

    We are currently investigating reports of sending delays. Messages are being accepted as normal, but for some messages there is a slight delay after the message is Processed and before it's actually sent to the recipient. We are investigating and will update shortly here. Avatar for Postmark Support Postmark Support 13:32 EDT - 25 July 2024 Began at: 13:32 EDT - 25 July 2024

  • Resolved There have been no other delays since our last update and so the incident has been resolved. Thank you for your patience! Avatar for Postmark Support Postmark Support 17:13 EDT - 25 July 2024

  • Recovering We have identified and mitigated the issue causing sending delays for some messages. Messages are now sending at normal Postmark speeds. We'll continue to monitor to ensure no further delays occur and will update the status shortly. Avatar for Postmark Support Postmark Support 14:13 EDT - 25 July 2024

    • Wednesday 24th July 2024

    Carrier (ISP) TAM Duluth - Network Outage

    7-24-24 08:05 EDT TAM Duluth - Network Outage. IT Teams working with provider, will update Status when more information is known.

  • 7-24-24 11:41 EDT ISP corrected issue, verified with yard their network functions have been restored.

  • 7-24-24 10:54 EDT ISP Consolidated Ticket 8841281

    • Carrier (ISP) CBSI/OVS - Stow - Network Outage

    07-24-24 00:29 CDT - Network outage - Power was verified and equipment restarted. Local contact Nick Svetahor 330-703-1989, Site came back online after restarting equipment on 07-24-24 05:19 CDT for a total outage time of 4 hours and 50 minutes.

    Tuesday 23rd July 2024

    UBM Greenfield - Network Outage

    7-23-24 07:42 EDT UBM Greenfield - Network Outage. IT Teams investigating, will update Status when more info is known.

  • 7-23-24 09:51 Power Restored, Networking restored.

    • Friday 19th July 2024

    CMI Ocean Springs - Network Outage

    7-18-24 20:02 EDT CMI Ocean Springs - Network Outage - 7-19-24 08:09 Power cycle of switches, ISR, ISP modem brought network back up.

    Carrier (ISP) JBM El Cajon - Network Outage

    7-19-24 00:17 EDT JBM El Cajon - Network Outage (1177 North Johnson) IT teams are investigating and will update when more information is known.

  • 7-22-24 11:55 EDT - Network Restored. El Cajon was in soft disconnect. Working with ACC to restore circuit resolved issue.

    • Carrier (ISP) Capitol - Marietta - HQ 07/19/24 12:01 EDT - Site lost network and phones. Investigating

    07/19/24 12:01 EDT - Capitol - Marietta - HQ Site lost network and phones. Investigating

  • Capitol - Marietta - HQ 07/19/24 13:10 EDT - Network restored after power cycling ISP network equipment.

    • Cloudstrike Security Event - Servers and workstations affected. Users may experience BSODs on Windows machines.

    The IT team has been notified via our internal monitoring systems that we have been impacted by a major worldwide event involving Crowdstrike security tools. They rolled out updates overnight that have cause Windows 10 servers and workstations to crash. The IT team is working through the servers to restore applications. At this time, you may experience performance issues with WebUI and Printing as we restore servers in those applications.

    If you have a workstation that will not boot or has a blue screen error on the screen, please contact PC help via pchelp@gms.com or calling 877-472-5788. They will assist you in restoring your workstation.

  • All services/servers and computers have been fixed as per Vendor Instructions

  • The IT team has restored all servers that were impacted by this morning’s Crowdstrike event. For Tannerbolt employees, the Epicor P21 company was also impacted by this event and they are still restoring their services which we use. We will send further emails on that status separately.

    The regional support team is working through the tickets around workstations to get them back online. If you have a workstation that is presenting a blue screen of death, please log a ticket with PCHelp@gms.com from another device or call 877-472-5788. If you log a ticket on PCHelp@gms.com, please make sure you put your name, a direct phone number to reach you, and your service tag in the ticket. This will help expedite the recovery. To find the service tag, it is normally on a sticker located underneath or on the back of your workstation.

  • If your PC is Blue Screening please start with a reboot. If this does not fix it, please contact PC Help at 877-472-5788

    • Thursday 18th July 2024

    COL Hickory - Network Outage

    7-18-24 15:25 EDT COL Hickory - Network Outage. IT Teams investigating and will update Status as more information is found.

  • 7-19-24 13:50 EDT - Lumen gear power cycled and Amiri worked with ISR - network restored.

  • as of 0643hrs 6/19/24 : No connectivity (starting Jul 18 15:25)

    • Wednesday 17th July 2024

    Tamarack New Richmond Network outage.

    7/17/2024 01:07 EDT Tamarack New Richmond Network outage.

  • 7/17/2024 06:57 EDT Tamarack New Richmond Network restored. ISP issue.

    • Monday 15th July 2024

    GMS Widespread - FEDEX currently unavailable

    7-15-24 12:35 EDT GMS Widespread - FEDEX currently unavailable FEDEX is currently working on issue

  • 7-15-24 24:00 EDT - Access to Fedex.com has been restored.