Details of the challenge and continued remediation are described below:
On September 9th, Concur experienced a recurrence of performance challenges due to unexpectedly exceeding a threshold in an internal router. At the initial onset of the problem, indications were that adjusting the rule set would resolve the issue. This approach was utilized after the initial degradation of performance on September 3rd. Adjustments were implemented and the performance issues were temporarily alleviated. Due to volume increases that were anticipated in the days ahead, Concur continued working with the firewall vendor to determine additional steps that could be taken to avoid performance challenges.
On September 9th as the load increased, the network latency also increased. This latency indicated that the threshold was exceeded again. Additional corrective actions were taken into the early part of September 10th. Concur still believes that the internal firewall is central to the issue and the engineering team has been focused on shifting load to another segment of the network. In addition, we have prepared additional measures and implemented additional monitoring in order to more quickly identify and resolve any issue that occurs in this layer of our system. Concur will be watching closely as load increases throughout the day. The peak volume is typically observed starting at 4 AM PT and through to 9 AM PT (12:00 GMT to 17:00 GMT).
Concur regrets the inconvenience and impact of the slowdown and outage. Please be assured that Concur is diligently working non-stop to make sure expected performance can be experienced under peak load.
Thank you,
Concur