On Wednesday November 16th at 3:40pm EST NetDocuments experienced an incident in the US datacenters that caused the service to be intermittently unavailable. Services in Europe and Australia were not affected.
The root cause of the issue was determined to be self-inflicted. During the routine deployment of a script, a syntax error in the script itself went undetected and caused our normal logging to become skewed and grow exponentially. This in turn caused a cascading effect that impacted our active web servers and continually stressed newly spawned replacement web servers, resulting in an extended period of connectivity and access issues across the Service.
We have robust procedures in place around this type of routine deployment. However, this situation exposed a need for an additional step to be taken that will avoid such a disruption in the future. This additional step has now been added to our formal change control process for this type of deployment.
We fully understand that NetDocuments is a mission-critical service for our customers and apologize for yesterday’s disruption. We remain as committed as ever to providing our customers with a highly reliable and predictable service. It’s important to stress that yesterday’s issue was not a result of any issues related to infrastructure (architecture, software, hardware, network, or telecom) or scalability (data or user growth).
Please contact us if you have further questions about this issue or any additional topics.