Today, we want to explain what happened lately with our network in Amsterdam.
Between Friday the 15th and Wednesday 20th, a large portion of our users in Amsterdam suffered from network troubles. We know that you placed your trust in our services, and an outage like this is unacceptable. We would like to apologize and assume full responsibility for the situation. We'd like to share all of the details about this event.
Last Friday, we started to receive notifications from our customers related to network performance issues.
Our engineering team started to investigate right away but nothing stand out. The uplinks of the Amsterdam platform were nearly full but we were not able to reproduce the dramatic performance issue reported.
The same day, the software powering our NAT layer was upgraded to support a new IP range as our current IP pool was completely empty.
Following the upgrade, two NAT server were not restarted, our initial assumption was that this was strengthening the bandwidth shortage. We restarted these servers to increase the NAT capacity and continued to monitor the situation.
Bandwidth usage was still high but links were not completly full.