BHS Datacenter Incident: November 2

On Monday November 2, 2015, the BHS datacenter suffered a new network problem impacting all BHS Datacenter customers. This incident was similar to 5 months ago, when 3 pairs of our optical fibers were cut between Beauharnois and Montreal on the North route.

Two years ago, we began adding redundancy on the 103 KMs of fiber on the South loop. Since then, there have been long delays concerning this project, bringing the time of completion to 24 months instead of the planned 12. Work was finished this week and only in response to the urgency of the situation. We forced the installation of the new South fiber optic line, thereby reducing the duration of downtime of the public network from 10 hours and 51 minutes to 4 hours and 55 minutes.

Now, the datacenter is connected in total redundancy by the North loop and the South loop. We have 5 directions: 2 towards New York, 1 towards Chicago and 2 towards Montreal.

Three years after starting the project, we are continuing the work on improving the 2nd route towards New York where all that remains is to solder the optic fiber at the Canada/USA border. Negotiations are underway to connect Toronto and Detroit, MI.

Here is what our dark fiber and DWDM network connecting BHS looks like:

We own:
— 4x10 Gbps between BHS and Montreal via the North loop
— 4x10 Gbps between BHS and Montreal via the South loop
— 6x100 Gbps towards New York/Newark via the South loop
We rent:
— 10x10 Gbps towards Chicago via South loop
— 20x10 Gbps towards Newark via the North loop
— 4x10 Gbps towards London (through Canada) via South loop
— 4x10 Gbps towards Paris (through USA) via the North loop



Within 3 to 4 weeks, we will finish the work on our own 2nd route towards New York/Newark and we are going to activate an optical failover of 6x100 Gbps through the following 2 routes: BHS/NYC-NWK. This will allow us to terminate the rental of the 20x10 Gbps through Newark.



We are going to increase the capacity of Montreal/Chicago to 2x100 Gbps then 4x100 Gbps while waiting for our own route to be finished between BHS and Chicago by Toronto/Detroit. We cannot yet provide an ETA, knowing that we lack 800 KMs between Toronto and Detroit, MI. We already have in place BHS towards Toronto and Detroit, MI towards Chicago, IL.



At the same time, we are working on the network in the USA between New York and Newark (NY) then Chicago (IL) and Ashburn (VA). This new USA network is planned to be deployed in Q2 2016 and will not only provide redundancy from BHS towards Chicago through New York but also allow for the connection of a datacenter in the USA.

We sincerely apologize for the latest BHS downtime. We underestimated the time and the complexity of the fiber optic network construction projects in Canada, thus creating an unreasonable delay in securing each kilometer connecting the BHS Datacenter. Even though today it's done and we have 100% redundancy, we are really sad to announce this under such circumstance.

Within the next 10 days, we will offer compensation which conforms to our SLA for the month of November.

The exact times concerning the incident:
— down at 11:53 AM GMT-5 (5:53 PM GMT+1)
— at 4:48 PM GMT-5 (10:48 PM GMT+1), traffic to the BHS datacenter from the outside returned to normal thanks to the optical fiber by the South
— in contrast, the links between BHS and the other OVH datacenters and the VAC (Anti-DDoS) were affected until the complete repair of the North fiber. Traffic was able to resume at 10:44PM (04h44 GMT+1).

Best Regards,
Octave