[LCH Announce] At Risk / Urgent Maintenance TBA

[Phillip Baker]

Hello,

On Wednesday last week, sw0.the.core started to display novel errors that
the manufacturer documentation would lead us to believe should not ever
occur on this particular model of switch. As the rate of errors increased, a
replacement switch was been ordered and is expected to arrive at the
facility on Monday.

At present the switch appears to be passing traffic fine, and so we will be
waiting until our ordered part arrives before replacing it; however there is
a possibility that the switch may reboot or fail completely due to the fault
currently occurring. If a failure occurs we will source a switch via
emergency channels and replace it immediately. With this in mind the network
is now considered "At Risk" until the switch is replaced.

Once we have confirmed delivery of the replacement switch and performed
basic burn in testing, we will schedule a maintenance window early next week
for it to be installed and replace the current sw0, and make a further
announcement to the list.

This work will disrupt all services from the AS25098 network, but not all
customers will experience a total loss of service - a number of services are
terminated without passing through sw0 and these will be largely unaffected
by these works. All peers on XchangePoint and Lipex will be shut down during
these works. Any customers who are not experiencing a total loss of service
may experience route changes and possibly increased loss/latency while the
traffic changes path once as the peering is shut down, and once again as it
is brought back online.

Marginally affected services/customers [loss of peering routes]:

UK Shared Hosting customers on cloud0.lchost.co.uk
Services to TFM40 H15 
Transit services to customers connected via Datahop MAN

All other services will experience a total loss of service whilst the switch
is replaced. We will work to minimise disruption as best we can but
unfortunately some loss of service is unavoidable.

A further announcement detailing the maintenance window date and times will
follow in the next couple of days.

We apologise for any inconvenience caused; please be aware that we are
closely monitoring the situation and should it deteriorate unacceptably we
will work to have it amended as soon as possible.

Regards,
Phillip Baker
LCHost Chief Technical Officer
phil at lchost.co.uk