From phil at lchost.co.uk Sun Aug 28 03:18:11 2005 From: phil at lchost.co.uk (Phillip Baker) Date: Sun Aug 28 03:18:17 2005 Subject: [LCH Announce] Degradation of network service 28/08/2005 00:40 BST onwards Message-ID: <20050828021811.GA7541@maelstrom.lchost.co.uk> Hello, Due to the total failure of the equipment we connect to on KPN's London routing platform at around 00:40 BST this morning we found ourselves without full IP connectivity. Initial attempts to load backup full table transits failed when it was discovered that the providers we had alternative full IP routing sessions configured for had disabled their side without alerting us. We have since made temporary changes to our network infrastructure to restore full IP connectivity by making emergency negotiations with an existing partial transit provider to support the peculiar setup our network presently requires. We apologise profusely for this unfortunate series of problems which has resulted in quite a long period of degraded and very brief periods of total loss in service. We believe that the network should now be globally visible and customer equipment should have no difficulties in reaching any hosts. Customers may experience different (and hence possibly suboptimal) routing and increased latency as a result of these emergency changes. Normal service will be resumed as soon as KPN make reparations to their network infrastructure. We find that our hand is being forced into making rash decisions about upgrading our network infrastructure and while we do not relish the thought of making snap decisions about the network we may be forced to upgrade our existing Cisco 7200 series routers rather than replace them altogether with a different architecture in 6 to 12 months which was our current plan. We will keep you posted on any developments and resulting network maintenance periods that may be required in order to instigate these changes. A further update will be sent once the network is back to it's normal operating state. Regards, Phillip Baker LCHost Chief Technical Officer phil@lchost.co.uk From phil at lchost.co.uk Sun Aug 28 03:21:42 2005 From: phil at lchost.co.uk (Phillip Baker) Date: Sun Aug 28 03:21:45 2005 Subject: [LCH Announce] Clarification Message-ID: <20050828022142.GB7541@maelstrom.lchost.co.uk> Further to my last mail it has become apparent that my wording was not expressly clear on the subject of ownership of (and hence responsibility for) the failed equipment. The equipment which has failed belongs to KPN, not ourselves. Sorry for any confusion. Regards, Phillip Baker LCHost Chief Technical Officer phil@lchost.co.uk From phil at lchost.co.uk Sun Aug 28 13:41:15 2005 From: phil at lchost.co.uk (Phillip Baker) Date: Sun Aug 28 13:41:24 2005 Subject: [LCH Announce] [RESOLVED] KPN Transit Message-ID: <20050828124115.GA3471@maelstrom.lchost.co.uk> Hello, While KPN have been very quiet about it it appears that the issue with their equipment in Telehouse East has been resolved. We have reconfigured our network and we are now routing traffic normally. If you continue to experience poor network performance, please let me know and we will investigate. We apologise on their behalf for any inconvenience caused by this outage. Regards, Phillip Baker LCHost Chief Technical Officer phil@lchost.co.uk From phil at lchost.co.uk Mon Aug 29 16:23:01 2005 From: phil at lchost.co.uk (Phillip Baker) Date: Mon Aug 29 16:23:12 2005 Subject: [LCH Announce] cloud0.lchost.co.uk outage 9:44am-12:20pm 29/aug/2005 Message-ID: <20050829152301.GA9042@maelstrom.lchost.co.uk> Hi, We are having some operational difficulties with cloud0 which culminated in it crashing this morning at 9:44am. Unfortunately I was away from my desk and our text alerting system is presently disabled due to a systems failure at our supplier (It's been a bad weekend on the whole, really). The outage then went largely unnoticed until 11am when reparations were made to bring the server back online. We apologise for the outage, and moreso for the length of time for which it lasted. We are chasing our text messaging platform suppliers on a daily basis trying to get the situation with our alerts system resolved, and with respect to cloud0 we are ordering new webservers with mirrored RAID next week as part of our ongoing infrastructure upgrades and we hope to migrate cloud0 sites and users across to one of these boxes in the next month or so - in the meantime we will keep a very close eye on cloud0 and keep you all alerted to any further issues as and when they develop. Regards, Phillip Baker LCHost Chief Technical Officer phil@lchost.co.uk From phil at lchost.co.uk Tue Aug 30 16:29:33 2005 From: phil at lchost.co.uk (Phillip Baker) Date: Tue Aug 30 16:24:12 2005 Subject: [LCH Announce] KPN failure (again) Message-ID: Hello, KPN's London router has once again disappeared into the ether. They assure me that they are working on the issue and that it is some sort of memory issue on their equipment causing it to fall over. We have put into place our emergency countermeasures again and we should be servicing a full table in both directions now. We will keep you posted on the situation as it develops, and can only pass on the really rather profuse apologies from their NOC about the situation. Regards, Phillip Baker LCHost Chief Technical Officer phil@lchost.co.uk 0870 068 HOST ext 50 From phil at lchost.co.uk Tue Aug 30 17:40:07 2005 From: phil at lchost.co.uk (Phillip Baker) Date: Tue Aug 30 17:34:40 2005 Subject: [LCH Announce] [RESOLVED] KPN failure (again) Message-ID: Hello, Our KPN transit has now been up and stable for 45 minutes or so. We will continue to watch their network and equipment with a suspicious eye for a while until we're sure it's going to stay that way, but it appears to be functioning fine at the moment. Regards, Phillip Baker LCHost Chief Technical Officer phil@lchost.co.uk 0870 068 HOST ext 50