From phil at lchost.co.uk Mon Feb 4 23:23:52 2008 From: phil at lchost.co.uk (Phillip Baker) Date: Mon, 4 Feb 2008 23:23:52 +0000 Subject: [LCH Announce] cloud0.lchost.co.uk EMERGENCY MAINTENANCE Message-ID: <28ea2461dcf2ae4bd2e38c54d4a06a4b@localhost> Hi, Those of you hosted on cloud0.lchost.co.uk will no doubt have noticed that it crashed earlier today, causing an outage for all services on this server. Since the server came back up I have been attempting to troubleshoot a) the cause of the crash b) considerably slower than expected mail processing since rebooting Despite attempts at nailing down b), we are still seeing considerably slower than expected mail processing, leading to many MUAs (mail clients, such as outlook, outlook express, thunderbird, et al) to complain that the server is not responding. I believe that both the crash and the poor performance are due to issues with the current driver and kernel versions in use on the server which were built manually with currently available code before Debian (the distribution of linux we use) natively supported the RAID controller in use on this server. The latest version of Debian includes built in support for this controller as part of a considerably newer kernel. It is with this in mind that I am scheduling an emergency rebuild of this server this evening in an attempt to nail down these issues entirely. This will unfotunately involve the server being taken completely offline so that it can be backed up safely prior to being rebuilt. The rebuild will start at approx 23:40, and is expected to take no more than 5 hours. We apologise for any inconvenience this may cause; but we cannot permit the current situation with potential instability and poor perfomance to continue. In the event that initial testing reveals the newer software release will not function correctly with the system, we will instead look to seperate the mail function out to another backend server over the next 24 hours. -- Regards, Phillip Baker LCHost Chief Technical Officer 0870 068 HOST ext 50 From phil at lchost.co.uk Mon Feb 4 23:25:46 2008 From: phil at lchost.co.uk (Phillip Baker) Date: Mon, 04 Feb 2008 23:25:46 +0000 Subject: [LCH Announce] cloud0.lchost.co.uk EMERGENCY MAINTENANCE Message-ID: <47A79EFA.2070707@lchost.co.uk> Hi, Those of you hosted on cloud0.lchost.co.uk will no doubt have noticed that it crashed earlier today, causing an outage for all services on this server. Since the server came back up I have been attempting to troubleshoot a) the cause of the crash b) considerably slower than expected mail processing since rebooting Despite attempts at nailing down b), we are still seeing considerably slower than expected mail processing, leading to many MUAs (mail clients, such as outlook, outlook express, thunderbird, et al) to complain that the server is not responding. I believe that both the crash and the poor performance are due to issues with the current driver and kernel versions in use on the server which were built manually with currently available code before Debian (the distribution of linux we use) natively supported the RAID controller in use on this server. The latest version of Debian includes built in support for this controller as part of a considerably newer kernel. It is with this in mind that I am scheduling an emergency rebuild of this server this evening in an attempt to nail down these issues entirely. This will unfotunately involve the server being taken completely offline so that it can be backed up safely prior to being rebuilt. The rebuild will start at approx 23:40, and is expected to take no more than 5 hours. We apologise for any inconvenience this may cause; but we cannot permit the current situation with potential instability and poor perfomance to continue. In the event that initial testing reveals the newer software release will not function correctly with the system, we will instead look to seperate the mail function out to another backend server over the next 24 hours. -- Regards, Phillip Baker LCHost Chief Technical Officer 0870 068 HOST ext 50 From phil at lchost.co.uk Mon Feb 4 23:27:25 2008 From: phil at lchost.co.uk (Phillip Baker) Date: Mon, 04 Feb 2008 23:27:25 +0000 Subject: [LCH Announce] cloud0.lchost.co.uk EMERGENCY MAINTENANCE Message-ID: <47A79F5D.8040900@lchost.co.uk> Hi, Those of you hosted on cloud0.lchost.co.uk will no doubt have noticed that it crashed earlier today, causing an outage for all services on this server. Since the server came back up I have been attempting to troubleshoot a) the cause of the crash b) considerably slower than expected mail processing since rebooting Despite attempts at nailing down b), we are still seeing considerably slower than expected mail processing, leading to many MUAs (mail clients, such as outlook, outlook express, thunderbird, et al) to complain that the server is not responding. I believe that both the crash and the poor performance are due to issues with the current driver and kernel versions in use on the server which were built manually with currently available code before Debian (the distribution of linux we use) natively supported the RAID controller in use on this server. The latest version of Debian includes built in support for this controller as part of a considerably newer kernel. It is with this in mind that I am scheduling an emergency rebuild of this server this evening in an attempt to nail down these issues entirely. This will unfotunately involve the server being taken completely offline so that it can be backed up safely prior to being rebuilt. The rebuild will start at approx 23:40, and is expected to take no more than 5 hours. We apologise for any inconvenience this may cause; but we cannot permit the current situation with potential instability and poor perfomance to continue. In the event that initial testing reveals the newer software release will not function correctly with the system, we will instead look to seperate the mail function out to another backend server over the next 24 hours. -- Regards, Phillip Baker LCHost Chief Technical Officer 0870 068 HOST ext 50 From phil at lchost.co.uk Mon Feb 4 23:28:57 2008 From: phil at lchost.co.uk (Phillip Baker) Date: Mon, 04 Feb 2008 23:28:57 +0000 Subject: [LCH Announce] cloud0.lchost.co.uk EMERGENCY MAINTENANCE Message-ID: <47A79FB9.30709@lchost.co.uk> Hi, Those of you hosted on cloud0.lchost.co.uk will no doubt have noticed that it crashed earlier today, causing an outage for all services on this server. Since the server came back up I have been attempting to troubleshoot a) the cause of the crash b) considerably slower than expected mail processing since rebooting Despite attempts at nailing down b), we are still seeing considerably slower than expected mail processing, leading to many MUAs (mail clients, such as outlook, outlook express, thunderbird, et al) to complain that the server is not responding. I believe that both the crash and the poor performance are due to issues with the current driver and kernel versions in use on the server which were built manually with currently available code before Debian (the distribution of linux we use) natively supported the RAID controller in use on this server. The latest version of Debian includes built in support for this controller as part of a considerably newer kernel. It is with this in mind that I am scheduling an emergency rebuild of this server this evening in an attempt to nail down these issues entirely. This will unfotunately involve the server being taken completely offline so that it can be backed up safely prior to being rebuilt. The rebuild will start at approx 23:40, and is expected to take no more than 5 hours. We apologise for any inconvenience this may cause; but we cannot permit the current situation with potential instability and poor perfomance to continue. In the event that initial testing reveals the newer software release will not function correctly with the system, we will instead look to seperate the mail function out to another backend server over the next 24 hours. -- Regards, Phillip Baker LCHost Chief Technical Officer 0870 068 HOST ext 50 From phil at lchost.co.uk Tue Feb 5 04:43:43 2008 From: phil at lchost.co.uk (Phillip Baker) Date: Tue, 05 Feb 2008 04:43:43 +0000 Subject: [LCH Announce] COMPLETED - cloud0.lchost.co.uk EMERGENCY MAINTENANCE Message-ID: <47A7E97F.3090105@lchost.co.uk> Hello, Further to the previous mails (apologies for the duplicates - even I had difficulties with the mailserver playing up) this work is now complete. Cloud0 has been completely rebuilt, and with the exception of Mailman and Webalizer statistics is completely up and running. As Mailman functioning only affects one customer and Webalizer statistics can be resumed at any point, these will be completed during the day. Restored services: HTTP/HTTPS MySQL SMTP [now *much* improved speedwise] POP/POP-SSL IMAP/IMAP-SSL FTP/FTP-TLS SSH The entire system is now running on Apache2.2 and a single PHP5.2 installation (debian maintained version) running under suphp rather than the old hacked-together methodology we were constrained to before - I've gone and tested a number of customer sites and things appear to be working as expected for most - some particularly broken PHP code may not work under PHP5.2 - this will need updating. I've also installed all of the commonly used PHP5 modules, so if you had gone to the trouble to make your own PHP interpreter under the old setup you should hopefully find that your code still works. Those of you using perl scripts may find that they no longer appear to function - please check your permissions - the scripts I've found that don't work have inadequate permissions set, which while they may have functioned under the old configuration no longer will do so. Do be aware that it's not just the script that's being called that needs correct permissions - directories and files that are relied upon by the script will also need checking. If in doubt, 755 will always work for perl scripts and the directories they are contained within, though you should consider trying more restrictive permissions on any perl scripts that contain sensitive data. If you notice something is awry or you believe you are missing data/services please let us know - support at lchost.co.uk . Regards, Phillip Baker LCHost Chief Technical Officer phil at lchost.co.uk From phil at lchost.co.uk Tue Feb 5 05:29:56 2008 From: phil at lchost.co.uk (Phillip Baker) Date: Tue, 05 Feb 2008 05:29:56 +0000 Subject: [LCH Announce] cloud0 SSH Service Message-ID: <47A7F454.3000701@lchost.co.uk> Hi, For those of you with SSH access to your account on cloud0, please note that we have moved the SSH daemon off the default port of 22 up to 222 in an attempt to eliminate or at least reduce the volume of SSH bruteforce attempts that the server has to handle. This will be reflected in other servers that offer customer-facing SSH over the coming weeks too; seperate announcements will go out when they are updated. You will need to change your bookmarks/preferences/options as appropriate to connect to port 222 instead. Regards, Phillip Baker LCHost Chief Technical Officer phil at lchost.co.uk From phil at lchost.co.uk Tue Feb 12 00:41:16 2008 From: phil at lchost.co.uk (Phillip Baker) Date: Tue, 12 Feb 2008 0:41:16 +0000 Subject: [LCH Announce] EMERGENCY MAINTENANCE - Budget dedicated servers 1-20 Message-ID: <143a8c0213675261028b819f08afd018@localhost> Hello, During routine maintenance checks the power distribution to the budget dedicated servers 1-20 has been found to be in a pre-failure condition. With this is mind the equipment power feed will need to be redone; due to access issues this will involve all of the servers being deracked in order to facilitate this. This unfortunately means that there may be downtimes of up to two hours per server whilst the works take place. Please accept our apologies for any inconvenience this outage may cause, but we will always seek to replace equipment in a controlled manner rather than permit it to fail with potential knock-on consequences. This work will commence in the next 20 minutes - an all clear will be sent upon completion. -- Regards, Phillip Baker LCHost Chief Technical Officer 0870 068 HOST ext 50 From phil at lchost.co.uk Tue Feb 12 10:08:59 2008 From: phil at lchost.co.uk (Phillip Baker) Date: Tue, 12 Feb 2008 10:08:59 +0000 Subject: [LCH Announce] All Clear: Budget dedicated servers 1-20 [3:20am] Message-ID: Hi, This work was completed successfully at 3:20am. Some servers experienced slightly more than the 2 hours downtime we were aiming for - apologies for this. If you are experiencing any difficulties in reaching your server, please contact us. Apologies again for any inconvenience caused by this necessary maintenance. -- Regards, Phillip Baker LCHost Chief Technical Officer 0870 068 HOST ext 50