Apology from our hosts

4 posts / 0 new
Last post
GeoffB's picture
Apology from our hosts
Following Monday nights outage - an apology has been sent from our hosts , sorry it's technical speak... but for all you whizz kids out there here is the detail :- Between 9 and 11:30 am and between 2 and 5pm EST, a software error caused a network misconfiguration. As a consequence, 1&1 Dedicated and Virtual Servers from our customers were not available on the Internet. The problem affected only customers using a 1&1 dedicated, cloud or virtual server. Our network configuration files are stored in a central Sybase database. For dedicated servers in particular, network data frequently changes (e.g. when customers change their firewall settings). These changes need to be broadcasted to all network components (routers, switches). About one month ago, we installed middleware which regularly fetches the settings from the database and centrally distributes it to the network components. Until yesterday, this system worked seamlessly. Yesterday, however, our database server was under extraordinarily high load and hence had longer reply times than usual. Due to an error in the middleware, a Sybase::DBI Connector, the software interpreted the long waiting time as an "empty" answer and as a consequence overwrote the configuration files of routers and switches in our server segments with empty data. When the source of the issue had been identified, we first had to manually restore the correct configuration files. At the same time we temporarily disabled the automated database queries by the middleware. During this time, our customers could not make any changes to their firewall settings or install new IP addresses. In order to avoid similar outages in the future, we have implemented additional check routines in our network which prevent the error from occurring again. Changes to network settings through the 1&1 Control Panel have been enabled again, but will take approximately 15 minutes to become effective due to the check mechanisms. Additionally, we closely collaborate with the middleware manufacturer to eliminate the software errors as soon as possible. We apologize for any inconvenience. As customer satisfaction is our fore most concern, we strive to find a satisfactory solution.

Is this a good topic?: 

No votes yet
barebob (not verified)
Anonymous's picture
nice change ......
Makes a nice change to actually get an apology out of anyone these days, although I think I'll leave the "techie" stuff to you William ..
Denny's picture
Put enough techie speak in there and everyone thinks - oh, thats ok then. Good to get the apology (if thats what it is!)
Andrew's picture
Well sounds like the software broke down and they had to call out green flag, they also blamed someone else classic case of passing the buck but as Brian and Susan said we lived through it. After all it is man made, at least they made contact got to give them that.