Following Monday nights outage - an apology has been sent from our hosts , sorry it's technical speak... but for all you whizz kids out there here is the detail :- Between 9 and 11:30 am and between 2 and 5pm EST, a software error caused a network misconfiguration. As a consequence, 1&1 Dedicated and Virtual Servers from our customers were not available on the Internet. The problem affected only customers using a 1&1 dedicated, cloud or virtual server. Our network configuration files are stored in a central Sybase database. For dedicated servers in particular, network data frequently changes (e.g. when customers change their firewall settings). These changes need to be broadcasted to all network components (routers, switches). About one month ago, we installed middleware which regularly fetches the settings from the database and centrally distributes it to the network components. Until yesterday, this system worked seamlessly. Yesterday, however, our database server was under extraordinarily high load and hence had longer reply times than usual. Due to an error in the middleware, a Sybase::DBI Connector, the software interpreted the long waiting time as an "empty" answer and as a consequence overwrote the configuration files of routers and switches in our server segments with empty data. When the source of the issue had been identified, we first had to manually restore the correct configuration files. At the same time we temporarily disabled the automated database queries by the middleware. During this time, our customers could not make any changes to their firewall settings or install new IP addresses. In order to avoid similar outages in the future, we have implemented additional check routines in our network which prevent the error from occurring again. Changes to network settings through the 1&1 Control Panel have been enabled again, but will take approximately 15 minutes to become effective due to the check mechanisms. Additionally, we closely collaborate with the middleware manufacturer to eliminate the software errors as soon as possible. We apologize for any inconvenience. As customer satisfaction is our fore most concern, we strive to find a satisfactory solution.
Wed, 28/09/2011 - 9:25pm#1
Apology from our hosts