This morning we suffered a severe outage that directly affected six production servers and indirectly several more.  The symptoms included profile problems and access to files. While resolving the issue, we were forced to reboot a number of servers interrupting SQL Database, Outlook, and logons.  

The problem initially was related to logons, and steps were taken to correct that.  In the process one of our critical storage systems became highly unstable. This system has repeated redundancies (both hardware and software), but our primary objective is always to get it back online first; should a failure be confirmed we then move to secondary measures.    This particular system runs our most advanced (and heavy duty) hardware and software and is used to store data and maintain backups.

We proceeded to identify the problem and made a decision to restart that aspect of our network.  The reboot brought the system back to stability, and we were able to recover an error message; it showed a momentary hardware failure. Immediately our vendors were brought in to diagnose, and were able to confirm that there was a hardware anomaly for a moment this morning; the hardware failure did not manifest itself before or after the said event.

New parts are on order and we will be able to replace them online (without bringing down the system).  The nature of the issue - and the available redundancies in place - should not have triggered an a catastrophic failure. In fact, the purpose behind the failed parts is to prevent disruptions in case of hardware failures. The lack of diagnostic errors from three seperate hardware profiling systems made the issue even more perplexing.

After the new parts arrive we will replace and proceed to explore any other vulnerabilities the hardware may have. If necessary, the components will be replaced with hardware from another vendor.   In December we expanded our services, installed new hardware, and added new software to advance our networks further.  There are additional upgrades planned for first quarter of this year as we remove legacy systems and shift our hosting assets to the latest Xen Server and Microsoft technologies. 

We will need to bring down several of our network assets this evening to execute several hardware and software maintenance operations.  Due to the nature of the routines, the systems will be offline for approximately 4 hours.    Since overall network utilization is below average during this holiday week, we plan to begin all maintenance operations about 8:30 pm.  Majorty of the systems will be back online within 2 hours, followed by momentary interruptions as isolated systems are modified.

• Storage server systems will receive replacement memory to correct a malfunction.  The complexity fo this upgrade means most of the systems will be brought offline for about 2 hours.
• Some of our hosting premium customers will be taken offline for about 30 minutes as we install new storage.
• All hosted Exchange/Outlook servers will receive a critial Sophos/PureMessage antispam/security/antivirus upgrade.
• Some hosting servers will be restarted for updates.