March
24, 2005 .As of 2:00 AM
MST
All dedicated IPs are
assigned. We have also completed our first round of security modifications.
We will also be working on the
mail system to increase performance and efficiency on the system. This should
not interrupt any mail service however. Ticket responses times should now be
increasing as our staff catches up with the work load this issue has caused.
Data recovery specialists
were able to access the original SCSI drives with more recent data on it
taking us up to the point of the incident for those needing files for this
past Sunday and Monday. The drives should be over-nighted back to our
Data
Center today, and we will make this
data available to fill any remaining gaps as soon as
possible.
March 23, 2005 .As of 4:30
AM
MST
All sites have been restored. We will
be performing tests, and will start security related installs later today on
this server. 1 or 2 reboots taking about 15 minutes should be expected over
the next 12-24 hours.
Anyone using SSL please create a
support ticket to have it re-enabled. If it was issued by another company,
please send the Key information to us in a ticket. If it was from CWI, simply
mention you need it enabled if for some reason it isn.t already.
JSP has been
enabled on applicable accounts. Any
sub-domains needing JSP or web apps enabled, please let us know. We
will fulfill requests for the above as quickly as possible as brought to our
attention.
March 22, 2005 .As of 10:15
PM
MST
We are currently re-enabling JVMs for accounts that are
using JSP. We will start reassigning the IPs to all accounts that had a static
IP. IPs will remain the same and should not
change.
March 22, 2005 .As
of 7:15 PM
MST
The majority of the
sites on this server are now online, however more items will need to be
restored such as any SSL Certs, and FrontPage for example. Performance will
increase once the file copying is completed. The files are being restored from
a remote backup server that CWI maintains for ALL its shared hosting clients
to help protect their data. These backups will be from this last Saturday
(3-19-05). If any critical files are needed from
between Saturday and Monday evening, once we receive the drives mentioned
below back, we will put them on a standalone box so those files can be
retrieved as
needed.
March 22, 2005 .As Of 3:30PM MST
Server 209.18.119.5 (elise.cwihosting.com) is currently
being worked on due to an emergency situation that required all the hardware
to be replaced. This was a precautionary step due to possible failing hardware
and that phase was completed last night due to operating system kernel panics.
It is now believed that a web exploit was used and was
the root cause for the server issues. Sites are currently coming online each
minute on the new server from a Saturday backup as neither of the redundant
RAID drives would mount. If your site is not up yet, it will be shortly and
the system goes through the list.
We have sent the original drives to a data recovery
center, and once available we will further investigate the exact exploit to
block against it specifically. Security and reliability is very important to
CWI and we highly appreciate our patience.
Sincerely;
Team CWI
