cPanel Technical Proxy Machine Compromised

Here at Canadian Web Hosting we have been paying close attention to the recent security issue at cPanel.  You may have already heard but cPanel released a statement on their forums earlier this week that it had detected a compromise of its system occurring last Thursday.

The issue is based around a proxy server that was being used by a limited number of cPanel technical analysts to access customer servers for the purpose of support. The breach itself was administered by what cPanel described as “a malicious third-party.”   Despite the breach, cPanel emphasized stability, noting it had worked to stabilize the problem along with the fact that no customer data was found to be compromised.  You can find cPanel’s entire statement from cPanel Forums is included below:

http://forums.cpanel.net/f133/cpanel-inc-announces-additional-internal-security-enhancements-326702.html

As mentioned in our email sent to cPanel Server Administrators who’ve opened a ticket with us in the past 6 months, on February 21 we discovered that one of the proxy servers we utilize in the technical support department had been compromised. The cPanel Security Team’s investigation into this matter is ongoing.

We’d like to relay additional details about the intrusion that we have gathered with you here, and we want to explain what preventative measures we’re putting in place that will introduce additional layers of security to our new and existing systems, already in place.

Here’s what we know:

The proxy machine compromised in this incident was, at the time, utilized to access customer servers by some of our Technical Analysts. It’s intent was to provide a layer of security between local & remote workstations and customer servers.

This proxy machine was compromised by a malicious third-party by compromising a single workstation used by one of our Technical Analysts.

Only a small group of our Technical Analysts uses this particular machine for logins.

There is no evidence that any sensitive customer data was exposed and there is no evidence that the actual database was compromised.

Documentation on how to Determine Your System’s Status is available and we encourage System Administrators to use those details to determine the status of their servers.

Here’s what we’re doing about it:

We have restructured the process used to access customer servers to significantly reduce the risk of this type of sophisticated attack in the future. We have also been working on implementing multiple changes to our internal support systems and procedures as outlined for your information below.

Our system will now generate and provide you with a unique SSH key for each new support ticket submitted.

We are providing tools to authorize and de-authorize SSH keys and instructions on how to use them whenever you submit a ticket.

Our system will generate a single-use username and password credentials for accessing WebHost Manager that are only valid while our staff is logged into your server.

Additional enhancements are also planned behind the scene that should be transparent to our customers.

With these new layers of security in place, it is now possible for our Technical Analysts to service your support requests without you providing your server’s password for nearly all requests involving machines running our cPanel & WHM product going forward. However, we will still offer the ability to provide your password for server migrations, or in the event you cannot use SSH keys.

cPanel’s Internal Development Team has been working on an automated solution with the end goal of eliminating the need for our Technical Analysts to view any passwords you provide during the ticket submission process. We are testing this solution right now, and hope to have it fully implemented in the next few days.

cPanel, Inc. understands your concerns expressed over the last few days, and we very much appreciate the cooperation and patience you have provided us during this time as we work through all of this.

Thank you.

cPanel is one of the industry’s leading control panel platforms for Canadian Web Hosting and its customers.  Utilizing cPanel, customers can easily manage their web presences with tools that allow them to coordinate files, create databases, set up email accounts along with performing a multitude of other tasks.

Kevin Liang
CTO / SEO Guru
Canadian Web Hosting

BlackBerry Z10 Email Issues

With the recent launch of Blackberry’s new devices, in particular the Z10, we have seen many of our Canadian Web Hosting customers embrace the new device and have heard a lot of positive feedback.   Unfortunately, there seem to be a common email issue for customers that is both troublesome and may require a patch from Blackberry to fix.  Customer should note that Blackberry z10 emails may not be able to be sent through your web hosting company.

Over the recent days, we had numerous chats with Blackberry support because of numerous complaints and reports of emails not being delivered. The technicians that we spoke to have also confirmed the way emails are being configured on the BlackBerry Z10 is not RFC2822 compliant.  Instead, the emails are being configured with LF (line feed) on the formatting and not CRLF (carriage return line feed).  This is significant, because in breaking RFC2822 (http://tools.ietf.org/html/rfc2822)  many mail servers and Spam appliances, including industry leading devices from companies like Barracuda, check against this standard and non-compliance will typically mark those emails as Spam and force non-delivery.

As an example, our email server logs also confirm this with messages like these below:

---- ---------------------- --------------------------------------------------

0.14 MISSING_MID Missing Message-Id: header

0.01 INVALID_DATE Invalid Date: header (not RFC 2822)

0.00 MIME_HTML_ONLY BODY: Message only has text/html MIME parts

0.00 HTML_MESSAGE BODY: HTML included in message

1.05 HTML_MIME_NO_HTML_TAG HTML-only message, but there is no HTML tag

0.20 MISSING_MIMEOLE Message has X-MSMail-Priority, but no X-MimeOLE

1.19 K2_MISSING_MIMEOLE Message has X-MSMail-Priority, but no X-MimeOLE

1.76 INVALID_DATE_2 Invalid Date: header (not RFC 2822)

=============

After further review, the only viable work around is to have that email address whitelisted or have BlackBerry come out with a patch.  Many web hosts like Canadian Web Hosting will be reluctant to whitelist IPs because of the potential that these mail IPs may be abused to get blacklisted, which in turn will cause even more emails to be blocked.    You can find further discussion in Blackberry’s support forum including threads like this one - http://supportforums.blackberry.com/t5/BlackBerry-Z10/z10-faulty-code-BUG-emails-may-not-be-sent-released-if-you-have/td-p/2195691.

Kevin Liang
CTO / SEO Guru
Canadian Web Hosting