Update on ConnectWise Manage® outage in the EU.

13/05/19 - Following a thorough investigation, we can confirm that the outage was caused by a security incident.

On Friday, 3 May, at approximately 7:30 am BST, the following series of events occurred:

•  ConnectWise was alerted by our internal monitoring and security systems that some of our SQL databases in our EU-AWS cluster were not accessible.

•  We quickly realized that several servers were inaccessible due to critical failures.

•  Our incident response procedures were immediately enacted, and our internal teams responded within minutes to assess the situation and began to monitor the environment and analyze the alerts.

•  These servers were immediately taken offline and access to the entire cloud network was restricted to a select number of colleagues.

•  Our initial examination pointed toward some type of malware.

•  The cloud team built and deployed new AWS clusters with known good backup restorations. This contributed to the downtime experienced by ConnectWise EU partners.

•  As our investigation ensued, our teams discovered that the malware was ransomware.

•  All partner access was restored by 3:16 pm BST.

•  Email Connector service was enabled at 4:20 pm BST.

•  Reporting services were back online by 5:15 pm BST.

•  A third- party forensics firm was engaged to perform a comprehensive investigation.

The forensics firm confirmed that the ransomware variant used in the attack only encrypts files, and is not designed or capable of reading, removing, or altering data. The only impact of the intrusion was loss of access to our hosted SaaS application. We found no indication that any personal data was destroyed, altered, disclosed to, or accessed by an unauthorized party. Accordingly, we do not believe there is a risk to the rights and freedoms of EU data subjects as a result of this outage. We were able to identify that the intrusion came from an offsite machine that was used for cloud performance testing outside of our network. Going forward, we have immediately prohibited any such offsite systems testing.

The following actions are being taken to prevent a similar incident from happening in the future:

•  We have completely rebuilt, scanned, and setup all new servers in our cloud infrastructure across North America, EMEA, and ANZ.

•  All passwords in the environment were reset immediately.

•  All access to the infrastructure from outside the network was blocked immediately

•  An additional layer of authentication was added to the environment for all users.

•  An additional layer of security was added between the SQL clusters and the rest of the environment.

•  An additional step was added to snapshot the transaction log backups each hour to reduce the recovery point in the event the transaction logs are compromised.

•  We have updated our procedures for remote access and added additional monitoring and training.

Now that our investigation is complete, we will be filing a complaint with the appropriate law enforcement agencies. Over the coming weeks we will provide more thorough documentation regarding our security practices, penetration testing, SOC, and product security analysis. Our team is here to help with any questions you may around an incident of this kind. Please direct questions to SecurityResponse@ConnectWise.com.

ConnectWise Automate Hot Fix

2/27/19 - Update:

•  We are working with less than 50 partner servers that still require either the hotfix or patch

•  ConnectWise Account and Product Management are actively calling remaining partners with unpatched servers.

•  More than 5,000 partners have completed the full 2019.2 patch since it was released earlier this month

•  All cloud partner servers have been updated to 2019.2

•  Currently there are no negative support trends related to the 2019.2 release or the hotfixes

•  We have released a utility to the community and our support group that can mass upgrade Automate agents that have not been updated to a fixed version prior to March 8th using active Control agents.

We will continue to update you on progress related to this issue.

2/8/2019 - We have been working to solve an issue discovered by the LabTech Geek/MSP Geek Community Administrators, for the past week. We take this issue seriously and have been working to build and test solutions for all impacted versions.

Automate partners on the following versions are affected :

•  Automate 2019.1 [19.0.1]

•  Automate version 12 patch 12 [12.0.12]

•  Automate version 12 patch 11 [12.0.11]

An update must applied to your Automate server that addresses how we handle the transition to our new code signing certificate. Failing to address this issue this month will result in the failure of agent and control center communication.

Next Steps:

•  All Automate cloud partners will be upgraded to 2019.2 next week to resolve this issue.

•  On premise Automate administrators running impacted versions will be receiving emails on Friday (2/08) with options to address this issue.

ConnectWise Manage and Kaseya Integration

2/5/2019 - We worked with Kaseya when the vulnerability was originally identified and we've been working with Kaseya to correct the issue for those MSPs impacted recently. By working closely with the Kaseya team, we determined that MSPs currently being impacted by this vulnerability may have installed the update incorrectly. We are pushing out an update today to ensure the plugin is configured correctly to prevent the previously known vulnerability.

Security is important to us and we always recommend that partners keep systems updated and use the ConnectWise team as a resource. When we provide updates to ConnectWise products, integrations or plugins, we send out emails and in-app messaging to alert MSPs of the update. Partners can learn more about the update by contacting ConnectWise support or by visiting the ConnectWise Marketplace here.