Top 10 Take-Aways From Colonial Pipeline That We All Must Take To Heart

Colonial Pipeline CEO Joseph Blount testified before the Senate Committee on Homeland Security & Governmental Affairs on June 8th, and his testimony helped fill gaps in our understanding of this attack which all of us, in every industry, who are in any way involved in information security need to take to heart and learn from.

Before I continue let me say that I’ve put this together not to condemn Colonial’s preparedness, nor besmirch their reputation, but to call out deficiencies in their preparedness that I have a strong feeling plague companies across sizes and industries throughout both our nation and the world. Let’s learn from this before we experience it for ourselves. After all, 2020 saw 65,000 ransomware attacks in the US, averaging more than 7 per hour.

I’ve spoken about some of these topics recently with John Williams on WGN Radio in Chicago, including as we first learned about Colonial’s troubles and even recently as we heard the FBI recovered some of the ransom, and these thoughts represent some of the concepts and ideas we were exploring in those discussions.

Initial Penetration to Detection

Based on reporting by Bloomberg, Colonial was penetrated on April 29th via a VPN account which was not protected by Multifactor Authentication (MFA). The password to this account was compromised. In his testimony, Mr. Blount assured the Senators that this password was a strong password. Instead of being a defense of why MFA was not in use for this Internet-facing system, it underscores the need to move away from relying on passwords of any complexity as the sole means of authentication for any Internet-facing systems.

Colonial information security resources identified the ransomware on May 7th, 10 days after the initial intrusion. They learned of it only because a ransom demand was posted to them, not through their own detection of the intrusion, also per the Bloomberg reporting. Colonial did not notice any of the activity leading up to this. While hindsight is always 20/20, those 10 days were an opportunity to identify this while it could be contained, and before it caused the wide ranging problems that it did. Unfortunately, their own detection capabilities did not allow them to do so.

We do not know how the infiltrators were able to move around the environment once they were in. It is possible that the password used for the VPN intrusion was a valid password for internal systems as well. I know from my time working with pen-testers that once you are on a network and can identify the domain controller, you can generally find a privileged account with a weak password to exploit further. Barring that, we know that you can often find unpatched systems once you’re inside an organization that offer similar capabilities. Regardless, it brings up the point that security isn’t just about the perimeter; we need our defenses in depth.

Mr. Blount continued to refer to the VPN system that was attacked as a “legacy” system. This reminds us that decommissioning systems is a critical information security requirement. How many other companies have some legacy system that they’ve gotten 99% of their users and systems migrated from, but is still hanging around years later because of that 1% that just can’t/won’t be moved? I know I’ve worked for one or two with this scenario.

Initial Response

I commend Colonial Pipeline’s quick reaction once they did detect the issue. Responding to this event was a complicated challenge, and it seems they moved with a balance of speed and caution, knowing that they needed to protect their operational technology (OT) environment that was responsible for the flow of volatile gasoline and other products across the country. They were quickly in touch with critical governmental agencies and teams, and it seems they received guidance and assistance from the government. Additionally, they quickly worked with a variety of contractors and vendors to assist them in the IT, IS, legal, negotiation, and payment processes required to sort this out. High marks there.

They brought in Mandiant, Dragos, and Black Hills all as forensic/InfoSec remediation professionals, paying for redundant services in an effort to be sure they resolved the issue as quickly and thoroughly as possible. Expensive to be sure, but given their status as critical infrastructure and the large impact to the country if they’re not thorough, it makes great sense.

Payment

Colonial began negotiations to make payment late in the day on May 7th. As part of that process, they had to validate that the entity they were paying was not on the OFAC list of sanctioned entities, which would have made paying the ransom a federal crime. They worked to negotiate the payment price down, all while they were beginning the process of planning how they were going to dig out of this event.

One disturbing realization is that at the time they made the decision to pay the ransom, they still did not know how much of their environment had been compromised.

While I would never encourage anyone to pay a ransom, I can empathize with Colonial’s rationale. They quickly calculated that restoring their systems on their own would not be a quick enough process, and they made the calculation to pay. Unfortunately, that decision came because of both the severity of the attack and their unpreparedness to quickly recover. This points to their business continuity and disaster recovery plans and exercises. Are any of us preparing ourselves for recovering from these attacks, or are we just kidding ourselves with small-scale disasters?

Ongoing Remediation

Included in Mr. Blount’s testimony was the admission that it took them a month to restore seven of their financial systems and get them back up and running. Mr. Blount states that he expects that it may well be months before they are 100% recovered, even with the decryption solution provided to them for paying the ransom.

Staffing

We’ve seen online that for months before this event, Colonial Pipeline was searching for a Manager, Cyber Security for more than 30 days before the event. Mr. Blount’s testimony included that they have over 100 people in IT (note, he doesn’t say Information Security) and that his CIO gets all the funding she needs. This suggests the following:

• Information Security is seen as a domain of Information Technology
• There is no C-level Security position
• The entire InfoSec team would roll up under a “manager” level resource, a role that has been vacant for months now (if not longer)

Attitude

“We take cybersecurity very seriously,” was a repeated talking point for Mr. Blount. I’m sure he means it. I would disagree with his assessment, looking at the evidence above. And again, I don’t think this is a problem unique to him and his company.

Information Security is not a revenue-generating part of nearly any company. We know this. We know that this means that funds are not unlimited. We know that this means we need to be efficient with our cybersecurity spending, but I think we’re clearly seeing that we have room to improve.

A C-level leader dedicated to Information Security cannot continue to be overlooked.

The Real Cost

We know that Colonial paid out $4.4 million in ransom, but what else did this cost? They clearly spent hundreds of thousands, if not millions, on third parties to assist with legal, InfoSec, negotiating, and other activities. The lost revenue from 6 days of empty pipeline is not likely to be trivial. And what did they have to pay in terms of the extra work their own teams did?

Moving forward, they’re still going to be paying for this for a long time. They’re still working to restore systems that have been unusable for a month. Mr. Blount’s testimony indicated that they’re doing a number of things manually that used to be automated, and it seems that won’t change any time soon, impacting productivity and likely raising internal costs as well.

A recent study noted that the average cost for ransomware recovery is in the ballpark of, coincidentally, $4.4M, which would bring Colonial’s out of pocket up to not quite $9M. We’ll likely have to wait for their next FTC mandated report to find out how much this cost Colonial, but I can already ponder the question of how much it would have cost for them to be better prepared than to have to pay for this cleanup.

TLDR; Just Give Me the Bullets

Colonial Pipeline’s experience is a great case study in all the ways organizations are not prepared for major information security events. They weren’t an exception, they’re quite likely the rule, and we need to all learn from their example. Reflecting on the ransomware scourge we’re facing now in the United States, with Colonial now in the rear-view mirror, here are my “Top 10” high-level take-aways that we have to prioritize now

10. Multi-Factor Authentication (MFA) for all Internet-facing systems (and as many internal systems as you can justify) is table-stakes. Passwords are not a reliable security control anymore.

9. Decommission legacy systems; don’t let them fade into obscurity while still representing risks.

8. Enhance your ability to monitor and detect “the bad guys” in your environment, with the aid of MDR and XDR services and solutions.

7. Being able to inventory your environment at the drop of a hat – not knowing what parts of your environment have been compromised is not helpful during an attack.

6. Re-evaluate your BC/DR plans and test them for these catastrophic scenarios, don’t just test them for “oops, the email system is out.”

5. Knowing that recovering from these attacks requires huge costs and long timelines.

4. Elevate your InfoSec to a C-level department; stop letting them be an off-shoot of IT.

3. Ask yourself, in light of an attack like this, are you REALLY taking InfoSec seriously? Could you say that straight-faced after an attack like this happened to your organization?

The two toughest lessons I’ve left for last. 

2. Law enforcement isn’t going to be able to turn back the clock to before you were impacted. Even if they somehow recover the ransom or bring the perpetrators to justice, you’re still going to see your business disrupted, you may have a PR issue and see diminished customer income for the long haul, and you’re going to have a large expenditure just to get back to where you were “the day before” the attack.

1. Your company, yes yours, is a target. You may only be a target of opportunity, but you’re still a target. You may not be a target for ransomware, but you’re still a target. So you better be prepared for the “what if” scenario, because it is much more likely than you probably thought in April.

Talk to us about your ransomware concerns and explore how advanced managed detection and response solutions can help reduce cyber risk for your organization. 

↑