Q&A: Colonial Pipeline Cyberattack Lessons Learned

By Theresa Houck, Executive Editor, The Journal From Rockwell Automation and Our PartnerNetwork

The ransomware attack that shut down the Colonial Pipeline on May 7, 2021, is considered the most impactful cyberattack against U.S. critical infrastructure to date, but it’s not the only one. Many infrastructure and manufacturing facility cyberattacks occur that we don’t hear about.

To find out what we can learn from this and other incidents, I spoke with Grant Geyer, chief product officer at Claroty, a leading industrial cybersecurity firm and a Rockwell Automation Digital Partner. We examined how the Colonial Pipeline ransomware attack happened, as well as how the February 2021 Oldsmar, Florida water treatment cyberattack occurred, and the far-reaching effects.

This Q&A also reveals some of the key lessons our industry can learn about cybersecurity to help the oil and gas industry, critical infrastructure facilities and all manufacturers.

You can listen to our full conversation in our “Automation Chat” podcast on your favorite podcast app or on the web, or watch our chat on YouTube.

Theresa: Describe what happened with the cyberattack that shut down Colonial Pipeline.

Grant: A piece of ransomware, which is malicious code that's used to seize up computers, found its way into Colonial Pipeline's IT environment. When they brought in incident response experts to help understand the attack and the potential impact, Colonial Pipeline reported they also shut down parts of their OT environment that stopped the transmission of fuel to the whole East Coast of the United States.

I've seen many attacks on the IT side of the environment. But as far as I can remember, this is one of the most impactful attacks that happened in the cyber world affecting events in the physical world. It led to the pipelines being shut down, which led to gasoline shortages, which led to gasoline stations being flooded by people trying to horde gasoline, which led to skyrocketing prices.

This is one of the first events that impacts the company itself, other businesses because of the fuel supply chain, consumers in terms of the cost of fuel, and the U.S. government from supply chain and critical infrastructure perspectives.

On the technical side, this is not just a ransomware attack, but a targeted ransomware attack. A group known as DarkSide tries to legitimize itself by appearing like a pseudo-legitimate business, but the reality is that this is a criminal gang that allegedly operates outside of the Eastern Bloc countries.

They figure out a target they want to go after, intimidate the target, put ransomware on their machines, and before locking up the systems, they'll steal the data.

Grant Geyer, Chief Product Officer at Claroty

NEW PODCAST

Lessons from the Colonial Pipeline Cyberattack & Steps to Take

The ransomware attack that shut down the Colonial Pipeline on May 7, 2021, is considered the most impactful cyberattack against U.S. critical infrastructure. In this “Automation Chat” podcast episode, Executive Editor Theresa Houck talks with Grant Geyer, Chief Product Officer at Claroty to examine how the Colonial Pipeline cyberattack happened and its impact.

Also learn about the asset operator’s role as the first line of defense; how converged IT/OT networks are vital for ICS efficiency, but also increase the attack surface available — and what to do about it; the technical and organizational features of a well-thought-out cyber defense; lessons learned that are useful for every industrial firm and critical infrastructure facility. And much more.

Listen on your favorite podcast app or on the web, or watch the conversation on YouTube.

Listen Now Watch Now

The thought process here is that, if a company has a good backup, a recovery program in place, and they can restore the data, the DarkSide group still has seized that data and can release it bit by bit onto the Internet to continue to raise the stakes around the release of sensitive information.

Theresa: Critical infrastructure is an easy target. For example, another high-profile one occurred in February at the Oldsmar, Florida water treatment facility. Talk about what happened there.

Grant: That’s an interesting case. A small water utility that leveraged a third-party tool for remote access called TeamViewer was all of a sudden connected to by someone outside of the environment. And an asset operator noticed that water levels were changing from 100 parts per million by volume drinking water to over 11,000 parts per million. And that is a lethal amount for consumers. Initially, the operator thought it was a mistake or that he was being tested, and he fixed it, but then it happened again. He literally saw the cursor moving across his screen and saw the levels being changed. This demonstrates how important it is to engage the asset operators as the first line of defense.

The second lesson is that often, we think cyberattacks that will come in from the IT side of the network and then go through the IT/OT bridge to try to compromise OT assets. But there are other entry points. This was a case where, because of the need for remote connectivity — especially during the pandemic — [remote worker access] provided access for attackers.

Theresa: Does the convergence of IT and OT make it more or less vulnerable to cyberattacks?

Grant: That's an interesting question because historically, OT environments were air-gapped from IT environments. I once heard that an air gap is a low-latency network, meaning that if business needs to happen, people will figure out a way of connecting the environments, whether it's having a workstation with an Ethernet cable into the IT network and another Ethernet cable into the OT network. Or they'll have a rogue access, or they'll put in USBs.

The advantages to be gained through digital transformation and convergence, whether they're in the public or private sector, are too high to be ignored. The key is how organizations do it safely and securely with good safe architectures in mind.

We're going to see a lot of converged OT/IT architectures where it will be important to have virtual zones, microsegmented environments, and what are known as zero-trust network architectures. These let users get to the assets they need, but you can prove they’re who they claim to be, and they have the rights to access those assets.

Theresa: What are other lessons learned not just from these two high-profile incidents, but also from the many cyberattacks we don't hear about in the headlines?

Grant: What is clear for any asset owner or operator is that you're going to have a variety of brownfield equipment that can’t be easily replaced due to technology obsolescence periods. It may not be patchable due to maintenance windows.

So, it’s clear that air gap is not the answer. You need to benefit from digital transformation efficiencies, but it needs to be done in a secure way.

Another lesson is understanding that it's a journey. Know your inventory of assets and which of those are vulnerable. It's a long game of reducing the inherent risks of those assets, and then monitoring for threats for the residual risk that remains.

A third lesson is how important it is to make sure your remote access is secure remote access. Make sure a users’ credentials to get into the environment can't be stolen by cyberthieves.

Key Lessons Learned from the Colonial Pipeline Cyberattack

1. Engage asset operators as the first line of defense. They know what’s going on in real time.

2. There are other entry points for cyberattacks besides coming into the IT network, including remote access points (remote workers).

3. With converged OT/IT architectures, it’s important to have virtual zones, microsegmented environments, and zero-trust network architectures. These let users get to the assets they need, but you can prove they’re who they claim to be and that they have the rights to access those assets.

4. Air gap is not the answer. When done in a secure way, you can benefit from emerging technologies and digital initiatives.

5. Understand that it's a journey. Know your inventory of assets and which are vulnerable. It's a long game of reducing the inherent risks of those assets, and then monitoring for threats for the residual risk that remains.

6. Make sure your remote access is secure remote access.

7. Conduct tabletop exercises. Ask, "What if what happened to Colonial Pipeline happened to us?"

And finally, conducting tabletop exercises are important. Ask yourself, “What if what happened to Colonial Pipeline happened to us? How do we respond? Do we have backup and recovery? Do we have a policy about whether we would pay ransomware or not? Who are the officials that we would need to talk to within the U.S. government? Do we require multi-factor authentication?”

Theresa: What else do you want industrial automation professionals to understand about cybersecurity

Grant: The most important thing that I’ve seen in successful initiatives in OT security is recognition that it's a team sport. It starts with an understanding of the mutual context between IT and OT. At the fundamental level, the least common denominator for both teams is managing risk.

It's getting to an understanding of what the differences are, what the organizations have in common, and through the different experiences, shared initiatives and shared goals, developing a strategy. That's where the magic happens — when it's not a political fight between IT security and the OT team, but working together and asking, “How do we win together to manage and mitigate risks knowing this very lethal threat environment that we're dealing with?”

^{The Journal From Rockwell Automation and Our PartnerNetwork™ is published by Endeavor Business Media.}

Q&A: Lessons from the Colonial Pipeline Cyberattack