2010 EnergySec Summit 9/21-9/22, Denver

For six years running, the EnergySec Summit has proven to deliver the highest value when it comes to Energy/Electric Sector security events. And no, I’m not just saying that because I’m on the Board of Directors [FD]. I say it because I have yet to see another conference where such a diverse body of professionals meet under one roof and actually talk to each other. The variety of attendees ranges from all organizational strata of electric asset owners spanning the operational and business groups, State Regulators, Feds of all agencies, national labs, consulting firms big and small, security product vendors, industrial control systems vendors, research firms, and universities.

But it’s not enough to just get great minds to the table. The magic is when those great minds begin sharing in ways they hadn’t before, thus generating a result that is greater than the sum of its parts. The EnergySec culture draws this out of you. The collaborative spirit is simply pervasive and infectious. There is no singular agenda or axe to grind. The audience isn’t just there to listen and blink at powerpoint slides from the so-called “experts” all day. Instead, all attendees are recognized as experts and they are active participants in the event. You can’t help but leave empowered with new connections, new knowledge and an invigorated desire to do more for your company and your industry. Sure, this happens at many conferences, but there is something unique about the EnergySec vibe that takes it to a different level.

The information sharing grass roots effort known as EnergySec today has matured from its humble beginnings back in 2004 as Energy Security Northwest (E-Sec NW). Those original meetings of adjacent asset owners over lunch and the mailing list were the seeds of something great. The membership started as an all-volunteer group with no real organizational structure other than a loose “board” of moderators who maintained the vision of the EnergySec culture. They fostered it through technological advancements and beyond into non-profit status as a 501(c)(3) company. Today, it boasts 380+ members, 102+ organizations, 50%+ North American Generation, 65.3%+ North American Distribution, etc.

Ok, so back to the Summit... ICF International [FD: my current employer] kicked things off with a networking breakfast on Day One. I followed with my “Energy Sector Crystal Ball” spiel to kick off the event and set the mood. I really tried to keep it FUD-free and just talk about trends I’m seeing within the industry. Carol Hawk of the Department of Energy then walked through some of the projects they are currently working/funding - one of which is the National Electric Sector Cybersecurity Organization (NESCO). More on this again later. Mike Mertz of PNM then presented on security policy. His subtitle was “The most important element for Security and Compliance.” By the time he was done, he had me convinced. Mike always has such a great way of getting to the core of the issue and providing actionable take-aways. Bill Hunteman of DOE was up next to tell the group about the department’s strategic direction for smart grid security, an area in which nearly everyone is interested. Bill was followed by Brian Girardi of NetWitness presenting on network forensics & advanced threat analysis for critical infrastructure - an area where NetWitness is recognized as a practice leader.

Then came lunch, sponsored by NetWitness and more networking. I eavesdropped on several conversations as I usually do and joined in as many as I could. I am always surprised to see how much fusion happens when the walls come down and the silos are broken.

Lunch was followed by an entertaining and insightful presentation from Mike Assante (now with NBISE, formerly NERC Chief Security Officer). I particularly enjoyed the parallels between suicidal squirrels and the threat actors, but this portion of his talk was curiously left out of the presentation he provided to EnergySec for posting (I’ll see if he’d be willing to furnish the un-redacted deck). Suicidal squirrels aside, Mike had some solid and sobering wisdom to share about how regulatory compliance has affected cybersecurity for the industry. Jack Whitsitt, Sr Critical Infrastructure Cyber SME supporting the Transportation SSA followed Mike with a technologist’s admission of inadequacy, the executive’s role in national cybersecurity. Though Jack doesn’t come directly from the power biz, he knows industrial control systems security and his subtle-but-wry humor and acute technical references really fit the audience well in my opinion. Tim Erlin of nCircle was next with a talk on configuration auditing and vulnerability management in the NERC CIP era. Short of the free tool Nessus, nCircle probably has the largest presence in the vulnerability and configuration management space within the electric sector, based on what I’ve seen as an auditor and consultant.

Day One closed with a panel on the Summit’s primary topic, the intersection of security and compliance. The panelists were: Josh Axelrod, WECC CIP Audits and Investigations Team Lead; Dave Norton, Entergy Policy Consultant - Critical Infrastructure Protection; Mike Assante, NBISE President and CEO; Mike Mertz, PNM Resources Sr Project Manager of FERC Compliance and Ben Miller, Constellation Supervisor of Information Security Operations. As you can probably imagine, this was a lively debate. There was no shortage of opinions, from the panelists or the audience. A recurring thread was that the existing NERC CIP standards may be decreasing security in some (but not all) implementations for the sake of compliance. Some great examples of satisfying both security AND compliance were also provided. It was a fantastic way to end the day and really primed the discussions during the reception.

Big thanks to nCircle for providing drinks and hors d'oeuvres. Liquor and food have a funny way of changing the dynamic. After a day immersed in the collaborative culture and high-value content, minds were reeling and ruminating. What was left of the inhibition to release opinions quickly vanished, which made for weighty and direct conversation. I eavesdropped some more, made more new friends and had my fair share of the hotel’s Bourbon.

Day two started with Tim Roxey, NERC Manager - Critical Infrastructure Protection. Tim’s presentation did a fantastic job of capturing the complexity of today’s power grid. The analogy of flocking birds as the intelligent future of power management was very intriguing. The vulnerabilities in such a complex system are not being fully considered - yet. Next came the highlight of the event in my opinion. James Arlen (Push The Stack Consulting) reinforced the information sharing purpose with his presentation “SCAD’oh - An Agnostic’s View: stop the ‘us v. them’ and come together to solve the problem.” James delivered the message with refreshing candor and corrosive humor. No one was safe. We all got a chance to laugh at each other and ourselves. We were all well aware of many opportunities for self-improvement by the time James left the stage. But everyone needed to hear every single word. If there was any ego remaining in the room, they simply weren’t listening. Up next was Steven Parker, EnergySec Director. Steve used puzzle pieces handed out at the beginning of the event to convey his message of sharing and collaboration. Each attendee put their name on their piece and then handed them back at the end of Day One. On Day Two, the puzzle was available for assembly at the registration table. The message: we are all parts of the cybersecurity puzzle. Only when we work together do we form the complete picture. Steve also included a briefing on the EnergySec state of the union including the announcement of the formation of the NESCO. The event was wrapped up by Sean McBride of Critical Intelligence with a presentation on developments in ICS defense. As usual, Critical Intelligence hit the mark based on their deep involvement in the subject. CI provides the light version of their open-source intelligence reports to EnergySec members.

Day Two ended with a final networking lunch. You probably guessed it but I eavesdropped again, chatted more and exchanged contact details on napkins because I was out of business cards.

Throughout the event I received strongly positive feedback from the attendees on the quality of the content in addition to the networking/collaboration benefit. It may sound like a big hugfest or kumbaya session, and it was to a certain extent but there was also plenty of dissenting opinion and experienced debate to keep things balanced. No doubt that everyone likes to know they're not alone in the difficult struggle to secure the North American power grid, and the level of professional exchange is what makes the EnergySec Kool-Aid taste so good.

Another interesting addition this year was the live tweeting from the event and commentary/discussion from those on the floor and those who were unable to attend. The real-time discussion went beyond the walls of the conference room. Very cool indeed. I finally met a bunch of people from Twitter face to face (@sintixerr @myrcurial @electricfork @CrucialCarl @tmdheard @jholcomb @pmhesse to name just a few).

Lastly, a HUGE thanks to Lisa James, EnergySec Chair and her supporting cast of volunteers, Jeri Freimuth and Ed Croft. I’ve been to very expensive conferences with paid armies of staff and the EnergySec logistics were better.

I’d like to close the post with a brief note about the National Electric Sector Cybersecurity Organization (NESCO) mentioned above. EnergySec was awarded a $5.8M grant over three years to stand up the NESCO. The NESCO will be supported by EPRI and their collaborative as the research element, NESCOR - the “R” is for resource. Some details are in the presentations from Carol Hawk, Bill Hunteman and Steven Parker. You can read more in the official press release. If you are interested in hearing more, contact Lisa James, EnergySec Chair at lisa@energysec.org.

The photo is the Sheraton Downtown Denver.

The Cabin Door Is Closed, Please Power Down All Electronic Devices...

I fly a lot. More than many. Sometimes over 50%. As such, I have some experience with the consumer side of the commercial aviation business. I am by no means an expert. Recently, I was asked to speak at an air traffic controller's conference on the subject of cyber security in the Next-Generation Air Transportation System. I was there to provide a perspective from the outside, more of a security technology discussion for what works in the overall CI/KR space. It was a panel, so the slide deck was short - which was good for me, because again, I'm not an expert in aviation. The panel had current and ex-FAA staffers, university professors, aviation consultants from the defense industry/sector and me. Do you remember the old Sesame Street song "one of these things is not like the other?" Throughout the event, I was constantly reaching for my smartphone to Google the acronyms that I'd never heard.

I had a similar newbie feeling when I started working in the electric power sector. Before then, I was just a consumer. I flipped the switch and expected the lights to come on. When the power went out at my house, I was the first to call my local utility and give them a piece of my mind as a paying customer. After all, us geeks can't live without our tech toys for longer than a few seconds. So the first time I actually spent more than five minutes on the Control Center floor, I was.. well.. floored. Even mild spring days in the 'shoulder months' can seem like a delicate balance of order and chaos. The electric system is interconnected, just like the aviation system. What happens in one area will quickly and directly affect other areas, some quite distant. The real-time seat-of-the-pants decisions by system operators is really what keeps the system running - not the technology. Sure, the technology is there, but it is only a tool.

I see a situation, whether it is Smart Grid in the power biz or the Next-Gen Air Transportation System in the aviation sector, where we are inserting a much wider technological distance between the human and the physical/kinetic endpoint. System operators are using ever-increasing layers of technology. Until fairly recently, they looked at some sort of analog or electro-mechanical instrumentation for operational decisions and then they would physically (manually) activate something. Today, we have operators using tools which are in turn, using other embedded tools, which may also be using further embedded tools - and so on. This can be a good thing for many reasons, but it can also be a bad thing. This trend, though perfectly natural - even expected, should be carefully monitored, carefully balanced. Especially when it involves critical infrastructure. We may even need to tip the scale toward sound security engineering instead of focusing solely the profit drivers. At least for a while.

We've ignored our critical infrastructures for so long that we are in desperate need of an overhaul. Nearly every one of the sectors in the National Infrastructure Protection Plan (NIPP) could be called brittle. Some money is starting to flow to these areas for much needed upgrades but the legacy technology and the bleeding edge enhancements need to work together in the same interconnected system. This creates a 'base of sand' problem. Legacy devices are underpinning tomorrow's technology gizmos with incredible distance between the two ends of the spectrum. We need a security engineer to put their stamp on the blueprints BEFORE they get the permit to build. When adding to or modifying an existing structure, the structural engineers factor those old trusses, supports and cracks in the foundation into the new design. I don't want to discount the great work being done here, but I think few would disagree that we have a cart-before-the-horse situation.

The most common recommendations I heard at the recent aviation conference were:
- Test bed for qualifying systems (approaching Certification and Accreditation)
- Minimizing potential of supply chain attacks
- Security Training for operators/controllers
- Situational Awareness (and integrity of decision support data)
- Information Sharing

Those of you following the power sector for the past few years should see some striking similarities. I'm willing to wager that nearly all CI/KR sectors are facing these same challenges. The only recommendation didn't see was slowing down to get security issues addressed in the design phase. I've been a security professional long enough to expect that, but I can't seem to bring myself to accept it - hence this post/rant.

Like nearly all of my posts, I am writing this as I fly home on a commercial airline. Now if only I can think of a solution to being crammed into a space smaller than my anatomical dimensions. I'm not important or rich enough for First Class seats. But every time I think I've got it bad, I remember my co-worker CJ and his 6'5" span. He's taller than most clearances at drive-thrus and parking garages. Unfortunately TSA frowns on bringing a crowbar to extract him from the seat.

The photo is from CodeProject

Is Reliability In Your Future?

I'm hearing a new wave of disdain for the NERC Reliability Standards from the industry. This happens from time to time and it isn't just about the CIP Standards. The Order 693 stuff gets its fair share of noise too. The most common thread is how all of this effort doesn't really improve reliability of the power system. I hear it from plant and system operators. I hear it from comm-techs. I hear it from all ranks of management, from the front lines all the way to the executive level (though middle management seems to be the loudest). I even hear it from the IT staff but to a lesser extent.

Granted. The Reliability Standards are a pain. Lots of work, lots of money and lots of time spent to reach the magical state of Compliance. It deserves some of the frustrated noise that it gets, but not all.

One benefit: Accountability.

I know I'll probably take some heat for saying it, but in my opinion, holding people accountable for their actions will improve reliability of the power system. Accountability is a powerful tool for maintaining integrity. Some of the most obvious examples of accountability in action are cameras. They are aimed at cash registers while capturing POS data, watching the watchers at daycare centers and schools and even publicly scrutinizing police officer actions via headcams. No, system and plant operators shouldn't be fitted with headcams, but they shouldn't fear accountability either. I can sense a strong authority vibe coming from them and it seems that they perceive these standards to be chipping away at their ability to freely make grid management decisions. The accountability elements built into the standards will only take away your ability to make decisions anonymously. Believe it or not, this could actually help you and your system.

The photo was taken by a friend of mine who says the graffiti isn't his - and I think I believe him. And to quote his response on the subject: "no, it isn't, but hopefully it is recoverable." Thanks SHP. Please, no bathroom humor.

CSO706SDT FAIL

Dear friends in the electric power industry: this CIP-010 and CIP-011 draft baffles me. I had a heck of a time trying to audit the first one and this new one leaves me deeply sympathetic for the poor auditors I left behind (sorry guys). You have no idea how challenging it is to call a ball or strike with CIP-002 through CIP-009 as an auditor. Well you might have an idea because you had to implement it - or should have anyway. With that, I hope you see my point that inserting additional flexibility and vagueness will only make your job implementing these requirements even harder. It will also make your auditor's job more difficult. These two facts increase your risk.

So, what happens if you get this one wrong? What happens if FERC remands it? Will it cause a ripple effect that could possibly spell the end of the ERO's oversight of security for the industry? Will Congress decide that our industry can't self-regulate, therefore they need to step in and "save" the grid from the cyber-boogeyman? Sure, these are extreme cases but they are still in the realm of the possible. And if we have an incident, think ESA. Remember what happened to the airline industry. You may not be able to enter a substation unless you've gone through a full body imaging scan and your liquids and gels are all less than 3.4 ounces in a one quart clear baggie.

CSO706SDT, especially after listening to the recent Version 4 Workshop, I implore you to listen to the auditors. They are not the enemy. A few points that bear repeating:

- Define stuff. If you haven't defined your terms, you haven't written a standard. "Annual" is only one of the many words you need to clarify.

- Attackers aren't constrained by budget and time. If we are, they have the advantage.

- Remember Moore's Law. Technology will transform significantly within ten years. Consider more realistic implementation deadlines. In fact, make it simple and give us a single [sane] date.

- Write the standards in such a manner as to eliminate the need for a Technical Feasibility Exception.

- Access points matter. Allowing anything is like saying a shoji screen is equivalent to a steel door.

- Go ahead and call it a firewall.

- Terms like boundary, border, perimeter are all acceptable. Most professionals know that this means "preventive control." Removing the ESP and PSP language may do more damage than good, despite the pre-existing confusion. Require a perimeter, with a DMZ.

- Low impact systems deserve protection. Packets don't care about arbitrary labels.The way it is currently designed, "stupid" would be a compliant password for low impact systems. Minimize the potential for gaming the system and labeling everything "low."

- Be thinking, with every requirement you construct, "how would someone evidence this?"

Electric sector, just go secure your systems. It will cost you money. It will take time and resources from other projects. Accept it. Embrace it. The sooner the better. If you start securing your stuff now, you will have less work to do when someone finally hands you a security standard. The situation won't get better in the future. There aren't enough security professionals who can spell R-T-U. The Feds aren't going to let sloppy or weak security standards prevail. The economy isn't going to turn around tomorrow with lavish profits to pay for it all. The time is now. Grab a spoon and start eating the elephant.

We owe it to ourselves to step this up. We owe it to ourselves to get it right. We are engineers, operators, security professionals and generally very smart people. We can do this. We've solved harder problems before. The reality, however, is that we will only solve problems we want to solve.

Oh, and Hello World. This is my first official blog post.