Managing Prescriptive &

Auditable Regulations in OT

August 2022
Introduction Verve Industrial
Protection
• 28 years experience working in
Rick Kaun ICS across all OEM vendor
equipment
• VP Solutions
• Team composed of ICS/OT
• 21 Years in ICS Security engineers and experts

• 7 years @ Verve • 15+ years developing and

deploying our ICS endpoint
• 2001-2015 management platform to help
o Honeywell/Matrikon clients achieve greater ICS
maturity
• [email protected] • Dozens of clients across all
industrial sectors
• Software & OT/ICS-specific
professional services

2
Agenda
• What is the objective?
o NERC CIP/TSA or….Generally better OT security?
• What has traditionally been done?
o Nothing, Manual or ‘easy button’
• Why have we done this?
o OT is not so ‘static’ or straightforward
• What have we learned?
• What are successful companies doing?
• Case Studies
• Questions

3
What are we after?
• NERC CIP
• Security Directive Pipeline-2021 (TSA)
• CIS 18
• IEC 62443/ISA99
• NIST CSF
• API 1164 Rev: 3
o Requires Risk Based Implementation
• **Informal poll question – would you like to attend a technical walk through
of specific standard tasks and suggested methods to achieve them?**

4
Common Objective
• Applying ‘Security’ Controls into OT environments
• Must use a ‘Risk Based’ approach
o Recent adjustments to TSA requirements are a reflection of the balance needed between
a prescribed action and the resulting risk reduction
o Not always worth the effort!
• Being able to demonstrate their application
• Being able to maintain their status

5
What have we traditionally done?
• Individual tool sets for individual functions
• Often ill-suited for OT
• Back fill with manual effort
• ‘Best Effort’ ensues
• Can cause Significant Gaps!
• Objective is to reduce RISK

6
Challenge is properly grasping risk

Source - https://usea.org/sites/default/files/event-/Webinar%20Presentation_0.pdf 7
OT makes this even more difficult

8
Applying Risk Analysis What is
nature and
in OT target of risk? Can we ‘risk’ it?

Compensating
What is controls?
at risk?
Well
what do
we have?

9
 What successful OT companies are using
Reporting

Alerting
Automatically Updated

Ability to take
Actions

Asset Context (Status)

Asset Inventory

10
How? Think Global, Act Local
1. Need security SME support to
Scaled
develop alternative approaches analysis/
as necessary (e.g., network automation
designs for firewalls, - Prioritization
appropriate compensating - Playbooks
- Controls
controls when patches cannot
prioritization
be deployed, how to ensure - Log analysis
backups are timely and
accurate)
SME’s

Actions
2. Automate remediation actions - Networking
Data - Compensating
as much as possible (use controls
playbooks, etc. developed - Specific
centrally) remediation
advice
3. Need local control and insight
on executing remediation
actions: patching by device
Local:
(when/testing), account - Plant IT
removal, network rules, etc. - Engineering
- Quality
- Building
controls

11
Case Study - TGAL
• Client Profile
o Global Pharma Company
o 10s of thousands of assets across 50+
facilities
o Wide array of asset vintage, criticality and
origin
o Various regulatory, board/corporate security
needs
o Shrinking head count to combat security (OT)

12
 Case Study One - TGAL
• Without Global/Local Program (Blue Keep)
– Dozens of people and spreadsheets
• Updates tracked manually
– Multiple Meetings over days and weeks
– No concrete understanding of scope
• General idea as to scale but nothing specific
– Many ‘guesses’ and assumptions made in determining facts like:
• System criticality, owner, vintage, etc.
– Manual patch process begins
– Manual tracking also begins
– No realistic option for the application of compensating controls
• Remote desktop needed for 24 X 7 staffed HMIs?
– Total effort – Multiple staff deferred from day jobs for multiple days/weeks
– Total cost not released but roughly this:
 Case Study - TGAL
• With Global/Local Program (Log4J)
o Central team with 360-degree view of assets
• Accurate list of in scope by type, location, criticality, owner and function
o Prep Work:
• Small local profiling tool (not network based scan) deployed to all OS assets via automated action
• Log4J specific dashboard created to populate results worldwide to single dashboard
• Specific host-based intrusion detection monitoring added for Log4J specific activities
o Patching team deploys files and begins testing/deployments (where patching possible)
o Non-patched systems tuned to minimize risk (compensating controls – where applicable – like deleting library files or un-
used/non-necessary software)
o All progress reported in live dashboards
o Total effort – 3 core staff plus field techs for fragile system ‘oversight’
o Total cost not released but roughly this (70% savings):

14
 Case Study 2 - Categorized Company
Workflow
COLLECT ALL DATA PREPARE Files/Patches PREPARE TICKETS
Tasks Tasks Tasks
•Collect patch list (OS) •WSUS? •Create tickets for all tasks
•Export Vendor list •Download files and capture hash •(by end point, action or task?)
•Look up ‘other applications’ •Export Asset Report (Before profile) Include current
software status and OS version and cumulative update

CLOSE TICKETS GATHER EVIDENCE DEPLOY PATCHES

Tasks Tasks Tasks
•Append individual asset reports manually •Screen shots of install •Manually load and install all patches
created/screen shots •Screen shots of software/OS version •Verify patch succeeded
•Troubleshoot if necessary

15
 Automating Company Workflow
COLLECT ALL DATA PREPARE Patching PREPARE TICKETS
Tasks Tasks Tasks
•Export Vendor list •Create Vendor list in VSC •Create tickets for all tasks
•Look up ‘other applications’ •Create actions for applications in Console including •Export Asset Report (Before profile) Include current
•Download files and verify hash hash from download software status and OS version and cumulative update
Benefits Benefits Benefits
•Verve Inventory provides automated, real time, •All hashes in console mean that agent and/or server •Captures current software status and OS version and
accurate status, versions, etc will reject file that does not match hash cumulative update
•Building actions along with OS download allows for •Reporting only display assets that have patches that
instant reporting on multiple fronts are relevant (by Vendor or application, etc)

Automation significantly increases accuracy and decreases time/effort (by over 60%)

CLOSE TICKETS GATHER EVIDENCE DEPLOY PATCHES

Tasks Tasks Tasks
•Append individual asset reports and/or BF audit •Export Asset Report (After Profile) •Schedule and deploy patches via VSC
records •Export Activity log (if desired) Benefits
Benefits Benefits •Agent allows for automatic deployment, verification of
•Reporting provides real time, pre and post action asset •Reporting shows now current status (of applications) success/failure and logs all actions in audit database
and program status and cumulative patch update (Microsoft is changing •Reporting shows updated status as devices check in
•Underlying audit log provides historical reference and patching)
tracking of actions

16
What should we do?
• Properly define and capture requirements
o What is an inventory?
• IP, Mac address, OS version?
• What about software, users, configurations, registry settings, system health
• What about meta data like asset criticality, location, owner, process or function of the asset
o How about vulnerabilities?
• Is scan based enough? (Hint – it misses lots of data due to infrequency, lack of scope in OT and/or ability to
really dig deep on the OT assets)
o Patches?
• Are they vendor approved?
• Are they associated with CVEs?
o What about threat data?
o Compensating controls?
• Backup, whitelisting/AV status?
• System hardening?

17
What should we do? (Continued…)
• Design and establish an organizational structure to properly support the
convergence of IT risk in OT environments
o IT and OT representation on central analysis and decisions as well as a collaborative
effort to design the organization and governance
• Design an architecture that analyze plan and deploy risk reduction
o Don’t forget – the objective is to remove/reduce risk, not just find it!
• Design and develop specific alerting and KPIs you want to track
• Implement a global reporting and tracking ability

18
Conclusion
• OT security risk is not as hard as everyone thinks
• It is not going away – it is getting worse
• The key challenges are:
o Need context (not just data)
o Need to act on risk (not just report)
• In an OT safe way
o Need to track progress
• So the solution must:
o Be rich in information (all info about all assets plus supporting context)
o Allow you to act (not just plan A but plan B, C or even D)
o Be continuously updated

19
Thank You
Rick Kaun
[email protected]
P: 403-827-5794

www.verveindustrial.com