Down Stream Manufacturing: Global Asset Mananagement System Deloyment

In 2012 a global program was initiated across all lines of business to bring compliance to a unified standard. 20 refineries were targeted across the world.
The focus of the program was on endpoint asset management using a custom in-house software toolset and redesigned security architecture that would collect all endpoint security data and allow global analysis of threats, compliance and security status across all operating assets. Starting in 2014 my role was to support the downstream manufacturing line of business on the global remediation program across all regions to ensure endpoint hardening and compliance, complete installation of the technology stack, and 100% compliance with 3rd party vendor systems at each operating asset.

Accurate security status of all endpoints in the OT environment is critical to resilient operations in downstream manufacturing. This program addressed those requirements using a combination of software tools used to manage each endpoint, report status and allow incident response to take place due to rapid security reporting and alerting. A second component was major network redesign to allow for a better segmented and more secure network environment, and lastly, gain full visibility into all 3rd party systems supporting the downstream operating assets.

Work Stream A: Asset Discovery
The first step was to discover what devices and systems were operating at all assets.

Work Stream B: Criticality Assessment
This workstream focused on evaluating the function of the endpoint in question, understanding its communication patterns and assigning a criticality rating to the device which would allow remediation efforts to be prioritized.

Work Stream C: Patching
Once function and criticality were established, the goal of this workstream was to patch the endpoint to an appropriate level based on manufacturers’ recommendations, and the security philosophy of the client.

Work Stream D: Hardening
All non-essential services were shutdown, unneeded communication ports closed, unnecessary applications removed and the endpoints were whitelisted as appropriate so that only essential software could operate on the device. General USB device use was blocked and a limited allow list of portable USB devices was enabled. Additionally, all local and global accounts were reviewed. Admin accounts were limited or deleted if unneeded, and default accounts were renamed or disabled, and a least privileged user philosophy was implemented to control account creation and management going forward.

Work Stream E: Endpoint Management
Each endpoint was integrated into the asset management system so that relevant hardware and software data could be routinely polled by a central collector which would forward data to the organizations security analysis team.

Work Stream F: PCN Network Re-architecture
Client owned L2 and L3 networks were rearchitected to the new reference architecture design, and all system to system network traffic was cutover to the new perimeter firewall and network monitoring systems.All firewall rules and access control lists were analyzed to determine the most restrictive model of network traffic.

Work Stream G: 3rd party Integration
Resulting from Workstream A, all identified assets had been split into internal organization or 3rd party ownership. In workstream F each 3rd party endpoint was integrated into the central management tool in the same manner as the internal client owned systems. Additionally, limited network redesign was implemented to allow secure communication between the 3rd party systems and the internal client endpoints.

Work Stream A: Asset Discovery
• Cause: Lack of a centralized asset management system.
• Effect: Inconsistent security posture from operating asset to operating asset, particularly by region.
• Solution: Dedicated team went from asset to asset collecting 100% asset inventory of all internal and 3rd party systems.
• Outcome: Global visibility into the current security posture of all operating assets.

Work Stream B: Criticality Assessment
• Cause: Lack of understanding of what the most important systems at any given asset were and how they should be secured.
• Effect: Inability to react effectively in the event of a security incident. Inability to be proactive and be aware of potential risks before the incident.
• Solution: Deep dive knowledge transfer with control system owners allowed the team to document and assign a criticality rating.
• Outcome: Understanding the criticality and relationships of all systems at the site allowed accelerated security remediation efforts. Most critical secured first, and most critical actively monitored first, as well as understanding how each system was interconnected, and thus vulnerable if one or more complimentary systems were affected by a security incident.


Work Stream C: Patching
• Cause: Lack of standards or guidance on patching operating systems meant most sites were not patched regularly or consistently.
• Effect: Inconsistent levels of security across the organization and lack of visibility meant the organization was unable to demonstrate any level of security consistency and had no knowledge of potential security risk.
• Solution: A large-scale global patching program was implemented to bring all sites to the same level of operating system patch status.
• Outcome: The organization had a clean and effective baseline to assess security posture for each asset going forward.

Work Stream D: Hardening
• Cause: Lack of planning during various deployments and lack of hardening philosophy during operation meant most systems were not hardened at all and had excessive attack surface exposed.
• Effect: Exposed attack surface could allow malicious actors to quickly gain control of various systems in the PCN.
• Solution: Turning off unneeded services and ports, restricting USB devices and removing unnecessary software, as well as limiting admin accounts and disabling or renaming default accounts.
• Outcome: The attack surface was greatly reduced on each endpoint, making an attackers job much more difficult.

Work Stream E: Endpoint Management
• Cause: No centralized, unified system was managing endpoints across the global organization, and most sites had inconsistent management within their respective operating areas.
• Effect: No ability to manage the lifecycle of the endpoint from a technology, security or cost perspective.
• Solution: Deploy a centralized management system that routinely communicated with each endpoint.
• Outcome: The central security team was able to rapidly analyze and react to potential security incidents, and were able to be extremely proactive on patching, system maintenance and health status reporting.

Work Stream F: PCN Network Re-architecture
• Cause: Legacy networks and organic growth were not configured in accordance with modern network architectural designs or standards like IEC62443-3/ISA-99.
• Effect: Lack of a defense in depth/3 tier architecture at most operating assets.
• Solution: Redesign all operating assets networks to match the Reference Architecture created to support a robust network design.
• Outcome: All firewall rules and access control lists were rationalized, unnecessary network traffic eliminated and modern hardware was installed to handle network intrusion detection, alerting and a 3 tier network design implemented that allowed a defense in depth perimeter defense.

Work Stream G: 3rd party Integration
• Cause: A critical lack of visibility into the systems and data being used by 3rd party vendors had left the organization extremely vulnerable to security incidents.
• Effect: Zero visibility into 3rd party systems and connections into the primary client owned PCN meant that there was zero chance to prevent or react to a security incident, which would inevitably affect the primary client control system as well.
• Solution: A full walkdown of each site was conducted to verify all 3rd party windows and network systems, and all communications into and out of the site and the primary client networks. Effectively, workstreams A through F were conducted for each 3rd party vendor at all client sites.
• Outcome: For the first time, the downstream manufacturing organization had full visibility into all systems deployed and operating at each asset, which allowed the central security team to proactively monitor and support each vendor. This meant the single largest security risk to the organization had been finally managed.

USA
Argentina
Germany
Norway
Netherlands
Denmark
Singapore