Whilst we could argue which of the UK water companies is the largest – it’s very much dependent upon how you perform the measurement – what can be agreed upon is that companies such as United Utilities and Thames Water have large physical footprints
The number of telemetry nodes is proportional to the size of the estate; Thames Water has 100 water treatment works, 288 clean water pumping stations, 30 raw water reservoirs and 235 underground service reservoirs, 348 sewage treatment works and more than 2,530 managed sewage pumping stations, giving a count of more than 3,500 potential telemetry nodes, with United Utilities having a similar number.
Assuming the use of ADSL or 3G/4G as the communication medium and they connect via the Internet, every one of those telemetry nodes is a weak point in the security defence between cyber-criminals and the control system that supplies your drinking water and disposes of your sewage.
At a recent Institute of Water Knowledge Exchange Visit where I was presenting on Cyber Security and its importance in industry, I was asked if I thought a lack of competence on the part of the maintainer was a risk to business.
This is an awkward question that I believe industry is unwilling to answer and I certainly am not going to put my neck on the block and cite any experiences I have in this area. What I am going to do is answer a question with a question: ‘what qualifications does industry require of the people, both staff and contractors, who work on their control networks?’
Looking at the Water Industry Mechanical and Electrical Specifications (WIMES) we see that in specification 3.02(A) Profibus Networks section 2.2 specifies the contractor competence, it is reasonable to assume that the industry demands the same for its own staff who are responsible for the maintenance of the Profibus networks installed by its contractors.
There is no equivalent for the Ethernet networks which are becoming more prevalent in the industry, though some individual water companies may require the installer of a fibre-optic network to be trained to terminate and test the fibre-optic cable.
At the time of writing (1st September 2016) I noted vacancies with numerous UK water companies in the Instrumentation, Control and Automation (ICA) field that require the successful candidate to carry out defect maintenance including fault diagnosis and repair of electrical, control system and instrument equipment, and carry out ICA installation work and commissioning of plant and equipment.
These roles specified City and Guilds Electrical installation (or equivalent), as well as having passed 17th Edition electrical regulations, and made passing mention that the work involves use of SCADA and PLC control systems, IP Networking and Instrument communications equipment.
These ICA technicians will be expected to add nodes to networks, configure routers, switches and firewalls, without any requirement for formal training in this field. Compare this to adverts for network administrators in the field of IT, who will also be required to add nodes to networks, configure routers, switches and firewalls, but are required to hold certification on the hardware being used such as Cisco Certified Network Associate (CCNA) as a minimum and a degree in a computing subject as the norm.
Reviewing the specifications for network hardware from four OEMs who supply the industry – Siemens, Rockwell, Westermo and Hirschmann – all support a veritable Smörgåsbord of features and protocols some of which if not properly configured or disabled severely reduce security of the network to which they are connected.
Everyone likes a web interface for making configuring a switch or router a click-click-go operation, but unless it is disabled, or only made accessible from the LAN ports it could be visible from the Internet, which is a weakness even the most basic ‘script kiddie’ can take advantage of via the power of Metasploit.
Whilst at this point I must take pains to stress that it is not the fault of the OEM that networks are penetrated due to configuration errors, I must also take this opportunity to ask them ‘why do you allow people to change and save the configuration of a device without forcing them to change the password of the administrator account from the default?’.
Some may wonder how big the problem of leaving the default username and password enabled is, after all unless the attacker has access to the documentation and has trawled it for this information they won’t know it – the World Wide Web is a powerful tool, with websites such as SCADAStrangelove publishing password lists, documenting how to identify and exploit weaknesses and generally naming and shaming operators with poor security, the problem is greater than some may like to acknowledge.
With an installed base of circa 3,500 telemetry nodes this equates to thousands of PLC’s, HMI’s and site SCADA servers – how are all of these managed?
Normal practice for a distributed system, in the IT realm, is to use domain management, this provides user validation and verification with the added benefit of auditable logs.
In the OT realm, this is not so simple, remote sites may not be connected via ‘always on’ links such as ADSL – PSTN dial up on exception is still the only option when your site is a long way from the exchange and in the middle of no-where with no cellular signal.
This forces the responsibility for the backups, patch management, anti-virus and firmware updates onto the site operators, it also opens up the possibility that change management may not be quite how the company procedures demand it.
For the site operators who are expected to keep on top of the patch management for their automation hardware there are excellent resources available from the OEM’s, such as Siemens who publish security advisories and, failing that, the US government helpfully publish all known Industrial Control System security weakness in a central location[, which coupled with their similar repository for IT security weaknesses provide a way for companies to easily keep up-to-date with the threats to their infrastructure.
This is a double-edged sword though, as a cyber-criminal now has a single source of information on the weaknesses associated with an asset, and ICS-CERT gives an overview of how to capitalise on the vulnerability.
Also, with a massive number of installed assets, how does the industry ensure they are all supported – WindowsXP ceased to be supported by Microsoft on April 8, 2014. That is more than two years ago yet there remains a large installed asset base within the industry, predominantly in the form of HMIs; these will not have received a security update for a minimum of two years. The savvy cyber-criminal will look to capitalise on this and if it has not already happened it is just a matter of time before an industrial asset falls foul of ransomware.
Assuming the patches and firmware updates are available, the operator is faced with the challenge of applying them as the world of OT is a very different one from that of IT:
Looking below at row two – OT is event driven and real-time – taking a PLC off-line to apply a patch or firmware update has a huge overhead associated with it and has a high level of risk for the business.
For example, the latest firmware upgrades from some OEMs are one way only, that is once they have been applied should there be unforeseen issues with other elements of the system it is not possible to roll-back.
This poses a problem for the industry, no company has the ability to run redundant control systems across the board so they are forced to either work on the theory of ‘if it’s not broken, don’t fix it’ and take the risk that their control systems are potentially vulnerable as they are not patched against known vulnerabilities, or take the risk associated with applying patches to deal with the vulnerabilities and hope that they have fully worked out the compatibility matrix to ensure they will not be left with an incompatible system.
The patching approach also assumes that sites, streams or processes can be taken off line to carry out the work, though in theory the control systems are designed to be resilient and it should be possible to take a single processor off line for an hour, experience from the hot seat indicates that in a lot of instances this is a once or twice a year process at best associated with project work that takes a lot of pre-planning and is not something Process Controllers are comfortable with happening regularly. The same is the case for the operating system the SCADAs and Historians are running on, normally a Windows product.
Microsoft issue security bulletins on the second Tuesday of each month, patch Tuesday, these should then be verified as compatible with their product by the OEM’s, and the compatibility of the security and critical patches published, examples being Siemens and Rockwell Automation.
Again, the requirement for restarting the server after applying the patches impacts on plant availability and this should be scheduled in on a monthly basis, with roll back and contingency planning taking place on a case-by-case basis.
Looking again at the above table at row six it is important to note that OT is key in delivering process and plant safety, which allows me to pose my favourite question when talking to senior management ‘how can you be safe if you are not secure?’.
Compromised control systems have the potential to cause damage to plant, the environment, injure or kill, examples being the now infamous Stuxnet threat, the nearly as famous German steel mill incident, and the little known tilting oil rig episode.
Whilst none of these injured or killed anyone they all had the potential to, and although none of them is related to our industry we are not immune – in March this year a WTW was targeted and the dosing levels changed to dangerous levels, in November 2011 pumps were shutdown at a WTW, the 2000 Maroochy Shire cyber event provides a lot of lessons that should be considered and the 2013 Trend Micro research experiment findings seal the deal, we are a vulnerable target.
When discussing process safety, everyone quite rightly quotes BS EN 61508, ‘Functional Safety of Electrical/ Electronic/ Programmable Electronic Safety-related Systems’. What people tend to miss is that this standard was updated in 2010 to consider the prevalence of networked safety systems. As a result, Cyber Security is now included in the revised standard and requires that in the case where the hazard analysis identifies that malevolent or unauthorized action, constituting a security threat, is reasonably foreseeable, a security threat analysis should be carried out.
Even when a security threat assessment is carried out, what tends to be overlooked is considering this in the safety lifecycle and as a result these one-time assessments become irrelevant as technology, vulnerabilities and threats move on.
The headline of this piece is ‘The Opportunities, Pitfalls and Dangers that the Water Industry Faces in Terms of Cyber Security’, so having covered a number of pitfalls and dangers we are left looking for the opportunities, which in reality are many
Unless you are passionate about the subject the adoption of ‘The Directive on security of network and information systems’ (the NIS Directive) by the European Parliament on 6 July 2016 may have passed you by. However, it has major implications on the way the industry, and its suppliers, will operate when it is passed into UK law, which will happen before May 2018.
This legislation requires operators of Critical National Infrastructure (CNI) to report all cyber security breaches to the UK Computer Security Incident Response Team (CSIRT), this will require all networks, both IT and OT, to implement intruder detection amongst other technologies which are still in their infancy. This will be an expensive and time consuming challenge and one that will push the boundaries of the capabilities of the people currently employed within the industry – it’s just a fact of life that this is new technology and the number of qualified people are limited. Early adopters and companies that embrace the challenge of raising the standard of cyber security within the industry will benefit but it is highly likely that it will take a full blown incident that compromises the supply of wholesome water or results in damage to the environment before the requisite level of investment is reached.
Having previously demonstrated live on stage how easy it is to compromise the security of an industrial control system, I know that the UK water industry is vulnerable and the attendees of these demonstrations also know this to be true. When opening the floor up for questions during my presentations one I am always asked is ‘what can we do?’ it sounds like a cliché but the help is there for you – Centre for the Protection of National Infrastructure (CPNI) has a section on its website specifically focusing on Security for Industrial Control Systems.
There are a number of companies who specialise in this area, OEMs such as Siemens and Rockwell have departments and personnel who will advise on how to secure industrial networks. The one take away tip I always give is employee engagement:
• teach your employees e-mail attachment security so they don’t fall for the phishing scams – I have conducted a scam for a PhD thesis, you will be amazed at the naivety of people
• explain to them why the policy is to disable the USB ports so they don’t find a way to enable them and open the network up to the threat from auto running USB files – an airgap is no protection when the USB ports are enabled with malware like AirHopper
• make it easy for your site staff to purchase new hardware so they don’t go to PCWorld and install an unmanaged home grade device on the industrial network – I have many photos of this, and there does seem to be a love of wireless when installing rogue devices
• invest in training staff to increase their competency in the world of network management, even the basics – if I had a pound for the number of times I have found duplicate IP addresses on a network I would be a rich man.
www.z-tech.co.uk
