An outline of steps to perform when upgrading a SCADA system
Upgrading a SCADA system in a large water/wastewater treatment plant can be daunting. There are a lot of tanks, pumps and systems in a plant, and trying to upgrade it all in one stage can seem overwhelming.
Many of these plants’ control systems were installed in the 1980s and 1990s and slightly upgraded around 2000. Since then, modern technology has made great strides. Computers are faster, graphics are better and no longer vector based, communication protocols—such as Ethernet—are faster, and there is less dependence on proprietary hardware.
While the benefits to an upgrade are obvious, the decision to do it is not as straightforward. In addition to a large scope, an upgrade involves training operators, testing and debugging the system, and keeping the plant operational. One way to get around this is to perform the upgrades incrementally. Upgrading one control area at a time takes longer, but minimizes risks and allows operators to gradually become familiar with the new graphics, alarm screens and trends.
By the time the final process area has been converted, the benefits have been realized through removal of outdated and addition of improved information. Operators gain access to new information that they have been requesting but that could never seem to be implemented because of the system’s age.
Running in Parallel
In addition to upgrading process areas one at a time, in most instances, both the new and old systems can be run in parallel to ensure the equipment is represented and operating properly on the new system. Operators have enough time to learn the new system without the risk of making a critical error, and integrators can debug the system without having to shut down operations.
Optimation, a certified member of the Control System Integrators Assn., recently completed a multi-year SCADA upgrade at a water authority that operates two separate wastewater treatment plants. One plant has 14 process areas, each with its own redundant distributed control system controllers. The other plant has six process areas, each with its own redundant programmable logic controllers from a different vendor.
The larger plant was completed first, one process area at a time. As the project team became more familiar with the conversion process and the operators more familiar with the new look and feel, a couple of smaller process areas were combined and implemented, one right after the other.
How to Perform an Incremental Upgrade
It is recommended that operators get involved early in the project. Operators work with the system each day and know how a design actually works when it is put into process.
Flexible Implementation Plan
- • Initial planning: Determine which systems will be upgraded, in what order and why. Think through how systems impact other systems.
- • Create a testing plan.
- • Shift coverage: Oftentimes, during low-flow conditions in the evenings, it is easier to make changes. Because night shifts do not typically have as many people around, plants may need to add personnel.
- • Training plan: All shifts and the maintenance crew need to learn about system changes and new information.
- • Seasonal impact: When it comes time to implement some changes, plants need to consider how weather will affect the process. Depending on the geography, there could be rainy seasons, dry seasons or snow. Some aspects of the upgrade may need to be deferred to a certain time of the year. Activated sludge processes, for example, are biologically controlled. High water and cold weather can have major impacts, and it is difficult to make a change without upsetting the plant. Storms also should be considered. If a storm passes through, it could throw plans off for a few weeks. The project plan needs to be flexible.
- • Coordination with other ongoing or future projects: Oftentimes there are projects that are seemingly unrelated, but could run into each other and negatively impact progress.
- • Update IT structure, security, credentials and authorization levels: Some of these may not have been changed for years. Today, there are new standards, methods and philosophies surrounding things like access.
- • Specification: Specify hardware and software.
- • SCADA design standards: Determine colors, shapes, alarms, trends, historian and navigation. If you do not have design standards, define them with the integrator before starting the project.
- • I/O tag list: Consider what is needed in the new system versus what is available in the old system. Not every existing tag is required. Not every original instrument or device is still installed.
The original design of these plants may have simulated plant operations, but never actually ran that way, even though this was how the process control/SCADA system was designed. Even though it is available and always has been does not mean that you will need to use it, especially if the plant was never actually run that way. Duplicating documentation for system design and hardware that was never there in the first place is not recommended. These distinctions should be included in the design; otherwise, you risk wasting time and money on extra work that may need to be undone.
- • Development hardware: Purchase development hardware that is similar to actual hardware.
- • Development software license: Remember to purchase one for development and for the plant.
- • I/O tag review meetings: Get operator buy-in, just to make sure that what the team is saying and what it is developing actually exist, in addition to looking and feeling the way they thought it would.
- • Graphics development meetings: You can talk all you want about colors and shapes, but operators might see something different than what they had in mind. Review graphics during development rather than at the end to avoid surprises.
Testing & Training
- • Offsite testing: Perform offsite tests of graphics, navigation, etc.
- • Onsite installation and testing with plant operators: If operators are involved in development, this step will go more smoothly. Operators know how the plant works, so they can determine whether the new system is working the way it should.
- • Parallel operation with the old system: This should run for a defined period, and it usually takes a couple of weeks. It can be beneficial not to completely rely on the new system, because there likely will be bugs that need to be worked out, and an upset can have a major impact.
- • Keep the old system: Even after turning off the old system, maintain access in the event something was missed. This is especially true for weather events that are difficult to test for.
Keys to Success
Take the lessons learned from each system upgrade and repeat for subsequent systems, building on them for the next increment.
Remember to create a sense of collaboration, not confrontation, with the systems integrator. Incremental upgrades can take longer, so seasonal events and unforeseen issues may arise as the project progresses. Everyone needs to be flexible and work together to overcome these challenges.
It is important to plan for remote access to the plant system through VPN or other secure means. This is especially true for those “things that go bump” in the night and on the weekend. It allows the systems integrator to get online quickly, rather than having to drive to the site before he/she can respond, saving everyone headaches and time. Plants may be apprehensive to allow such access because of concerns about cyber terrorism, but it can be done securely, and is a major key to success and more efficient than having the technicians on site when it is not necessary. Your integrator should be able to establish secure VPNs through its own system.