Imaginative, innovative, and creative.
These terms are commonly used to describe artists, architects or musicians but are not usually associated with a utilities maintenance department. But these words accurately describe the attitude Environmental Services Director Richard Hasko instilled into the Utilities Maintenance Department of Delray Beach, Fla.
"I encourage all my divisions to practice creativity because the results have such a positive impact," said Hasko. "It's a motivating factor for [the employees] and they get a sense of satisfaction from it," he said. "It has the direct result of making the division more cost-effective and more time efficient."
Bob Bullard, utilities maintenance manager, oversees maintenance for the city's water and wastewater collection departments. Bullard sustains a cost-effective, modern department by promoting Hasko's concept of an open-minded work environment and by encouraging the creative use of the city's SCADA system.
"My people are only limited by their imaginations," Bullard said. "I attribute most of the money-saving steps we've been able to accomplish to our SCADA system and the ways we've been able to use it." By encouraging innovation, Bullard is steering his department through a difficult transition that many utilities face.
As the end of the 1990s grew closer, the Delray Beach's water treatment plant and lift stations department were faced with a common problem--aging.
How could the utility take advantage of newer technology while salvaging still useful equipment?
Delray Beach installed a radio telemetry system in 1990 to monitor and control 123 wastewater pump stations and 29 remote fresh water sites. The use of radio-based remote terminal units (RTU) eliminated the reliability problems associated with telephone lines, but more importantly, the system had a modular design that allowed for occasional upgrades without having to replace the entire system. Rather than burdening the city's budget by replacing the system every 7-10 years, the utility chose a design with an RTU architecture that could last more than 20 years.
Soon after the installation of the SCADA system, Delray Beach personnel began to demonstrate their penchant for creativity. They quickly took advantage of traditional telemetry benefits: alarms warned personnel of pending problems; an overflow prevention feature practically eliminated the possibility of wastewater spills; pump data was analyzed to for preventative maintenance purposes; and more.
But they realized that there were potential system benefits that had not been exploited.
Using parts left over from a lift station rehab, Clayton Gilbert, utilities supervisor, assembled a "portable" RTU to collect more accurate data and save valuable man-hours in chlorine residual tests. At that time, the city had been installing chlorine analyzers throughout its service area to measure residual chlorine levels in the distribution system. Utility personnel would visit the site regularly during the testing period to log the readings before the analyzer was moved to a new site. Gilbert mounted an RTU and a chlorine analyzer on a hand truck, configured the unit into the telemetry system and left it at a test site for the duration of the sampling period. The data was automatically logged into the SCADA system, 24 hours a day, including weekends. The idea produced more data while netting considerable manpower savings. When the tests were complete, the unit was moved to a new test site.
The immediate cost savings was obvious and the idea ballooned. Another portable RTU was assembled to monitor conditions during construction at lift station rehab sites. Next, they mounted RTUs on portable generators so that the SCADA system could follow the mobile unit, monitoring the generator's activities, as well as the conditions at the temporary site.
Bullard explained that the system also was used to handle odor control problems. The city had been using about six control scrubbers--costing $40,000+ each--to help control odors at several key lift stations. Bullard's employees installed hydrogen sulfide monitors that were monitored through the telemetry system. The accumulated data allowed them to isolate odor problems at their source. As a result, all but one of the scrubbers was eliminated.
The SCADA system was also used to assist the water plant as well by acting as a safety device for plant personnel. The utility was concerned with the vulnerability of lone operators on night and weekend shifts. An RTU was assembled with a timer that created a telemetry alarm every 30 minutes. If the operator did not acknowledge the alarm, the SCADA system would telephone on-call personnel to let them know that the operator could be in trouble.
At that time, the WTP's treatment process ran under a separate system based on Allen-Bradley programmable logic controllers (PLC) and a QNX-based SCADA software package acting as the human-machine interface (HMI). While the system had run satisfactorily for several years, parts availability for the aging PLCs forced the city to decide whether to continue supporting two separate SCADA systems or to find a single solution for both entities.
Winds of change
Bullard assumed the reins of a Maintenance Department that took responsibility for both the lift stations and the water treatment plant. It seemed logical to use a single SCADA system for both applications. The water plant system was limited to hard-wired connections and utilized PLCs that were becoming obsolete. The lift station SCADA system was reliable and cost-effective, but the communications speed was not fast enough to manage the process controls for the water plant. Three events took place that shaped the utility's decision.
First, the SCADA manufacturer, Data Flow Systems (DFS), released the industry's first Internet browser-based SCADA HMI. With this server-based product, any client computer connected to the network could access the SCADA information with the use of a simple Internet browser. Second, DFS developed a network interface module (NIM) that could be interchanged with RTU radios. Now the utility could mix network-linked RTUs--nearly instant communication--with their traditional radio RTUs at remote locations. Third, the water plant system was designed, installed and maintained by a single, independent systems integrator. Only this individual had a full, working knowledge of the system. When his quotation to upgrade the HMI and provide a maintenance contract for the coming year was more than the cost of converting the WTP to the lift station SCADA system, the decision was made.
City personnel installed a fiber network throughout the water plant and adjacent maintenance department and to the most critical plant sites where network RTUs would replace PLCs. The SCADA system was logically partitioned so that water plant and lift stations appeared to have stand-alone systems. Previous PLC functions now would be performed by the SCADA system's new server design, the Hyper-SCADA Server (HSS).
The HSS changed the manner in which Delray Beach personnel used their telemetry system. Its computer was a printed circuit board-mounted, modular design that supported a hot-standby redundant configuration. The primary CPU took care of all polling and processing chores. The data stored on the primary unit was automatically copied to the redundant, backup CPU. The system was designed so that if the backup unit stopped receiving data from the primary CPU, the system powered down in an orderly fashion and rebooted with the backup unit assuming primary responsibilities. Each midnight, the primary and backup CPUs automatically swapped roles.
The HSS can create ladder logic-type programs to control various functions. These programs use data points throughout the system as well as "virtual points" to offer custom graphical screen functions to the user.
Bob Williamson, senior instrumentation technician, said more than 60 virtual point programs (VPP) created since the system was installed have shaved considerable sums from the department's budget. VPPs range from simple to complex:
* A ratio program for the WTP chlorinators that paces the chlorine input based on the raw water intake prevents operators from leaving their posts to manually adjust the input;
* A program that controls drain and flush valves at the WTP automates the regular flushing of two lime pots; and
* A tank fill program prevents overflows at a remote water storage facility by overriding the local controls. If the water level exceeds a certain height, the VPP locks out the fill
valve and creates a lockout alarm.
Another program protects a costly pump at a recovery basin. When the pump is supposed to run, the system watches to see if a local check valve opens. If the check valve does not open within a virtual time limit, the VPP shuts down the pump and locks out the control point.
Williamson estimated that up to eight additional personnel would be needed to perform the functions of the VPPs.
Increased security concerns also have been remedied. Three network security cameras act as sentinels at the compound gates. The cameras are connected to the SCADA system through wireless bridges and, besides keeping a record of activity at the compound entrances, offer the operators--without leaving their posts--the ability to inspect and admit visitors when the gates are locked.
The future
Williamson has more plans for the system's future. The massive Monthly Operators' Report not only takes 25 man hours to create each month, but the manual entering and copying of countless columns of numerical entries opens the door for errors. Williamson plans to use a report program that uses the same SQL database format as the SCADA system so that the relevant data that is already being collected by the SCADA system is automatically entered into the report format.
Following the goal to contain the most critical water plant process within the SCADA system, all remaining PLCs involved in process controls eventually will be replaced by RTUs with network communications. Similarly, radios in all remaining RTUs that are involved with the water plant processes also will be replaced with NIM modules. The water plant inevitably will expand and the department feels that data acquisition will play a prominent role in its effective management.
Hasko and Bullard created an environment in which productive daydreaming is encouraged. With each innovative idea that their personnel employ, budget dollars are conserved and precious manpower is freed to perform those special tasks that can only be accomplished by flesh and blood workers.
DFS President, Tom Smaidris summarized, "Some utilities use a telemetry system as nothing more than an alarm system while others utilize all or most of our system's tools. But it's very rare to find a group like the guys from Delray Beach who stretch the limits of the system and push us to develop new capabilities."
