Jan 22, 2004

Pushing the Envelope

Delray Beach, Fla., Utilities Maintenance Department creatively maximizes the capabilities of its SCADA system

Imaginative, innovative, and creative.

These terms are commonly used to describe artists, architects
or musicians but are not usually associated with a utilities maintenance
department. But these words accurately describe the attitude Environmental
Services Director Richard Hasko instilled into the Utilities Maintenance
Department of Delray Beach, Fla.

"I encourage all my divisions to practice creativity
because the results have such a positive impact," said Hasko. "It's a
motivating factor for [the employees] and they get a sense of satisfaction from
it," he said. "It has the direct result of making the division more
cost-effective and more time efficient."

Bob Bullard, utilities maintenance manager, oversees
maintenance for the city's water and wastewater collection departments. Bullard
sustains a cost-effective, modern department by promoting Hasko's concept of an
open-minded work environment and by encouraging the creative use of the city's
SCADA system.

"My people are only limited by their
imaginations," Bullard said. "I attribute most of the money-saving
steps we've been able to accomplish to our SCADA system and the ways we've been
able to use it." By encouraging innovation, Bullard is steering his
department through a difficult transition that many utilities face.

As the end of the 1990s grew closer, the Delray Beach's
water treatment plant and lift stations department were faced with a common

How could the utility take advantage of newer technology
while salvaging still useful equipment?

Delray Beach installed a radio telemetry system in 1990 to
monitor and control 123 wastewater pump stations and 29 remote fresh water
sites. The use of radio-based remote terminal units (RTU) eliminated the
reliability problems associated with telephone lines, but more importantly, the
system had a modular design that allowed for occasional upgrades without having
to replace the entire system. Rather than burdening the city's budget by
replacing the system every 7-10 years, the utility chose a design with an RTU
architecture that could last more than 20 years.

Soon after the installation of the SCADA system, Delray
Beach personnel began to demonstrate their penchant for creativity. They
quickly took advantage of traditional telemetry benefits: alarms warned
personnel of pending problems; an overflow prevention feature practically
eliminated the possibility of wastewater spills; pump data was analyzed to for
preventative maintenance purposes; and more.

But they realized that there were potential system benefits
that had not been exploited.

Using parts left over from a lift station rehab, Clayton
Gilbert, utilities supervisor, assembled a "portable" RTU to collect
more accurate data and save valuable man-hours in chlorine residual tests. At
that time, the city had been installing chlorine analyzers throughout its
service area to measure residual chlorine levels in the distribution system.
Utility personnel would visit the site regularly during the testing period to
log the readings before the analyzer was moved to a new site. Gilbert mounted
an RTU and a chlorine analyzer on a hand truck, configured the unit into the
telemetry system and left it at a test site for the duration of the sampling
period. The data was automatically logged into the SCADA system, 24 hours a
day, including weekends. The idea produced more data while netting considerable
manpower savings. When the tests were complete, the unit was moved to a new
test site.

The immediate cost savings was obvious and the idea
ballooned. Another portable RTU was assembled to monitor conditions during
construction at lift station rehab sites. Next, they mounted RTUs on portable
generators so that the SCADA system could follow the mobile unit, monitoring
the generator's activities, as well as the conditions at the temporary site.

Bullard explained that the system also was used to handle
odor control problems. The city had been using about six control
scrubbers--costing $40,000+ each--to help control odors at several key lift
stations. Bullard's employees installed hydrogen sulfide monitors that were
monitored through the telemetry system. The accumulated data allowed them to
isolate odor problems at their source. As a result, all but one of the
scrubbers was eliminated.

The SCADA system was also used to assist the water plant as
well by acting as a safety device for plant personnel. The utility was concerned
with the vulnerability of lone operators on night and weekend shifts. An RTU
was assembled with a timer that created a telemetry alarm every 30 minutes. If
the operator did not acknowledge the alarm, the SCADA system would telephone
on-call personnel to let them know that the operator could be in trouble.

At that time, the WTP's treatment process ran under a
separate system based on Allen-Bradley programmable logic controllers (PLC) and
a QNX-based SCADA software package acting as the human-machine interface (HMI).
While the system had run satisfactorily for several years, parts availability
for the aging PLCs forced the city to decide whether to continue supporting two
separate SCADA systems or to find a single solution for both entities.

Winds of change

Bullard assumed the reins of a Maintenance Department that
took responsibility for both the lift stations and the water treatment plant.
It seemed logical to use a single SCADA system for both applications. The water
plant system was limited to hard-wired connections and utilized PLCs that were
becoming obsolete. The lift station SCADA system was reliable and
cost-effective, but the communications speed was not fast enough to manage the
process controls for the water plant. Three events took place that shaped the
utility's decision.

First, the SCADA manufacturer, Data Flow Systems (DFS),
released the industry's first Internet browser-based SCADA HMI. With this
server-based product, any client computer connected to the network could access
the SCADA information with the use of a simple Internet browser. Second, DFS
developed a network interface module (NIM) that could be interchanged with RTU
radios. Now the utility could mix network-linked RTUs--nearly instant
communication--with their traditional radio RTUs at remote locations. Third,
the water plant system was designed, installed and maintained by a single,
independent systems integrator. Only this individual had a full, working
knowledge of the system. When his quotation to upgrade the HMI and provide a
maintenance contract for the coming year was more than the cost of converting
the WTP to the lift station SCADA system, the decision was made.

City personnel installed a fiber network throughout the
water plant and adjacent maintenance department and to the most critical plant
sites where network RTUs would replace PLCs. The SCADA system was logically
partitioned so that water plant and lift stations appeared to have stand-alone
systems. Previous PLC functions now would be performed by the SCADA system's
new server design, the Hyper-SCADA Server (HSS).

The HSS changed the manner in which Delray Beach personnel
used their telemetry system. Its computer was a printed circuit board-mounted,
modular design that supported a hot-standby redundant configuration. The
primary CPU took care of all polling and processing chores. The data stored on
the primary unit was automatically copied to the redundant, backup CPU.  The system was designed so that if the
backup unit stopped receiving data from the primary CPU, the system powered
down in an orderly fashion and rebooted with the backup unit assuming primary
responsibilities. Each midnight, the primary and backup CPUs automatically
swapped roles.

The HSS can create ladder logic-type programs to control
various functions. These programs use data points throughout the system as well
as "virtual points" to offer custom graphical screen functions to the

Bob Williamson, senior instrumentation technician, said more
than 60 virtual point programs (VPP) created since the system was installed
have shaved considerable sums from the department's budget. VPPs range from
simple to complex:

*                A
ratio program for the WTP chlorinators that paces the chlorine input based on
the raw water intake prevents operators from leaving their posts to manually
adjust the input;

*                A
program that controls drain and flush valves at the WTP automates the regular
flushing of two lime pots; and

*                A
tank fill program prevents overflows at a remote water storage facility by
overriding the local controls. If the water level exceeds a certain height, the
VPP locks out the fill

valve and creates a lockout alarm.

Another program protects a costly pump at a recovery basin.
When the pump is supposed to run, the system watches to see if a local check
valve opens. If the check valve does not open within a virtual time limit, the
VPP shuts down the pump and locks out the control point.

Williamson estimated that up to eight additional personnel
would be needed to perform the functions of the VPPs.

Increased security concerns also have been remedied. Three
network security cameras act as sentinels at the compound gates. The cameras
are connected to the SCADA system through wireless bridges and, besides keeping
a record of activity at the compound entrances, offer the operators--without
leaving their posts--the ability to inspect and admit visitors when the gates
are locked.

The future

Williamson has more plans for the system's future. The massive
Monthly Operators' Report not only takes 25 man hours to create each month, but
the manual entering and copying of countless columns of numerical entries opens
the door for errors. Williamson plans to use a report program that uses the
same SQL database format as the SCADA system so that the relevant data that is
already being collected by the SCADA system is automatically entered into the
report format.

Following the goal to contain the most critical water plant
process within the SCADA system, all remaining PLCs involved in process
controls eventually will be replaced by RTUs with network communications.
Similarly, radios in all remaining RTUs that are involved with the water plant
processes also will be replaced with NIM modules. The water plant inevitably
will expand and the department feels that data acquisition will play a
prominent role in its effective management.

Hasko and Bullard created an environment in which productive
daydreaming is encouraged. With each innovative idea that their personnel
employ, budget dollars are conserved and precious manpower is freed to perform
those special tasks that can only be accomplished by flesh and blood workers.

DFS President, Tom Smaidris summarized, "Some utilities
use a telemetry system as nothing more than an alarm system while others
utilize all or most of our system's tools. But it's very rare to find a group
like the guys from Delray Beach who stretch the limits of the system and push
us to develop new capabilities."  

About the author

Steve Whitlock is vice president of customer relations at Data Flow Systems, Inc. For further information, phone 321/259-5009.