California water district enhances communications system to navigate difficult terrain
Located on the southwestern edge of the Mojave Desert about 85 miles northeast of Los Angeles lies the city of Victorville, Calif. Summer temperatures in Victorville often reach triple digits, and its annual precipitation is scantly more than 6 in. It is imperative, therefore, that the city has a reliable SCADA system in place to help manage and ensure water availability to the area’s 100,000 residents.
Victorville Water District (VWD) was established in 1931 and has grown substantially to consist of 80-plus sites, including wells, booster pump stations, 70 million gal of storage tanks and four water treatment plants. The topography of the area made communications difficult, as many of the wells and stations are located among rolling hills and crevices covering approximately 54 sq miles.
Jeff Zizzi joined VWD in 1995 as SCADA and telemetry coordinator. He was tasked with upgrading an aging infrastructure and, more specifically, the communications system.
Coming Up With a Plan
When Zizzi began evaluating the existing communications system, he found modem serial technology that took up to 10 minutes to contact and poll each of the district’s sites. Crosstalk and interference from the 50-plus other repeaters in the immediate area (used by TV stations and telephone companies) often made it impossible to get any information from the remote sites. Weather was another factor, causing power outages and rendering the communications system useless at times.
Over the next few years, Zizzi began replacing key components of the communications system. He increased the baud rate to 9,600 by replacing the existing PLCs and modems with the SCADAPack Micro16s, which reduced the system poll time down to three minutes. Then he installed Lookout HMI and decreased the polling time to less than two minutes. With the addition of iNET 900 spread-spectrum radios, the overall baud rate jumped to 256,000. By replacing the SCADAPack Micro16s with SCADAPack 32s, Zizzi successfully migrated the entire system to an IP basis.
Problems began to mount for this system, however, as the repeater continued to be plagued with power outages and interferences from the neighboring sites. The city also began to face issues with IP addresses. In some cases, it had as many as five IP addresses at a site: the radio, remote telemetry unit, variable-frequency drive, power monitors and local site computers. Then the city discovered that its centralized control system was prone to failure if the main HMI control were to go down or if the repeater was disabled.
Collectively, this posed challenges for the well sites, as they were unable to communicate directly with the tank sites in order to know when to operate. But the issue paining Victorville the most was that the company that installed the current system was based in Chicago; if they had to send out a service person for any reason, it ended up costing the district $2,500 or more. Zizzi knew things had to change.
He decided that the single repeater system was “a thing of the past” and that the city needed to look to the future and find a better solution. He considered several approaches, from a multi-repeater system to a peer-to-peer store and forward system utilizing report-by-exception.
New Communications Infrastructure
After months of research, Zizzi came across a new type of communications infrastructure that seemed an ideal fit for the district: a mesh network. He immediately prepared a detailed list of requirements covering the district’s needs and addressing the unique challenges of the system. He then submitted them to several vendors for bids.
The main requirements included maximum throughput, ease of installation and low maintenance requirements. The request for bids also stipulated that the vendor provide a minimum warranty period, technical support for the term of the installation period plus one year, and ongoing product and technical support. Most importantly, Zizzi wanted a highly reliable system, with the ability to “heal” itself quickly and without packet if a single system or multiple systems were disabled. These features would decentralize the system, allowing wells to run autonomously even if the central control became disabled.
Ultimately, Zizzi decided to go with Firetide, a vendor known for deploying wireless mesh infrastructure networks in challenging terrains.
To date, the backbone of the system has been completed and has provided the district with unexpected benefits in addition to those anticipated. Due to the flexibility and high-speed performance of the mesh, the district has the ability to connect IP cameras to the network.
Network performance has exceeded expectations, with throughput speeds exceeding 7Mbps and some sites achieving speeds in excess of 200Mbps. Additionally, Victorville has reduced IP addresses by more than 40% and now has a foundation in place that will support the addition of more video security cameras and integration with its automatic water meter reading system.