Loading...
HomeMy WebLinkAboutItem HAGENDA STAFF REPORT DATE: May 22, 2017 TO: Honorable Mayor and City Council THRU: Jill Ingram, City Manager FROM: Steve Myrter, Director of Public Works SUBJECT: REVISION TO RESOLUTION NO. 6708 AND RECEIVE AND FILE AKM AUDIT FOR WEST END PUMP STATION EMERGENCY PROJECT - EM1701 SUMMARY OF REQUEST: That the City Council: 1. Adopt Resolution No. 6735 revising Resolution No. 6708, reallocating approved funding for the West End Pump Station Emergency Project No. EM1701; and 2. Receive and File the West End Pump Station 2017 Audit by AKM Consulting Engineers. BACKGROUND /ANALYSIS: The City's West -End Storm Water Pump Station (WEPS) receives surface water run -off from portions of Old Town, Bridgeport and the Hill which is then pumped into the San Gabriel River. This pump station is critical to preventing storm flooding within these neighborhoods. It contains two 36 -inch propeller pumps, complete with 350 HP, 500 RPM motors, which are used to pump water out of the station's forebay and into the San Gabriel River. At the end of December 2016 during a rain event, one of the pumps registered a failure notification. Staff responded and began the process to identify and repair the problems with the pump with the assistance of pump repair contractors. During the early part of January 2017, while still working on the first pump, the second pump at WEPS registered a failure notification as well. Due to both pumps not working at the time and with rain forecasted, on January 17, 2017 the City Council adopted Resolution No. 6708 which contained the following provisions: Agenda Item H im- 1. Declaring an Emergency Proclamation for the West End Pump Station — Project No. EM1701 and dispense with public bidding for this project; 2. Approve Budget Amendment No. 17 -07 -02 in the amount of $200,000; 3. Approve a Purchase Order to Flo - Services, Inc. in amount of $150,000; 4. Approve a Purchase Order to Rain for Rent in the amount of $10,000; and 5. Authorize the City Manager to execute all necessary contracts. At the time the purchase orders were approved, the predicted storms had not yet occurred. As the storm approached, additional pumps were ordered that were above the initially estimated $10,000 to Rain for Rent. Other purchases and rentals were also made that were not fully anticipated and that went above the remaining contingency funds set aside in this budget amendment. The full breakdown of money spent/encumbered for this emergency project is as follows: Vendor /Contractor Description Amount Rain For Rent Emergency Pump DV150, two Emergency Pumps DV200's, additional Emergency Pump DV200, three additional Emergency Pump DV150's, Hoses,;connectors and adaptors and Technician /mechanic for weekend $ 36,957.49 Flo- Services Maintenance of pumps on -site, removal and repair of pumps and return and install pumps $ 115,077.77 Wood Oil Company Diesel Fuel for Emergency Pumps $ 1,063.65 Bay Hardware Supplies $ 73.67 Industrial Technical Services Toshiba Representative $ 1,990.00 AKM WEPS audit $ 35,000.00 Lee & Ro Inc. WEPS audit - 3rd party Review $ 7,550.00 Misc /Other Misc. expenditures (parts, supplies, food, etc.) $ 793.93 West Marine Pro Inflatable boat $ 1,183.11 Total $ 199,689.12 The total cost incurred for Flo- Services is less than the approved Purchase Order of $150,000 because required repairs of the pumps were less than anticipated. The total cost incurred for Rain for Rent is above the approved Purchase Order due to the need for additional emergency rental pumps. Even with these modifications, total project funding required will be less than the $200,000 approved at the January 17, 2017 City Council meeting. Page 3 At the January 23, 2017 City Council meeting, staff informed the City Council that the approved contingency funding would be used to hire AKM Consulting Engineers to perform a comprehensive audit of West End Pump Station. This study has been completed and is included as an attachment to this report. The major items that are considered in the audit are: • History of West End Pump Station. This includes discussion on the original construction of the station and the interim improvements in 2007. • Chronology of Events. This is a detailed breakdown of the events that took place and the City's responses between December 21, 2016 and February 20, 2017. Included in the discussion of the events are: • Pump 1 failure to operate in December 2016 • Pump 2 failure to operate in early January 2017 • Field inspections performed by staff, consultants and pump maintenance contractors • Emergency City Council Authorization to remove and repair pumps • Pump repairs at manufacturer with detailed reports • Reinstallation, startup and systems testing • Pump 1 failure to operate in February 2017 with corresponding field inspection and repair. • Equipment Evaluation. This section conducts a full evaluation of all equipment at the West End Pump Station, their use, their settings and operations and a list of recommendations for future modifications and settings that are intended for increased efficiency of the pump station. The equipment evaluated in this section includes: o West End Pump Station Watershed and inlet storm drain system o Cascade 48 AP single stage propeller pumps 0 350 Horsepower Motors o Toshiba Variable Frequency Drives o Murcal 555 PLC Control Systems 0 350 kW Caterpillar Emergency Generator o West End Pump Station Trash Rack • Maintenance Program. The audit details the City's past maintenance activities, its current maintenance plan, and recommendations for future maintenance and training activities designed to improve operator efficiency at the West End Pump Station. • Responses to Community Comments and Questions. The final section of the Audit focuses on answering the questions and comments received from the public in relation to the events and maintenance activities that lead up to both pumps becoming non - operational in late 2016 into early 2017. Page 4 In addition to the Audit conducted by AKM Consulting Engineers, staff also indicated to the City Council at the January 23, 2017 City Council meeting that there would be a 3`d party verification (peer review) of the Audit conducted by a separate consulting firm that also specializes in storm water flow and pump station design. The City hired Lee and Ro. Inc. to perform this audit. Their review and comments /conclusions to the AKM Audit are attached to this staff report. ENVIRONMENTAL IMPACT: This is not a project within the California Environmental Quality Act. LEGAL ANALYSIS: The City Attorney has reviewed and approved as to form. FINANCIAL IMPACT: All charges related to the West End Pump Station Emergency Project No. EM 1701 were funded by the City's General Fund. On January 17, 2017 the City Council approved Budget Amendment 17 -07 -02 which allocated $200,000 for the Project from the General Funds. No additional funds are required to cover the final costs of the project. RECOMMENDATION: That the City Council: 1. Adopt Resolution No. 6735 revising Resolution No. 6708, reallocating approved funding for the West End Pump Station Emergency Project No. EM1701; and 2. Receive and File the West End Pump Station 2017 Audit by AKM Consulting Engineers. SUBMITTED BY: NOTED AND APPROVED: Steve M Jill . Ingram, City " ager Director of Public Works Page 5 Prepared By: David Spitz, P.E. Associate Engineer Attachments: A. Resolution No. 6735, A Resolution of the Seal Beach City Council Revising Resolution No. 6708 for West End Pump Station Emergency Project — EM1701 B. Resolution No. 6708, A Resolution of the Seal Beach City Council Declaring an Emergency Proclamation for the West End Pump Station— Project No. EM1701 C. AKM Audit D. Lee and Ro. Inc. letter "Attachment A" RESOLUTION NUMBER 6735 A RESOLUTION OF THE SEAL BEACH CITY COUNCIL REVISING RESOLUTION NO. 6708 FOR WEST END PUMP STATION EMERGENCY PROJECT — EM1701 WHEREAS, on Friday, January 6, 2017 City staff determined that both West End Pumps failed diagnostics with no ability to make immediate repairs, leaving the station with no remaining operational large capacity storm drain pumps; and WHEREAS, with no operational large capacity storm drain pumps in operation, the City was in danger of severe flooding in parts of Old Town and Bridgeport in the event of a moderate to severe rain event; and WHEREAS, on Tuesday, January 17, 2017, pursuant to Resolution No. 6708, the City Council issued an Emergency Proclamation for the West End Pump Station — Project No. EM1701 enacting City Charter Section 107 for repairs for the preservation of life, health or property; and WHEREAS, pursuant to Resolution No. 6708, the City Council approved Budget Amendment No. BA17 -07 -02 modifying the Fiscal Year 2016/2017 budget to provide a total of $200,000 in funding for said project; and WHEREAS, pursuant to Resolution No. 6708, the City Council approved Purchase Orders in the amounts of $150,000 to Flo- Services and $10,000 to Rain for Rent, and allocated the remaining $40,000 for contingencies for Project No. EM1701; and WHEREAS, following the City's Emergency Proclamation, the City was required to obtain additional unanticipated pumps from Rain for Rent to prepare for upcoming storms in the City, and also incurred additional unanticipated purchases and repairs that exceeded the remaining contingency funds set aside in the budget amendment for the West End Pumps repairs, while the total cost incurred for pump repair services from Flo- Services was less than the amount of $150,000 previously allocated; and WHEREAS, the total project funding required for the entire project based on actual repairs, maintenance and requirements, will be less than the $200,000 approved pursuant to the Emergency Proclamation; and WHEREAS, the City Council finds it necessary to revise Resolution No. 6708 to reflect the actual repairs and purchases necessary to complete the project. THE SEAL BEACH CITY COUNCIL DOES HEREBY RESOLVE: SECTION 1. Sections 4, 5 and 6 of Resolution No. 6708 are hereby revised in their entirety as follows: A. Approve a Purchase Order to Flo- Services, Inc., in the amount of $115,077.77. B. Approve a Purchase Order to Rain for Rent in the amount of $36,957.49. C. Allocate the remaining sum of $47,964.74 for contingencies for Project No. EM1701. SECTION 2. All other provisions of Resolution No. 6708 shall remain in full force and effect. PASSED, APPROVED and ADOPTED by the Seal Beach City Council at a meeting held on the 22nd day of May , 2017 by the following vote: AYES: Council Members: NOES: Council Members: ABSENT: Council Members: ABSTAIN: Council Members: Sandra Massa - Lavitt, Mayor ATTEST: Robin L. Roberts, City Clerk STATE OF CALIFORNIA } COUNTY OF ORANGE } SS CITY OF SEAL BEACH } I, Robin L. Roberts, City Clerk of the City of Seal Beach, do hereby certify that the foregoing resolution is the original copy of Resolution Number 6735 on file in the office of the City Clerk, passed, approved, and adopted by the Seal Beach City Council at a special meeting held on the 22nd day of May 2017. Robin L. Roberts, City Clerk I RESOLUTION NUMBER 6708 A RESOLUTION OF THE SEAL BEACH CITY COUNCIL DECLARING AN EMERGENCY PROCLAMATION FOR THE WEST END PUMP STATION — PROJECT No. EM1701 WHEREAS, on Friday, January 6, 2017 staff determined that both West End Pumps failed diagnostics with no ability to make immediate repairs, leaving the station with no remaining operational large capacity storm drain pumps; and WHEREAS, With no operational large capacity storm drain pumps in operation, the city is in danger of severe flooding in parts of Old Town and Bridgeport in the event of a moderate to severe rain event. THE SEAL BEACH CITY COUNCIL DOES HEREBY RESOLVE: SECTION 1. An Emergency Proclamation for the West End Pump Station — Project No. EM1701. SECTION 2. The City's Charter Section 107 is enacted for said emergency repairs for the preservation of life, health or property. SECTION 3. The Council hereby approves Budget Amendment No. BA17 -07 -02 to modify the Fiscal Year 2016/2017 budget as follows: Description Account Revised /Adopted Proposed Budget (diff) Budget Budget Amendment Contract 045- 333 -44000 $ 12,322,100 $ 12,522,100 $ 200,000 Professional Transfers In $ 12,322,100 $ 12,522,100 $ 200,000 045 - 000 -31500 Transfers Out 001- 080 -47000 $ 9,083,500 $ 9,283,500 $ 200,000 SECTION 4. Approve a Purchase Order to Flo- Services. Inc. in amount of $150,000. SECTION 5. Approve a Purchase Order to Rain for Rent in the amount of $10,000. SECTION 6. Allocate $40,000 for contingency purposes for Project No. EM 1701. PASSED, APPROVED and ADOPTED by the Seal Beach City Council at a special meeting held on the 17th day of January , 2017 by the following vote: AYES: NOES: ABSENT: ABSTAIN ATTEST: Council Members: Massa - Lavitt, Varipapa, Deaton Council Members: None Council Members: Moore, Sustarsic Council Members: None Sandra Massa - Lavitt, Mayor Robin L. Roberts, City STATE OF CALIFORNIA } COUNTY OF ORANGE } SS CITY OF SEAL BEACH ) I, Robin Roberts, City Clerk of the City of Seal Beach, do hereby certify that the foregoing resolution is the original copy of Resolution Number 6708 on file in the office of the City Clerk, passed, approved, and adopted by the Seal Beach City Council at a special meeting held on the 17th day of January 2017. Robin L. Roberts, City Cle c 1 L :I Attachment "C„ CITY OF SEAL BEACH West End Pump Station 2017 Audit May 2017 Submitted to: City of Seal Beach 211 Eighth Street Seal Beach, CA 90740 ,x Submitted by: AKM Consulting Engineers CITY OF SEAL BEACH WEST END PUMP STATION 2017 AUDIT Submitted to CITY OF SEAL BEACH 211 Eighth Street Seal Beach, CA 90740 Submitted by AKM CONSULTING ENGINEERS 553 Wald Irvine. California 92618 May 2017 TABLE OF CONTENTS SECTION PAGE 1.1 History of West End Pump Station and Past Studies .......................................................... .......... 1 -1 1.2 Chronology of Events ..................................................................................... ............................... 1 -1 1.3 Equipment Evaluation ................................................................................... ............................... 1 -2 1.4 Maintenance Program ................................................................................... ............................... 1 -3 1.5 Responses to Community Comments and Questions on Maintenance Logs ............................... 1 -4 2 Pump Station History and Past Studies . I I I � 1 2.1 Original Pump Station and Mainline Influent Storm Drains .......................... ............................... 2 -1 2.2 West End Pump Station Tributary Area ......................................................... ............................... 2 -1 2.3 1999 Master Plan of Drainage ....................................................................... ............................... 2 -1 2.4 West End Pump Station Preliminary Design Report ...................................... ............................... 2 -1 2.5 Ad Hoc Street and Storm Drain Committee Studies ...................................... ............................... 2 -4 2.6 Design and Construction of West End Pump Station Improvements ............... ............................2 -4 2.7 Tributary Storm Drain System .......................................................................... ............................2 -4 2.8 2008 Master Plan of Drainage Update ............................................................. ............................2 -6 2.9 2010 West End Pump Station Study .............................................................. ............................... 2 -6 Chronology 3.1 Storm of Wednesday, December 21, 2016 ................................................... ............................... 3 -1 2.2 Storm of Wednesday, January 4, 2017 .......................................................... ............................... 3 -1 3.3 Inspection by Cascade Pump Company on Friday; January 6, 2017 ............. ............................... 3 -1 3.4 City Council Funding Authorization and AKM Initial Field Inspection; Tuesday, January 17, 2017 ........................................................................................................... ............................... 3-2 3.5 Pump Disassembly and Removal by Flow Services; Thursday, January 19, 2017 ........................ 3 -2 3.6 Pump Disassembly and Inspection at Cascade's Shop; Wednesday, January 25, 2017........... 3 -2 3.7 Cascade Service Report and Repairs; Friday, January 27, 2017 ....................... ............................3 -3 3.8 Installation of Re -Built Pumps; Wednesday, February 1, 2017 ........................ ............................3 -3 3.9 Start-up of Re -Built Pumps; Thursday, February 2, 2017 ................................. ............................3 -3 3.10 Pump 1 Failure; Friday, February 17, 2017 ....................................................... ............................3 -6 3.11 4- Equipment Pump 1 Repair; Sunday, February 19, 2017 through Monday, February 20, 2017 ...................... Evaluation 3 -7 4.1 Pumps ............................................................................................................... ............................4 -1 A. Pump Construction .................................................................................... ............................4 -2 s B. Pump Failures ............................................................................................. ............................4 -2 C. Evaluation ......................................................... ...................................................................... 4 -2 D. Recommendations ..................................................................................... ............................4 -5 KASeal Beach \West End Pump Station 2017\Audit \Seal Beach Comments 5 4 17\Table of Contents.doc I 4.2 Inlet Storm Drains ............................................................................................. ............................4 -5 4.3 Motors .............................................................................................................. ............................4 -6 2 -2 A. Evaluation ................................................................................................... ............................4 -6 2 -3 B. Recommendations ..................................................................................... ............................4 -7 4.4 Variable Frequency Drives ................................................................................ ............................4 -7 4 -2 A. Evaluation .... ........................................................................................................................... 4 -7 B. Recommendations ..................................................................................... ............................4 -8 4.5 Control System ............................................................ ................................................................. 4 -8 A. Evaluation ................................................................................................... ............................4 -9 B. Recommendations ..................................................................................... ............................4 -9 4.6 Standby/ Generator .......................................................................................... ...........................4 -10 A. Evaluation .................................................................................................. ...........................4 -10 B. Recommendations .................................................................................... ...........................4 -10 4.7 Trash Rack ........................................................................................................ ...........................4 -10 A. Evaluation .................................................................................................. ...........................4 -11 B. Recommendations .................................................................................... ...........................4 -11 5 Maintenance Program 5.1 Current Maintenance Program ......................................................................... ............................5 -1 5.2 Recommended Additional Maintenance Practices .......................................... ............................5 -1 A. Pump Station ........................................................................................... ............................... 5 -1 B. Storm Drain System ................................................................................ ............................... 5 -2 5.3 Operator Training ............................................................................................. ............................5 -2 6.1 Responses to Community Comments and Questions on Maintenance Logs ............................... 6 -1 A Cascade Service Reports B Pump Curves C Torque Curves D Motor Data Sheets FIGURE PAGE 2 -1 Original Pump Station Site Plan ........................................................................ ............................2 -2 2 -2 Pump Station Tributary Area ............................................................................ ............................2 -3 2 -3 West End Pump Station Tributary Storm Drains .............................................. ............................2 -5 4 -1 Existing Pump and System Curves .................................................................... ............................4 -2 4 -2 Pump Construction ........................................................................................... ............................4 -3 K: \Seal Beach \West End Pump Station 2017\Hudit \Seal Beach Comments 5 4 17 \Table of Contents.doc i i Tr PHOTOGRAPH PAGE 3 -1 Discharge Bowl Upper Bushing ......................................................................... ............................3 -4 3 -2 Shaft Enclosing Tube ......................................................................................... ............................3 -4 3 -3 Suction Bowl ........................ ......................................................................................................... 3 -4 3 -4 Disassembly of Pump ..................................................................................... ............................... 3 -4 3 -5 Enclosing Tube Connection to Discharge Bowl ............................................. ............................... 3 -4 3 -6 Pump Propeller .............................................................................................. ............................... 3 -4 3 -7 Debris between Propeller and Bowl Liner ..................................................... ............................... 3 -7 3 -8 Debris between Propeller and Bowl Liner ..................................................... ............................... 3 -8 3 -9 Debris between Propeller and Bowl Liner ..................................................... ............................... 3 -8 4 -1 Cascade storm water pumps installed at West End Pump Station ( 2007) ....... ............................4 -1 4 -2 US Motors, 350 HP, 514 RPM Pump Motor ( 2007) .......................................... ............................4 -6 4 -3 Murcal 555 Pump Control Panels ( 2007) .......................................................... ............................4 -8 4 -4 Caterpillar 350 kW Natural Gas Generator ( 2007) .......................................... ...........................4 -10 4 -5 Recommended Trash Rack Improvements - 1 ................................................. ...........................4 -11 4 -6 Recommended Trash Rack Improvements - 2 ................................................. ...........................4 -11 KASeal Beach \West End Pump Station 2017\Audit \Seal Beach Comments 54 17\Table of Contents.doc I I I SECTION 1 EXECUTIVE SUMMARY WEST END PUMP STATION AUDIT EXECUTIVE SUMMARY The West End Pump Station Audit was authorized by the City of Seal Beach to: 1. Document the history of the pump station and past studies 2. Review the recent failures and document the chronology of events from Wednesday, December 21, 2016, when Pump 2 failed, through Monday, February 20, 2017 3. Evaluate the existing equipment and assess their capacities and conditions 4. Review the past maintenance practices and provide recommendations for future maintenance of the facilities 5. Respond to questions and comments by the community The West End Pump Station was constructed in 1959 with (2) natural gas engine driven pumps, each with a rated capacity of 55.7 cfs. It serves a tributary area of 173 acres covering portions of Marina Hill North, Old Town, and Bridgeport communities. The original pump station's total capacity of 111.4 cfs is less than the capacity required by the current criteria (295 cfs). Its influent storm drain system, which has been developed in phases, is very shallow, and has 15 houses constructed over it between Corsair Way and the pump station. Additionally, there are several oil and gas lines that cross the pump station site. Further, the original pump station design does not meet many of the current industry standards. The shallow influent storm drain, the houses constructed over it, and the existing utilities on the pump station site impose extreme constraints on what can be done to improve it. The West End Pump Station was evaluated by several studies, including the 1999 Master Plan of Drainage, the 2001 Preliminary Design Report, 2005 Ad Hoc Street and Storm Drain Committee Street and Storm Drain Assessment Report, 2006 -2007 design and construction of the Interim improvements to the West End Pump Station, the 2008 Master Plan of Drainage Update, and the 2010 West End Pump Station Study. Due to funding limitation, only interim improvements were constructed in 2007, which increased the pump station's capacity to over 200 cfs (approximately 10 year protection). However, the existing drainage system cannot convey more than 115 cfs to the pump station. Chronology Wednesday, December 21, 2016 - Pump 2 failed. Pump 1 was switched to lead status. The equipment was inspected on Thursday December 22, 2016. It was determined that the failure was not due to the pump's variable frequency drive (VFD) or the motor. Attempts to reconnect the pump shaft were unsuccessful. Wednesday, January 4, 2017 — The City contacted a pump station maintenance vendor to reconnect the pump shaft. The vendor was on site on Thursday, January 5, 2017, but could not perform the work. Pump 1 also failed to start on this date. The City moved all portable pumps available to the pump station, and rented additional pumps. K: \Seal Beach \West End Pump station 2017\Audit \Sea1 Beach Comments 5 4 17\Section 1 - Executive Summary 5 4 17.dw 1- 1 City of Seal Beach Section 1 West End Pump Station Audit Executive Summary Friday, January 6, 2017 — Cascade Pump Company, along with Flow Services, inspected the failed pumps. It was determined that the pumps had to be removed for further inspection. Flow Services provided a proposal for the work on Wednesday, January 11, 2017. Tuesday, January 17, 2017 — AKM reviewed the pump station, and agreed that the pumps had to be removed and disassembled to determine the cause of the failures. The City Council held an emergency meeting and authorized the pump removal and repair work. Thursday, January 19, 2017 — Flow services started disassembling the pumps. The pumps were transported to Cascade's facility in Santa Fe Springs on Tuesday, January 24, 2017. Wednesday, January 25, 2017 — Cascade inspected the pumps. Pump 1 had a nylon rope stuck between the propeller and the discharge bowl liner. There was damage to the Pump 2 shaft due to the prior attempts to recouple it at the pump station. Both pumps were repaired. Friday, January 27, 2017 - Cascade provided service reports on the two pumps, stating that both were in good condition and the dimensions were well within acceptable tolerances. Wednesday, February 1, 2017 — Flow Services installed the pumps at West End Pump Station Thursday, February 2 2017 — Operation of the pumps and controls were tested. Adjustments were made to the minimum pump speed and pump ramp up time. Friday, February 17, 2017 — AKM was at the pump station to observe its operation during a storm event expected on this date. Pump 2, which was the lead pump, started and operated with no problems until 5:30 pm. Pump 1 was then switched to the lead position. It too operated without problems until about 8:00 pm, when it failed. Pump 2 started automatically as it was programmed. Sunday, February 19, 2017 — Flow Services arrived on site and found plastic bags wedged between the propeller and the discharge bowl liner. On Monday, February 21, 2017, Flow Services was able to remove the wedged material and reassembled the suction bowl. The ramp up time was reduced to 4 seconds to further increase the power applied to the shaft faster and promote movement of debris through the pump. 1.3 Equipment Evaluation Pumps - Each pump was determined to be capable of discharging from 109 cfs to 115 cfs at full speed. They were found to be in good condition with their dimensions well within acceptable tolerances. Any failure that occurred was due to long, nylon and plastic debris being wedged between the propeller and the discharge bowl liner. The pump oiling system, which had been operated manually by the City, was changed to automatic operation. Recommendations were made to minimize the entry of debris into the storm drain system, the pump station sump, and into the pump; installation of a low level sump pump to completely remove the water from the sump, repairs to the equipment and the building, and how to respond to future similar failures. Inlet Storm Drain Systems — This section describes the tributary storm drain system, and the need to improve the system for conveying the runoff equaling the existing pump station capacity to the pump station. Motors — The motors at the pump station are sized adequately with ample torque to start and operate the pumps. It is recommended to change the oil after each storm season, replace the oil level site gages, and K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 I7\Section I - Executive Summary 5 4 17.doc 1- 2 City of Seal Beach Section 1 West End Pump Station Audit Executive Summary test the motor winding insulation annually, 3 months before the storm season. Variable Frequency Drives — The original Benshaw variable frequency drives (VFDs) have been replaced with Toshiba VFDs in 2015 and 2016. Several settings were the initial factory settings, and some of the controls were missing. Recommendations were made to reprogram the Pump 1 VFD to eliminate the over torque alarm, and program both VDFs to include the time /date calendar so accurate record of alarm events can be maintained. Control System — Both pump control systems were found to be operating properly. The existing control system cannot alternate the pumps automatically. The SCADA system is not recording the events accurately. The alarm dialer is working correctly. Recommendations included reprogramming the control system for automatic pump alternation, verifying back -up float system operation, and correcting the cause of inaccurate data being collected at the SCADA central monitoring station. Standby Generator — The standby generator is working properly, and is in good condition. Recommendation was made to test its operation under load annually. Trash Rack— Several issues were identified with the trash rack. Recommendations were made to reduce the openings at the lower levels, and sealing the openings under the bottom of the trash rack and on the sides. Program 1A Maintenance Current Maintenance Program — The existing program consists of weekly inspections which includes the pump oil reservoir; pump run times; forebay water level; generator run time, battery status, oil pressure and coolant level; inspection of the pump control panel, VFDs, level controllers, and generator panel for alarms; and general inspection of the site, forebay, outlet structure, and security. The generator is exercised automatically every week under no load conditions. The main pumps are not exercised regularly. The sump pump runs regularly and does not require exercising. There is no annual or other service, maintenance, and testing program in place. Pump Station Recommendations - Recommendations for pump station maintenance include exercising of the pumps monthly; cleaning the forebay, trash rack, sump, and the pump following every storm where main pumps operate; changing motor oil and motor insulation testing annually; load bank testing the standby generator annually; testing the float switch operation annually; cleaning the level transducer heads monthly; triggering alarms to verify their operation monthly; checking the flap gate operation annually, and following the preventative maintenance recommendations included in the operation and maintenance manuals for each equipment. Storm Drain System Recommendations - Recommendations for the storm drain system include leaving the catch basin screens in place year round; verifying that catch basin trash screens are in place and are free of debris monthly; patrolling the catch basins during storm events to ascertain they are not obstructed with debris; and video inspecting and cleaning the storm drain system annually. Operator Training - It is recommended that operators receive annual training on all pieces of major equipment, including the main pumps, VFDs, pump control panels, and the emergency generator. Training should be conducted by factory trained representatives of the equipment installed, focusing on maintenance practices, emergency service, use and programming of electronic control devices, trouble shooting, and general operation. All City staff which operate the pump station, or are required to be on call to respond to an alarm, should receive this training. K: \Seal Beach \West End Pump Station 2017\Audit \Seal Beach Comments 5 4 17\Section I - Executive Summary 5 4 17.doc 1-3 City of Seal Beach West End Pump Station Audit 1.5 Responses to Community Comments and Questions on Maintenance Log's Section 1 Executive Summary This section provides responses to community's comments and questions. It addresses each comment and question. It acknowledges the problems with record keeping, describes the reasons for the discrepancies between the run times for the pumps, addresses why the past practice was to bypass the pump oiler solenoid, and describes corrective actions that will be taken where existing methods were inadequate. K \Seal Beach \West End Pump Station 20I7\Audit\Seal Beach Comments 5 4 17\Section I - Executive Summary 5 4 17.doc 1-4 SECTION 2 PUMP STATION HISTORY AND PAST STUDIES WEST END PUMP STATION AUDIT PUMP STATION HISTORY AND PAST STUDIES The West End Pump Station was constructed in 1959. It was designed with two (2) natural gas engine driven pumps, each with a rated capacity of 55.7 cfs at a total dynamic head of 16 feet. Each pump had a 30 -inch diameter reinforced concrete pipe (RCP) discharge pipe terminating at San Gabriel River with flap gates. The site plan for the original pump station is shown on Figure 1. The standards for stormwater pump station intake structure design are included in the Hydraulic Institute Standards, and the Los Angeles County Flood Control District's ( LACFCD) Pump Station Design Manual. Further, the LACFCD Pump Station Design Manual is the industry standard for design of stormwater pump stations in Southern California. The original West End Pump Station design does not meet many of the standards included in the above mentioned documents. The pump station's main tributary storm drain system consisted of 92 feet of 48 -inch diameter reinforced concrete pipe (RCP); 907 feet of gunite lined trapezoidal channel (4 ft wide base, 4 ft deep, and 1.5:1 side slopes) to the current location of Corsair Way; and 104 feet of double 36 -inch x 58 -inch corrugated metal pipe arch (CMPA). The upstream drainage system that existed at that time included a gunite lined trapezoidal channel and 22 -inch x 36 -inch CMPA. The West End Pump Station tributary area covers approximately 173 acres of primarily residential and partially commercial land uses. The tributary area, shown on Figure 2, includes parts of the Marina Hill North, Old Town, and Bridgeport communities. It is bounded by Hellman Ranch to the north; Crest Drive, Balboa Drive, Ebb Tide Place, Silver Shoals Avenue, 10th Street and Main Street to the east; Ocean Avenue, and Marina Drive to the South, and the San Gabriel River to the west. The street elevations in the tributary area vary from 52 feet above mean sea level (amsl) near Crestview Avenue east of Avalon Drive, to 6.3 feet in the Seal Beach Shores Mobile Home Park. The maximum water level elevation in the San Gabriel River is about 12 feet, and the top of the levee elevation is approximately 14 feet. The West End Pump Station was evaluated during the preparation of the City's previous Master Plan of Drainage, which was completed in 1999. The 1999 Master Plan evaluated the drainage system in general accordance with the 1986 Orange County Hydrology Manual and Orange County Flood Control District flood protection criteria. The 1999 Master Plan determined that the pump station capacity, as well as the capacities of all storm drains within the West End Pump Station drainage basin were inadequate. It recommended that the construction of the drainage system improvements initially focus on the upgrade of the West End Pump Station. The City completed a preliminary design report for the West End Pump Station in 2001. This study formulated and evaluated ten (10) alternative projects to provide protection from a 25 -year storm runoff. K:\Scal Beach \West End Pump Station 2017\Audit \Seal Beach Comments 5 4 17\Section 2 - History 5 4 17.doc 2 - 1 N � C 9 y0 , V N N La a c m 0 2 c O N UI a. E a' Y a 7 Q C O Y L Vf M O- m E CL � o. a C Y- W O rr N d 'u 3 'JU //r VJ[l/ O V lby I � I.I Y lJ0 a 4_oC �� e � v� .) Ts • D _ IpSI O l .. r- D.ri ` •d.L • a I V�Y L V Z Y ^. $� ��^.'°. �� � ..� we Liu w L•�' �V }'Emd$ v i4' p any'J �rrav Gr'aYi,v is ,.o -, r/I/ .6f L cP e � D rM; ally' �sr� nDrr. '00 or O D•�a apt'�i u+ n o P O _ � �iPr¢o95sN i> aao czy;, pie / d o 1 rw � e sr d C O a' 3 5 7 ao N a h CL m O .LA 0 m E r 0 m CL E CD 3 wr w 1; 00 UP IRV E4- -�Jjll R- t�,[ L- r Em 5 F 4 1/1 /� �-! ON illelill I WMI l4k, ♦ Mgt PY0 , City of Seal Beach Section 2 West End Pump Station Audit Pump Station History and Past Studies Construction of a new facility at the existing site was deemed not feasible because of the small size of the lot (80 ft x 110 ft), and three oil and gas lines that cross the property along its west and north sides. The preliminary design study recommended Alternative 5A, which would construct a supplemental pump station east of the existing pump station to provide 25 year protection along with the existing pump station. )2:5^, - Ad Hoc.Mreet and storm. uraln Committee mucHeS ' - ,•„ Following the intense storm of October 17, 2004, the City formed an Ad Hoc Street and Storm Drain Committee consisting of 11 members. The Ad Hoc Committee was charged with reviewing the existing drainage and street needs within the City; identifying possible funding mechanisms to finance the priority projects; and provide recommendations for the best method to finance these projects. In 2005, the Ad Hoc Street and Storm Drain Committee finalized the Street and Storm Drain Assessment Report. This report formalized six (6) high and eleven (11) moderate storm drain improvement projects and possible measures to fund these projects. The high priority projects included enlarging the storm drains on Electric Avenue connecting to the terminal branch at western end of Marina. The medium priority projects included paralleling the existing RCBs between Corsair Drive and the West End Pump Station. The City purchased the property adjacent to the east boundary of the pump station property, and authorized the design of the supplemental pump station in 2006. However, due to the high estimated cost of the facility, the City decided to construct an interim improvement project that would implement only the necessary improvements to increase the capacity of the existing pump station to provide protection from an approximately 10 -year storm (200 cfs). The interim improvement project constructed only those components that would provide this capacity, and utilize all other parts of the existing station. It replaced the top slab of the pump station to support the larger /heavier new pumps and motors; provided two new variable frequency drive (VFD) operated electric motor driven pumps, new electrical and control system, housing for the mechanical and electrical equipment, and a natural gas standby generator that could operate one pump during a commercial power outage. Only portions of the 30 -inch diameter discharge pipes were replaced with 36 -inch diameter pipes due to funding limitations. With the pumps provided, the upgraded pump station can provide from a minimum of 213 cfs up to 225 cfs when operating at full speed. Due to numerous existing constraints, the interim improvements to the West End Pump Station implemented in 2007 could not mitigate many of the non - standard features. The most prominent constraint imposed by the design of the existing facility is the very shallow invert elevation of the influent storm drain into the pump station's forebay. This necessitated the construction of the upstream storm drain system at very shallow depths. It may have been possible to reconstruct the pump station and the upstream storm drain system if it were not for the additional constraints imposed by the construction of 11 houses over the double 6' (W) x (3' (H) RCB between 1`t Street and Corsair Way, as well as four (4) mobile homes in the Seal Beach Shores Mobile Home Park. The existing storm drain system serving the West End Pump Station's tributary area is illustrated on Figure 3. The mainline storm drain system tributary to the pump station starts at a catch basin at Coastline Drive, east of Driftwood Avenue in Marina Hill, which collects the runoff from the southerly portions of Marina Hill. It continues as a 5' (W) x 1.92' (H) RCB across an easement and Pacific Coast Highway to Marina Drive. KASeal Beach \West End Pomp Station 2017 \Audit \Seal Beach Comments 5 4 17\Section 2 - History 5 4 17.doc 2 - 4 N �+ c O u � d Y � N a m c m 0 x 0 n N E 6 v c O A L Vf CL A m Ea , — a d � N C w O Y N Y � 3 �• � • �i\, Vii•'! -1, --) y'i iy a ••'tip., ' - r�V LIr'• pC�J1 rte. .J law z W V J W i J N a 0 City of Seal Beach Section 2 West End Pump Station Audit Pump Station History and Past Studies The storm drain transitions first to a 4' (W) x 1' (H) RCB, then to a 30 -inch RCP, and then to a 36 -inch RCP on Marina Drive. It turns northwesterly through and easement as a 36 -inch RCP, extending to a junction structure at 5" Street and Electric Avenue. It continues northwesterly on Electric Avenue as a double 7' (W) x 3' (H) RCB to Corsair Way. It is a double 6' (W) x 3' (H) RCB between Corsair Way and First Street, and is located under 11 single family houses. It is a double 6' (W) x 5' (H) RCB across First Street. It then extends to the pump station forebay as a double 6' (W) x 3.25' (H) RCB. A portion of this most downstream reach is located under four mobile home park structures. One secondary storm drain system tributary to the mainline system starts at a catch basin on Coastline Drive just south of Carmel Avenue, which collects the runoff from the northerly portions of Marina Hill. A 36 "x22" CMPA conveys the runoff from the area through an easement to Pacific Coast Highway where an 18 -inch drain carries the runoff to Galleon Way. The storm drain increases first to 24 -inch then to 30 -inch RCP, and outlets to the double 7' (W) x 3' (H) mainline drain on Electric Avenue. Another secondary storm drain is a 5' (W) x 1.25' (H) RCB, which starts at the low point on Corsair Way and extends easterly to a junction structure on the mainline drain at Corsair Way and Electric Avenue. Runoff from portions of the Old Town and Bridgeport communities drain to the mainline facility through the streets and small secondary storm drains as shown on Figure 3. The City updated its 1999 Master Plan of Drainage with significantly more information that was developed through the above listed studies, as well as detailed topographic mapping of the City that was completed in 2005 and 2006. The 2008 Master Plan incorporated the recommendations of the Ad Hoc Street and Storm Drain Committee. Detailed hydrologic studies determined that the design flow for the pump station (peak runoff from a 25- year storm) was 295 cfs. The recommended high priority improvements consisted of the storm drains on Electric Avenue from Corsair Way to Marina Drive; Electric Avenue from Marina Drive to Main Street and Marina Drive from Electric Avenue to Pacific Coast Highway. The recommendations of the 2008 Master Plan included a parallel 200 cfs pump station to increase the West End Pump Station capacity to handle the peak flow from a 25 year storm, and a storm drain along Electric Avenue and the Oakwood Apartments parking lot from Corsair Way to the new pump station. Since construction of the second pump station along with its influent storm drain system is costly (about $11.4 million), they were categorized as medium priority projects. 12�9 2010 West End Pump Station Study There was concern that the construction of the three high priority storm drains would convey higher runoff to the West End Pump Station than the pump station's capacity, and reduce the level of flood protection in the area near the pump station. The purpose of this study was to determine the 25 -year storm runoff that can reach the pump station during phased construction of the improvements to verify that the runoff conveyed to the pump station did not exceed the existing pump station capacity. The study determined that the existing system cannot convey more than 115 cfs to the West End Pump Station. Extending the system to Marina Drive could convey 184 cfs if sufficient catch basins are provided. Further extension upstream will need to limit the inlet capacity so that the capacity of the existing pump station is not exceeded until the parallel pump station and its influent storm drain are constructed. K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17\Scction 2 - History 5 4 17.doc 2 - 6 SECTION 3 CHRONOLOGY OF EVENTS WEST END PUMP STATION AUDIT CHRONOLOGY OF EVENTS WEDNESDAY, DECEMBER 21, 2016 THROUGH MONDAY, FEBRUARY 20, 2017 This section provides a chronology and description of events starting with Wednesday, December 21, 2016, when a pump failed to start when called by the pump station control system, through Friday, February 17, 2017, when the last pump failure and repair occurred: Pump 2 failed to operate when called on the night of Wednesday, December 21, 2016. Pump 2 is the unit to the north (towards PCH). The City maintenance staff reported that Pump 2 started and ran satisfactorily during the storm that occurred the week before. Maintenance staff responded to the failure alarm on December 21, 2016, and manually switched operation to Pump 1, prior to it being automatically called. Pump 1 started, and ran satisfactorily through the entire rain event. The following day (Thursday, December 22, 2016), the City maintenance staff called MacDonald Electric (a Toshiba service representative), to arrange a service call to check if the VFD had failed. MacDonald Electric suggested that the motor be uncoupled from the pump, and the motor be started manually to determine if the drive was non - functional. Maintenance crews performed the test, and found the drive and motor to be operational, eliminating both items as the source of the failure. The maintenance staff attempted to recouple the pump shaft to the motor shaft, but were unable to do so. 3Z Storm of Wednesday, January 4j 2017 On Wednesday, January 4, 2017, after returning from the City's holiday closures, maintenance staff contacted Orange County Pump, a maintenance vendor that had been used by the City in the past. A service call was s scheduled for the following day (Thursday, January 5, 2017). Rain also fell on this day, in a sufficient amount to require Pump 1 to operate. When called, Pump 1 (south pump) failed to start. Attempts by crews to manually turn the pump shaft with wrenches to free the pump from whatever had caused it to seize were unsuccessful. Orange County Pump arrived on site on Thursday, January 5, 2017 to investigate the cause of the pump failures. Personnel from Orange County Pump appeared to be unfamiliar with the type of equipment installed and were not able to diagnose the problem. At this time, City maintenance crews moved all portable pumps, which the City was renting to the West End Pump Station and obtained additional rental pumps from Rain for Rent to also be placed at the site. (Approximately 16.7 cfs capacity) 33. Inspection by Cascade Pump Company on Friday, January 6, 2017 On Friday, January 6, 2017, City maintenance staff contacted Cascade Pump Company (Cascade) of Santa Fe Springs, California (manufacturer of the pumps at the West End Pump Station) to inspect the equipment. Eric Summerfield (President of Cascade), and John Krukowski with Flow Services (an authorized Cascade service provider) arrived on site later that same day. Attempts were made to manually turn each pump using a strap wrench on the shaft, but the effort was unsuccessful. An inspection of the interior of the pump bowl was then made from the sump. Nothing was observed in the pump bowl that would prevent it from turning. Noticing oil pooled in the head of Pump 1, Eric Summerfield believed that the problem was a pump issue, and most likely a seized bearing, as such a failure would prevent oil from flowing down the pump shaft and would cause the oil to overflow into the head. Eric Summerfield advised the City that the only way to repair an internal K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17 \Section 3 - Chronology of Events 5 4 17.docx 3- 1 City of Seal Beach Section 3 West End Pump Station Audit Chronology of Events pump problem, such as a seized bearing, was to remove the pumps from the site and have them repaired in Cascade's shop. The City requested that Cascade and Flow Services provide a proposal for the repairs. That proposal was submitted to the City on Wednesday, January 11, 2017. Due to the estimated cost of the repair work, City Council authorization for the expenditure was required. An emergency City Council meeting was held on Tuesday, January 17, 2017 to brief the Council on the issue, and obtain spending authorization for the repair work. On the morning of Tuesday, January 17, 2017, the City contacted AKM Consulting Engineers for background information on the pump station, in preparation for the emergency session of City Council (AKM had prepared the contract documents for the construction of the 2007 interim improvements to the pump station). At that time, AKM was asked to come out to the site and render an opinion on what might have caused the pumps to fail. AKM arrived on site in the early afternoon with a representative from Flow Systems (a sales representative for Cascade). At that time, both pumps were uncoupled, and oil was observed to be pooled in the head of Pump 1 (south pump, nearest to the ocean). City staff briefed AKM on when the pump failures had occurred, and actions taken by the City to place the pumps back on line. Staff indicated that the owner of Cascade and their service representative, Flow Services, were on site the week before to inspect the pumps and provide recommendations for their repair. After being briefed at the site by City maintenance staff, AKM decided that all necessary steps had been taken by the City, and no additional course of action to assess the cause of the failures could be offered at that time. Flow Systems contacted Vaughan Industrial Services (a company specializing in the repair of pumps) to look at the pumps. A representative from Vaughan Industrial arrived on site later that afternoon. After examining the pumps and discussing the previous inspection efforts, Vaughn also concluded that the problem was internal to the pump, and could only be repaired in the shop. On Thursday, January 19, 2017, Flow Services began the pump removal process. AKM arrived on site in the afternoon. Flow Services was in the process of, removing the pumps' suction umbrellas and discharge pipe couplings, so that the pumps could be pulled from the sump. While on site, Flow Services stated that a crane was scheduled to remove the pumps over the weekend and then ship to Cascade's shop in Santa Fe Springs for disassembly, inspection, and repair. However, due to continuing rains, pump removal could not be performed during the weekend. On Tuesday, January 24, 2017, the pumps were pulled from the sump using a large crane, and shipped to Cascade Pump Company for inspection and repair. AKM arrived at Cascade's shop in Santa Fe Springs in the morning of Wednesday, January 25, 2017, with a representative from Flow Systems. Upon arrival, engineering and maintenance staff from the City of Seal Beach and a representative from Flow Services were already at the shop. Both pumps were lying on their side, on the shop floor, with most of the bolts removed but still assembled. Pump 2 was the first to be taken apart. Prior to disassembly, Cascade crews successfully turned the pump propeller by hand, using a long piece of lumber. As the pump was disassembled, the bearings, line shaft, enclosing tube, and propeller were all inspected. A thin film of organic material covered all internal parts of KASeal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17 \Section 3 - Chronology of Events 5 4 17.dooc 3- 2 City of Seal Beach Section 3 West End Pump Station Audit Chronology of Events the pump. Aside from the line shaft, all parts were in good condition. The line shaft had been scarred near the coupling from attempts made to manually turn the pump with wrenches while installed at the West End Pump Station site. Because of this scarring and the close machining tolerances, the coupling was stuck to the shaft and could only be removed by cutting it off. Although all parts looked to be in good condition, the City directed Cascade to install new bushings, and machine and install a new line shaft. Pump 1 was then disassembled. Attempts to manually turn the pump propeller were unsuccessful. Like with Pump 2, all parts, when removed, appeared to be in good condition. As the propeller was pushed out of the bowl, a short piece of nylon cord, which was lodged between the propeller and bowl liner fell out, freeing the propeller to turn. The City directed Cascade to provide new bushings for Pump 2 prior to being cleaned up and reassembled. The line shaft was not damaged on this pump, and did not require replacement. Photos of the pump disassembly are provided on the following page. Service reports prepared by Cascade were furnished to AKM on Friday, January 27, 2017. AKM provided comments and asked for additional information from Cascade. The updated reports were e- mailed to AKM on Monday, January 30, 2017. The reports summarized observations and repairswhich were being performed. They also provided measurements for all pump assembly components as well as the design tolerances. All components showed little to no wear, indicating that the pumps were well balanced, not damaged by debris, and well lubricated. Cascade concluded that the pumps did not fail from an internal defect or wear, but from trash being lodged between the propeller and bowl. The repairs for both pumps consisted of providing new bushings throughout the pump, balancing the propeller, and reassembling the pumps using new stainless steel bolts. Repairs also included replacing the lineshaft and the pump half coupling for Pump 2, because these had been damaged in the field while trying to test the newly installed VFD operation. Cascade's inspection reports are included in Appendix A of this Audit. Cascade informed AKM through Flo- Systems that the repaired pumps had left their facility on Wednesday, February 1, 2017. The rebuilt pumps were installed on February 1, 2017. AKM went to the site in the afternoon, and observed the unloading and installation of the pumps. On February 2, 2017, AKM was on site to verify that the pumps were installed correctly and operated properly. Personnel from Flow Services, Industrial Technology (service technicians for the Toshiba VFD) and the City maintenance crews were also at the pump station. To test the operation of the pumps, sand bags were installed blocking the double 6' (W) x 3.25' (H) RCB inlet to the forebay, and a fire hydrant connection was used to fill the forebay. While the forebay was filling, Industrial Technology checked the VFD programming parameters and noted those which were not factory default settings. This information was recorded by Flow Services. Industrial Technology also checked the alarm history for each drive. No history was available forthe Pump 1 VFD (newer drive). Apparently, due to a software change by Toshiba, acknowledged and cleared alarms are no longer stored for this VFD. For the Pump 2 VFD, 27 alarms were logged. However, because the calendar date and time were not programmed into the drive, it is not known when these alarms occurred. K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 417 \Section 3 - Chronology of Events 5 417.doot 3- 3 City of Seal Beach West End Pump Station Audit Section 3 Chronology of Events Photograph 3— Suction Bowl Photograph 4 — Disassembly of Pump Photograph 5 — Enclosing Tube Connection to Discharge Bowl Photograph 6 — Pump Propeller K:\Seal Beach \West End Pump Station 2017\Audh\Seal Beach Comments 5 4 17\Section 3 - Chronology of Events 5 4 17.dor 3- 4 City of Seal Beach Section 3 West End Pump Station Audit Chronology of Events Most of the alarms stored in the VFD were power failure, which probably happened during start-up testing. Three overload alarms were listed, likely generated bythe initial seizing of the pump and subsequent attempts to restart it after the failure. A single alarm related to power quality from SCE was also logged. After the forebay was filled, the pump oiler by -pass valve was opened and the oil drip rate was adjusted to 10 drips per minute. The pump was then allowed to manually lubricate for a period of 15 minutes, before starting was attempted. After the lubrication period was completed, Pump 1 was called to operate in hand mode. During the pre -lube period, it was observed that the pre -lube system solenoid valve failed to open, and the pump control switch was turned off before the pump was able to start. Pump 2 was then called to operate in hand mode, however, its oiler solenoid valve also did not open, and the pump was turned off. Flow Services then checked the operation of both solenoid valves by plugging them directly into a 120 volt AC power source which caused both valves to open. Wiring terminations in the VFD cabinets were then checked, however, because the wires were not marked, it was difficult to determine which wires went to the solenoid valves, and if the wires were landed correctly. To solve the problem, Flow Systems connected each solenoid valve to line cord and strung the cord overhead to the VFD cabinets. Pump 1 was then called to start, again in hand mode. This time the oiler system solenoid valve functioned correctly. The timed pre -lube period before the pump started was 2 minutes. When the pump started, it took 60 seconds to ramp up to minimum speed, which was set at 30 Hz (50% of full speed). Based upon information from Toshiba, the 60 second ramp up time is the default factory setting for this size motor. The ramp up time was discussed, and was decreased to 10 seconds. A shorter time is preferred so any material in the pump at start is evacuated quickly, before it can wrap around the propeller. The pump was started again with a 10 second ramp up time, and operated with no issues. While the pump was starting, an attempt was made to monitor the incoming voltage, however, the power monitor was not working. The power monitor should be repaired. The minimum speed setting was then discussed. Higher minimum speed settings will create more velocity in the pump bowl to help move debris downstream. Too high a speed setting will however, cause short cycling of the pump, which could reduce its operational life. It was decided that the minimum speed should be increased to 40 Hz (67% of full speed). The same ramp up time and minimum speed settings were then programmed into the Pump 2 VFD. When operated in hand mode, the pump started and ran properly after the pre -lube period had expired. Each pump was operated in automatic mode. The level of the wet well was allowed to rise to the pump's on (start) set point (El. 4.0). When called, each pump successfully went through a 2 minute pre -lube period and then started, ramped up to 40 Hz, and then the speed was slowly increased to about 42 Hz before the water level in the forebay was lowered to within the maintain level set point (El. 3.0) deadband (deadband is 0.5 ft). The speed then froze and as the water level in the forebay decreased below the maintain level deadband, the speed was reduced to 40 Hz, where it stayed until the water in the forebay receded to the off (stop) set point of El. 2.0. The maximum current reading on the drive during this operation was approximately 380 amps. When Pump 1 was operating, the VFD human - machine interface (HMI) displayed an over - torque alarm. The Toshiba field technician was not sure why the alarm was being generated. However, he believed it had to do KASeal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17 \Section 3 - Chronology of Events 5 4 17.docx 3 • 5 City of Seal Beach Section 3 West End Pump Station Audit Chronology of Events with programming parameters, and was not an indication of something being wrong with the pump, motor, or VFD. Subsequent to the testing on February 2, 2017, recommended programming changes to eliminate the false over- torque alarm was forwarded to AKM. Pump 2 did not display an over - torque alarm when operated. Each pump was operated and tested through several start and stop cycles to verify reliable operation While on site, a light rain was falling throughout the field testing period. Several leaks in the roof of the building were noted. Although mainly an annoyance, if a leak were to develop over the VFD, it could cause the equipment to be damaged. It is recommended that the roof be inspected and repaired to eliminate the current and future leaks. Heavy rains were forecasted for the Seal Beach area on February 17, 2017. The City requested AKM to be on site to observe pump operation during the storm. AKM arrived atthe West End Pump Station at 10 AM, priorto the start of rainfall. Strong winds were preceding the storm, and a lot of branches, palm fronds, and trash from blown over waste containers were observed in the streets. Because no rain was falling, the City's Public Works Department asked AKM to leave and return around 4 PM when the main body of the storm was expected to hit. AKM returned to the site about 3:30 PM. Rain was falling and strong winds were still present. All City Public Works staff were patrolling the City, clearing obstructed catch basins, operating portable pumps, and responding to fallen tree reports. While AKM was at the pump station, two City operators were present. Pump 2 was operating as it was set to be the lead pump. The operators indicated that the pump was working well and there were no issues. Pump 1 had not been called to operate at that point. It was observed that Pump 2 was being called at the correct forebay elevation (4.0 ft). The oiler solenoid was then observed to open, and the pump went through a 2- minute pre - lubrication cycle. Drip rate of the oil was 10 drops per minute. After the pre -lube cycled expired, the pump started and ramped up to minimum operating speed of 40 Hz in 10 seconds. The pump speed then continued to increase to about 45 Hz, until the water level in the forebay lowered to within the deadband of the maintain set point (El. 3.0 ft). The water level in the forebay continued to steadily drop below the maintain set point, and out of the deadband at which time the speed of the.pump was slowly reduced to 40 Hz (the minimum speed setting). When the water level in the forebay reached El. 2.0, the pump shut down, the oiler solenoid valve closed, and the sump pump started. The pump run cycle was about 4 minutes. After the main pump was shut down, the forebay began to fill. After about 4.5 minutes, the forebay reached the start set point for the main pump, and the start -up sequence was repeated. While AKM was on site, forebay fill times ranged from 3 to S minutes. Forebay pump down cycle times ranged from 2 to 5 minutes. The pump never operated faster than slightly over minimum speed (45 Hz) and quickly pulled the forebay level down. At about 5:30 PM, Pump 1 was switched to the lead pump operation by changing the pump start set points in the Murcal controller. The switch was made to even out run times. K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 417 \Section 3 - Chronology of Events 5 4 17.doa 3- 6 City of Seal Beach Section 3 West End Pump Station Audit Chronology of Events Several on -off cycles for Pump 1 were witnessed. The Pump 1 operation was similar to the Pump 2 operation. Oiler drip rate was verified at 9 drips per minute. The maximum operating speed witnessed for Pump 1 was 45 Hz. Run time cycles were between 3.5 and 4.5 minutes. At approximately 6:30 PM, AKM left the site with Pump 1 still operating as lead. At approximately 8:00 PM, AKM received a call from City maintenance staff that Pump 1 had failed. Pump 2 had automatically started as it was programmed to do, when the water level in the forebay reached El. 5.0 ft. AKM returned to the site at approximately 9:00 PM. Pump 2 was on -line and operating. Pump 1 had failed on motor overload. Maintenance crews and the Public Works Director were on site. AKM requested the operator to manually start Pump 1. The operator switched the pump on. The current on the motor rose to about 700 amps without turning, at which point the pump was switched off. A call was made to Industrial Technology (Toshiba VFD Service Technicians) to discuss if the problem was possibly related to the VFD. Industrial Technology was of the opinion that the pump had seized. No attempt was made to test the motor and drive by operating uncoupled from the pump. AKM remained on site until midnight when rainfall had subsided. Pump 2 ran without incident while AKM was on site. On Saturday, February 18, 2017, AKM returned to the site in the morning. The Public Works Director and maintenance crews were present. Pump 2 had operated without a problem through the morning. The City had contacted Flow Services to inspect and service Pump 1. On Sunday, February 19, 2017, Flow Services arrived on site and found plastic bags wedged between the propeller tip and the bowl liner. The pump was manually rotated about Y. turn before seizing. Attempts to cut the bags out of the impeller were unsuccessful. Flow Service returned to the pump station on Monday, February 20 with all necessary equipment to remove the pump suction bowl to gain better access to the debris in the pump. Flow Services successfully removed the material preventing the pump from turning, re- assembled the bowls, and placed the pump back on line. At that time, the pump ramp up period from start to minimum speed was reduced to 4 seconds. Photographs 7, 8, and 9 illustrate the debris lodged between the propeller and the discharge bowl liner. Photograph 7 — Debris between Propeller and Bowl Liner K:\Seal Beach \West End Pump Station 2017\Audit\Seal Beach Comments 5 417\Sectlon 3 - Chronology of Events 5 417.doct 3- 7 City of Seal Beach Section 3 West End Pump Station Audit Chronology of Events Photograph B— Debris between Propeller and Bowl Liner Photograph 9 — Debris between Propeller and Bowl Liner KASeal Beach \West End Pump Station 2017\Audh\Seal Beach Comments 5 4 17\Section 3 - Chronology of Events 5 4 17.docc 3- 9 SECTION 4 EQUIPMENT EVALUATION WEST END PUMP STATION AUDIT EQUIPMENT EVALUATION The original pump station construction plans are dated January 26, 1959. Per the plans, it was constructed with two (2) Fairbanks Morse single stage Figure 6360 mixed flow pumps, each rated at 25,000 gpm (55.7 cfs) and 16 ft TDH at a speed of 655 rpm. The interim improvements implemented in 2007 provided two new pumps, which are Cascade Pump Company 48 AP, single stage propeller pumps, shown on Photograph 1. The factory pump test information, including the pump curves at 500 rpm speed are included in Appendix B. Figure 1 shows the pump curves at full speed (100 %), 90%, 80 %, 67 %, and 50% of full speed, as well as the system curves under two conditions. One system curve (lower) is for when the water level in San Gabriel River is below the centerline elevation of the discharge pipes at the outlet (7.5 ft above mean sea level), and the second system curve is used to evaluate the pump operation when the San Gabriel River Water level is at the maximum elevation of 12 feet above mean sea level. Under the worst conditions, where the forebay level is at 3 feet and the San Gabriel River level is at 12 feet, each pump can deliver 47,800 gpm (106.5 cfs) when operating at 500 rpm. They can deliver 50,600 gpm (112.75 cfs) each (total of 225.5 cfs) at 500 rpm, when the water level in San Gabriel River is at or below 7.5 feet. The operating full speed of the pumps is approximately 510 rpm, where the pumps will deliver approximately 109 cfs to 115 cfs, depending upon the water level in San Gabriel River. The torque requirement for the pump at full operating speed (514 RPM) is 2,962 lb-ft. The required starting torque for the pump is about 150 lb-ft. The type of pump installed is typical for low head, high flow applications, and is commonly used for stormwater pumping. The certified pump curve and generic torque curve for the pump is included in Appendix C of this report Photograph 1 - Cascade storm water pumps installed at West End Pump Station (2007 Photo) K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17 \Section 4 - Equipment Evaluation 5 4 2017.docx 4 - 1 City of Seal Beach West End Pump Station Audit 25 20 0 0 �`— 15 x 0 30 5 Section 4 Equipment Evaluation WEPS Cascade Pumps 0 30000 32000 34000 36000 3800D 40000 42000 44000 46000 48000 50000 52000 54000 56000 58000 60000 62000 Flow (gpm) Figure 1— Existing Pump and System Curves A. Pump Construction t 100% Speed t Low SGR Level High SGR Level 67% Speed 80% Speed t 90% Speed Pump construction is shown on Figure 2. It consists of an aluminum- bronze propeller located in a stainless steel lined pump discharge bowl, which is submerged in the fluid being pumped. The propeller is connected to the motor located above by a stainless steel shaft. The shaft is enclosed by a tube, and lubricated using a food grade oil. The line shaft is supported by four bronze, sleeve type bearings (also called bushings). B. Pump Failures Pump 2, the north pump, failed on Wednesday, December 21, 2016. Pump 1, the south pump, failed on Wednesday, January 4, 2017 and Friday, February 17, 2017. The pump is a standard design used in many stormwater pump stations in Southern California. Pumps manufactured by Cascade are particularly prevalent in the area, as they have been proven to be reliable, and support for pump repair is local (Santa Fe Springs). The pumps installed at the West End Pump Station were observed to be in good condition when inspected at Cascade's shop on January 25, 2017. There were no signs of corrosion on the bowl, column, shaft enclosing tube, or discharge elbow. The internal components of the pump showed no wear and were also in good condition. The line shaft was straight, bearings were not deformed or discolored, and the line shaft was well oiled. Dimensional measurements performed by Cascade while the pump was disassembled showed all internal components to be well within design tolerances. Cascade's inspection report for each pump is provided in Appendix A. The initial failure of Pump 1 (Wednesday, January 4, 2016) was caused by plastic cordage becoming lodged between the propeller tip and the pump bowl. The failure, which occurred on February 17, 2017, after the pump was refurbished, was caused by plastic bags twisted into a long rope like shape, also caught between the propeller and bowl liner. KASeal Beach \West End Pump station 2017\Audx \seal Beach Comments 5 417\Section 4- Equipment Evaluation 5 4 2017.doa 4-2 City of Seal Beach West End Pump Station Audit Section 4 Equipment Evaluation R E V l S 1 0 N 5 WEIGHTS PUMP - 12,]00 LDS. MOTOR 10,350 LBS. PUMP k MOTOR = 22.850 LBS, MAX, PUMP TTRUS1 = 9,953 LBS. THRUST FACTOR - 442 IMPELLER = 405 LDS. BOWL/LINE SHAFT = 319 LOS, HEAD SHAFT = 172 L65. PAINT NOTE ABOVE OA E SAND BLAST ASE P TE /MOTOR �PjLQE5G AND BASE INSIDE AND OUT FR $jPC -SP6. APPL LUCK PRIIAE_GQAT q'65 FQLQR HI -BULO EPO%ODUg TQ 4 MIS DIP. F_QLI,DWE'J BT INIEI2MajLATE COAT N66 HI -BUILQ EP jgU.E TO ! MIL Of FIN4HEp_W TH FNSH GOAT RJS CO;�O_R AgRA -5H J (1 4 M Fi B D BA< ❑ 1� t EA$S_6Q_' S9N0 BLAST ELBOW BELOW 84S NO_UQM AITI I 111 111, nSSEMBLY NSOE AND OUT AND OUTSIpE OF ERCy� 1S i TUQU PFR 551C -SPID APP T PR MF CQAjJQ5_QSLQR t -BUILD P% I! -RIMER T 4 M F G W Y FINI H L'Q4f �'fi�COIOR ENO, PSL.A -$HI _i0_�_M_L$_[IGI, AI B04TING DE OW PA$E -_±57M A 320 GRAOF E=8 < N <jL, }_ _ T j�V �LEC_.D k❑TOR \3/ �/ 3 PH GD Hj 4fi I' 514 RPM. LL` 5F NRC WP -II ENGI O' R' �2 III QSEM FRAMF fiB08P LT[M$ NO } J B �, 1 I. AND 14 ARE y[QpMMEN❑F" PAP RT`; t/Z _.�l__.��__ I/2" ( 'II � 4.OVPLNG GUARDS -�� - 15' t _ L "0 RAINA - -/ �- DISCHARGE 1 is HOLES NOZZIL Y I — -Y 37 TIE KOp BRACHFT TAI �14J , I 3' -6 7/8" I� r 72 -,J (91 —'LH' U-1 - -- -- -- - - i 7. -10.. z 117 I / Z 4 _._ 1 PR TIE �Rc❑f. L I. a _Q_J Q-9 APA T \\ 3D" O.D. 84' r fin' FOUNDATION j \, - OP Nlll' / 1 � I 13 -Z. mm_ BA PLAT' D-1AII 9 a A a ��\ a 3 u^ y n 6 i a as v m� I'D vN LVLL- EL__50' J 4• T4 36 a c o� c y OO ZN 0 3 _ 1/4" a (Jm �b nr EL.: -2.59' — AO „ - ---- -- 8A' �l a NOT,: 7" MIN SPHERE EL.: -5.09' SIZC TD PASS PUMP LILY iM1rl l� W:CMI4aN ._ sLw. uelo�. we . -e4n n sm III Figure 2 -Pump Construction K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17 \Section 4 - Equipment Evaluation 5 4 2017.doa 4 - 3 rrrt . e4[x eaw _ eroa onA /:.a su� �m rve trXn w "wr S: w:M ftr/�r[ �R:S Moe. /.r.NW T i u cwR c Figure 2 -Pump Construction K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17 \Section 4 - Equipment Evaluation 5 4 2017.doa 4 - 3 City of Seal Beach Section 4 West End Pump Station Audit Equipment Evaluation The seizing of Pump 2 could not be definitively determined. No material was found in the pump bowl at Cascade's shop, and the pump propeller was able to be turned freely by hand while lying on its side. It is presumed that whatever was preventing the pump from turning while installed at the West End Pump Station in the vertical position became dislodged during the pump removal and shipping process. It is certain however, that the failure is not attributable to a malfunction of the VFD or motor. The design of the pump requires extremely close running clearances between the propeller and the bowl to prevent cavitation and produce greater pump efficiencies. There is not an established standard for what this gap should be. It is determined by each manufacturer for their specific pump design. For the pumps at the West End Pump Station, this gap is designed to be 0.025" according to Cascade engineers. The gap measured in the field by Flow Services was 0.018 ". Cascade considers the 0.018" gap to be within the design tolerances for the pump. Since being installed in 2007, no pump failure from debris had been reported by the City until the events of Wednesday, December 21, 2016. It is therefore our opinion that the pump seizing from stringy debris is a result of a heavy trash loading in the storm flow created by years of drought and trash accumulation in the storm drain system. Recent video inspection (conducted in March 14, 2017) of the storm drain on Galleon Way upstream of the West End Pump Station showed the pipe to have a substantial amount of debris deposit, and is probably indicative of the rest of the City's storm water collection system. It is not believed that the failure of either pump was caused by lack of maintenance or non - operation for a long period of time. Both pumps were in good condition and internal parts were well oiled. The pumps also successfully started when first called after a long period of non - operation over the summer. Pump 2 ran successfully the week before it failed on Wednesday, December 21, 2016. Pump 1 ran several times between Wednesday, December 21, 2016 and Wednesday, January 4, 2017, when it first failed. Although axial flow pumps are capable of passing large solids (the pumps at West End Pump Station are rated to pass a 7 -inch diameter sphere), long boards, rope, and an excessive amount of debris in the storm flow, can and does cause damage and seizing occasionally. While inquires to the County of Orange revealed that they had not recently experienced a failure due to debris, they have in the past at all of their pumping stations. When the City's pumps were being inspected at Cascade's shop on Wednesday, January 25, 2017, Cascade staff informed the City and AKM staff that they had a pump in their shop from the City of Huntington Beach, which had seized similarly to Seal Beach's Pump 2. The City's practice of removing screens at the inlets to catch basins during the winter increases the amount of debris collected in the storm drain system that is eventually conveyed to the pump station. It is a program that was put into place to prevent catch basins from becoming obstructed during a rain event. This should be discontinued immediately. During a storm, crews should patrol the City and manually clear debris from catch basin inlets. As the City builds more catch basin capacity into its system, catch basins should be designed with excess capacity to account for being partially obstructed by debris during storm events. The pump oiling system solenoid valves were found not to be operating when first tested. Because the City was bypassing the solenoid, with the intention of constantly delivering oil to the pump's line shaft, the failures did not impact lubrication of the pumps. It just wasted food grade oil. The problem with the solenoid is a wiring issue. Temporary wiring was installed by Flow Services so that the solenoids would work. The bypass valves on the oiler systems are currently closed. K: \Seal Beach \west End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17 \Section 4 - Equipment Evaluation 5 4 2017.docx 4 - 4 City of Seal Beach Section 4 West End Pump Station Audit Equipment Evaluation iD. Recommendations The following recommendations are offered to minimize the chance of future failure of the pumps due to trash and debris. 1. Leave catch basin screens in place during the storm season. Patrol flood prone areas of the City during rain events to ensure catch basin screens are unobstructed. 2. Improve the trash rack at the pump station by closing off ungrated openings, and reducing the slot lengths of the existing grate. 3. Clean the pump station forebay and the wet well after every major storm event to prevent the accumulation of trash. 4. Implement a program to clean accumulated trash from the storm drain system 5. Inspect the interior of each pump after every storm for debris. Remove any foreign material which may be observed. 6. Provide permanent wiring to each pump's oiler system solenoid valve. 7. Install a small wet well low level evacuation pump to drain the wet well to facilitate inspection and maintenance of the pumps. Additional recommendations are: 1. Repair or replace the power monitors 2. Repair roof to eliminate any leaks It is recommended that if a similar failure occurs in the future, that the first course of action should be to inspect the pump bowl from underneath, and look for debris. If nothing is observed, the suction bowl should be removed to provide an unobstructed view of the pump bowl. If nothing is found after removing the pump suction bowl, the coupling connecting the pump line shaft and motor shaft should be loosened so that the propeller will drop downward, potentially dislodging the material between the impeller tip and bowl. This is typically performed by a pump contractor experienced in this work. The original pump station was designed with 92 feet of 48 -inch diameter inlet storm drain, and a gunite lined trapezoidal channel extending to the current location of Corsair Way. The 48 -inch RCP and part of the trapezoidal channel to the north of 11t Street were replaced with a double 6' (W) x 3.25' (H) reinforced concrete box (RCB) storm drain in 1978 (plan date). It connected to the double 6'x5' reinforced concrete box which extended across 11t Street. The inlet storm drain continues southeasterly as a double 6' (W) x 3' (H) RCB under the homes located to the west of Electric Avenue between 11' Street and Corsair Way. Ajunction structure and 5' (W) x1.25' (H) RCB storm drain were constructed between Electric Avenue and the low point in Corsair Way west of Electric Avenue in 2005. The storm drain in Electric Avenue was extended upstream between Corsair Way and SO Street as a 7' (W) x 3' (H) RCB in 2013. Further upstream, the storm drain continues southeast as a 36" RCP to Marina Drive, turns east, and reduces to 30" near 7th Street. It is a 4' (W) x 1' (H) RCB across Pacific Coast Highway (PCH), and a 5' (W) x 1.92' (H) RCB through an easement, terminating at a sump with a 14 -foot wide catch basin on Coastline Drive east of Driftwood Avenue. K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 417 \Section 4 - Equipment Evaluation 54 2017.docx 4 - 5 City of Seal Beach Section 4 West End Pump Station Audit Equipment Evaluation The influent storm drain system between Corsair Way and the pump station can convey the design capacity of 200 cis to the pump station. However, there are not sufficient storm drains further upstream, nor catch basins to intercept the tributary runoff. Therefore, the runoff that cannot enter the storm drain system ponds in the streets, and causes flooding in several areas. In order to be able to convey the existing pump station capacity to the pump station and mitigate the flooding, additional storm drain and sufficient catch basin improvements need to be implemented in the tributary area. Each pump is driven by a 350 HP, 514 RPM, premium efficiency motor, with a service factor of 1.15 (US Motors 6808P HUEI). Full load current draw of the motor is 495 amps. When operating in the service factor range, the maximum current draw is 560 amps. Starting torque rating for the motor is 2178 lb-ft. Available torque at full load is 3,630 lb-ft. Breakdown torque, which is the maximum torque the motor is capable, is 6,353 ft -lb. The motor is a solid shaft type. It is oil lubricated, with a space heater that prevents internal condensation when the pump is not operating, and temperature switches in the windings to prevent overheating. Photograph 2 —US Motors, 350 HP, 514 RPM Pump Motor (20071 The data sheet for the pump motors is located in Appendix D A. Evaluation Both motors were operational and looked to be in good condition when inspected. A minor oil leak was observed from the oil fill site gauges. These gauges should be replaced. The oil on the Pump 1 motor was darker than that for Pump 2, which could be an indication of operation at high temperatures. Although both motors have very little run time, oil should still be changed after every storm season as recommended by the motor manufacturer. The City staff noted that the oil in the motors had never been changed. Both motors ran smoothly, and during unloaded operation, they drew about 240 amps, as indicated on the VFD HMI screen. Under loaded conditions (driving the pump), operating at 45 HZ (75% of full speed), the current reading was approximately 380 amps. Each motor has more than ample torque to drive its pump. The motor's available starting torque is 14 times greater than the torque required by the pump (2178 ft -lb available; 150 ft -lb required). At full load operation, the motor's torque exceeds the pump requirement by 22% (3630 ft -lb available; 2962 ft -lb required). K:\Seal Beach \West End Pump Station 2017 \Audit\Seal Beach Comments 5 417 \Section 4- Equipment Evaluation 5 4 2017.doat 4-6 City of Seal Beach Section 4 West End Pump Station Audit Equipment Evaluation B. Recommendations Both motors appear to be in good operational condition. The following preventative maintenance is recommended. 1. Oil should be changed after each storm season. 2. Testing of the motor winding insulation should be performed once per year to verify that it is in good working order. The test should be performed at least 3 months prior to the storm season so that if a problem is discovered, the motor can be repaired and reinstalled prior to the start of rain. 3. The oil level site gauges should be replaced soon to stop the minor oil leaks which have developed. 4.4. Variable Frequency Drive� Each pump is started and its speed modulated by a variable frequency drive (VFD). There is a dedicated VFD for each pump. The VFDS serve two purposes: 1. Motors are started using much less current than would be required by a standard starting contactor, which can draw up to 6 times the motor's full load current rating. For large motors, like the ones at the West End Pump Station, SCE requires a reduced voltage starting method to prevent voltage drop and resulting flicker in the service area during starting. 2. Pumping rate can be adjusted to match the influent flow rate. This reduces the on -off cycling of the pump, which is hard on the electrical and mechanical equipment, reducing their service lives. It is also more efficient as the pumping head is lower due to less friction with the lower flows, reducing the power required to pump the stormwater. The original Benshaw drives were found to be unreliable. They suffered numerous failures of electrical components during their short service life. The City replaced them with Toshiba VFDs. The 400 HP Pump 2 drive was installed in 2014. The 500 HP Pump 1 drive was installed in the summer of 2016. Both VFD's are fully operational. Failure records were logged by the older of the two VFD's (Pump 2). Due to software changes by Toshiba, the newer drive (Pump 1) does not store historical alarms. Of the 27 alarms logged, almost all were power failure, which were presumably triggered during start-up testing. There were three overload alarms, probably generated by the initial seizing of the pump, and subsequent attempts by operators to restart it after the failure. A single alarm was logged that was related to power quality from SCE. Because the calendar on the drive was not programmed, the actual time and dates of the alarms is not known. After the pumps were serviced by Cascade and re- installed, an over - torque alarm was observed on the display screen of the Pump 1 VFD. Although the alarm was generated, it did not fail the pump. Further investigation of the programming showed that the alarm had been disabled from shutting down the drive. No such alarm was observed on the Pump 2 VFD. After consulting with Toshiba service representatives, it is believed that the alarm is the result of mis- programming of the VFD. Adjustments to parameter F015 "V /f pattern" and F016 "Torque Boost" have been recommended, but have not yet been implemented. The remaining programming of both drives was otherwise similar. K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17\Section 4 - Equipment Evaluation 5 4 2017.doa 4 - 7 City of Seal Beach Section 4 West End Pump Station Audit Equipment Evaluation During startup of the re -built pumps on February 2, 2017, it was observed that the ramp up time (from start to programmed minimum speed) was 60 seconds, which is Toshiba's factory default setting value for this size motor. This is extremely long. The slow ramp up time increases the chance for debris to become entangled with the pumps propeller. This setting was initially changed to 10 seconds, and later reduced to 4 seconds, so that any material in the pump is immediately evacuated. It was also observed that the pump's minimum speed setting was set at 50% of full (257 RPM). To help mitigate future obstructions, this speed setting was increased to 67% of full (344 RPM). This increased pump cycling, but helps move debris through the pump more quickly. If problems with clearing the pump of debris continues, increasing the minimum speed further should be considered. B. Recommendations 1. Reprogramming of the Pump 1 VFD should be conducted to eliminate over torque alarm which is being generated. 2. If continued failure of the pumps due to debris occurs, consideration should be given to increasing the minimum speed setting. 3. The time /date calendar should be programmed into both drives so that an accurate record of when an alarm event occurred can be maintained. A Murcal 555 PLC based control panel is used to start/stop and command the speed of each pump. The control panel receives a level measurement signal from an ultrasonic level transducer located in the station's forebay. The pumps are started at a pre - programmed high level set point. When called to start, a 2 minute pre - lubrication period, which is appropriate for the length of pump shaft, (this value is adjustable) is initiated. During this time, a solenoid valve is opened on the pump's oiling system allowing oil to drip down the pump's line shaft, lubricating the bearings. After the pre -lube period times out, the control panel calls a pump to start. When started, the pump is Photograph 3— Mutual 555 Pump Control Panels 120117) ramped up to its pre - programmed minimum operating speed (currently set at 40Hz; 67% of full speed). The controller then increases or decreases the speed of the pump as necessary to maintain a programmed constant level in the forebay. As the level in the forebay is lowered below the "maintain" set point, the speed of the pump is reduced, but is not allowed to fall below the minimum speed setting (40 Hz). When the forebay level is lowered to the off set point, the pump is shut down, and the oiler solenoid valve is closed. K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 54 17 \Section 4- Equipment Evaluation 5 4 2017.dm 4-8 City of Seal Beach Section 4 West End Pump Station Audit Equipment Evaluation In the event of a level transducer failure, float switches are installed as a back -up control system for each pump. When actuated, a pump is started and runs at full speed until the forebay is lowered to the level of the off float switch. Each pump has a dedicated control panel and level sensor, providing complete redundancy. The control system does not have the ability to alternate pump operation after each on-off cycle. The pump with the lower start set point will always operate first. Pump operating set points are as follows. Consequently, to change a pump from lag to lead status, the starting and stopping levels have to be changed manually. Transducer Settings Float Switch Settings Pump 1 Pump 2 Pump 1 Pump 2 On EI.5.0 EI.4.0 EI.6.0 El. 6.5 Maintain El. 3.0 El. 3.0 Off El. 2.0 El. 2.0 E.1.5 E1. 1.5 The control panel also generates status and alarm signals, which are transmitted to the pump station's remote telemetry unit (RTU). The RTU then sends the signals, via a radio network, to the City's SCADA central monitoring station, located at the Adolfo Lopez Maintenance Yard. Critical alarms (power fail, pump fail, high forebay level) are transmitted by the controller to an alarm dialer, which then calls the dispatcher at the Police Station, who then contacts City maintenance crews for response. A. Evaluation Both pump control panels were found to be operating properly. They start and stop the pumps at the programmed set points and accurately modulate the speed of the pumps. Both level transducers also were operating properly, accurately measuring levels. The float switch back -up system was not tested as part of this review, but should be tested by City staff to verify that it is also working correctly. The control panel system is not capable of pump alternation. Alternating the pumps increases the time between successive starts, which extends the life of the mechanical and electrical equipment. It also ensures even run time, achieving equal wear, extending the time before replacement is required. The data captured by the City's SCADA system was reviewed, and found to be inaccurate. It is important that the SCADA system accurately record what is happening at the station, so that operational issues and station performance can be reviewed. It is recommended that the City's SCADA system consultant investigate the data issue, and implement corrective measures so that the system is operating properly. The alarm dialer was found to be working correctly. It is our understanding that alarm notifications are sent to the police dispatch center since someone is always available to receive the call. The dispatcher however, must then contact the maintenance person that is on -call to respond to the station alarm. This system seems to be working. The City could consider having the dialer directly call maintenance staff, eliminating the extra step of having the Police Department involved. The dialer can be programmed to call a list of numbers. It will continue to call until a person answers and acknowledges the alarm condition. B. Recommendations 1. Reprogram the Murcal panels to provide automatic pump alternation. 2. Verify back -up float system operation. KASeal Beach \West End Pump Station 2017 \Audh\Seal Beach Comments 5 4 17\Section 4 - Equipment Evaluation 5 4 2017.d= 4-9 City of Seal Beach Section 4 West End Pump Station Audit Equipment Evaluation 3. Have City's SCADA consultant review and correct the cause of inaccurate data being collected at the City's SCADA central monitoring station. A 350 kW Caterpillar emergency generator and automatic transfer switch are used to provide electrical power in the event of a commercial outage. The generator is fueled by natural gas. It is sized to start and run one pump only. During a power outage, one of the two pumps is locked out from operating. A switch is provided on one of the control panels to select the pump which will remain active when the generator is in operation. The automatic transfer switch (ATS) is located outdoors in the electrical service meter cabinet. The ATS is constantly monitoring the incoming power. If it Photograph 4— Caterpillar 350 KW Natural Gas Generator (2007) senses low voltage, phase loss, phase imbalance, high voltage, or total loss of power, it will call the generator to start. After a pre - programmed warm up period, and if the failed power condition is still present, the ATS switches the line side power source from SCE to the generator. When normal power is restored (as sensed by the ATS), the line side power is switched back to SCE and the generator goes into a preprogrammed cool down period before being shut down. The ATS and generator are also capable of being operated manually, if needed. The generator operation was not tested as part of this audit. City staff indicate that it operates properly. The generator is automatically exercised weekly by the transfer switch under no load conditions. B. Recommendations It is recommended that the generator be tested once per year, either with a load bank, or by running a pump to verify it is working properly under loaded conditions. The trash rack at the West End Pump Station is the original device installed in the pump station when it was built in 1959. It was not upgraded as part of the 2007 interim improvements project due to budget limitations. The rack is constructed of galvanized steel and inclined at an angle of approximately 45 degrees. The grates are attached to a galvanized steel frame. The lower level of grates are hinged to allow access into the pump sump. The openings in the grating are approximately 36- inches long by 3- inches wide. KASeal Beach \West End pump Station 2017 \Audit \Seal Beach Comments 5 4 17\Section 4 -Equipment Evaluation 54 2017.docv 4-10 City of Seal Beach West End Pump Station Audit A. Evaluation Section 4 Equipment Evaluation While the design of the trash rack exceeds what has generally been observed from that time period, its openings are larger than what is currently employed at most storm water pump stations today (standard opening is 6" x 4 "). Because trash and debris is a problem at the pump station, it is recommended that the openings on the lower rack be closer together to catch more debris. A 2 -inch x 3 -inch opening is recommended. This could be accomplished by welding bars across the existing rack or replacing it with a new grate manufactured with the smaller openings. Because a rack with smaller grate openings has a greater chance of being obstructed, the grates above the bottom row should have larger openings (6 -inch x 3 inch is recommended). Again, this can be accomplished by welding bars horizontally onto the existing grates or providing new ones with the recommended opening size. The existing rack also has openings between the debris sill and the grate, as well as on the side of the rack, that should be blocked. The recommended improvements are shown on Photographs 5 and 6. e. Recommendations 1. Either modify or replace existing trash rack grates to reduce the current slot opening. 2. Seal existing openings under the bottom row of grates, and on the sides of the rack. Photograph 5— Recommended Improvements -1 Photograph 6 - Recommended Improvements - 2 K!\Seal Beach \West End Pump Station 2017\Audit\Seal Beach Comments 5 417 \Section 4- Equipment Evaluation 5 4 2017_docx 4-11 SECTION 5 MAINTENANCE PROGRAM WEST END PUMP STATION AUDIT MAINTENANCE PROGRAM The City's current maintenance program includes weekly inspections of the pump station. During inspections of the station, the following is conducted: 1. Checking and refilling of pump oil reservoir. 2. Logging of pump run time (all pumps). 3. Logging of forebay water level. 4. Logging of generator runtime, battery status, oil pressure, oil level, coolant level. 5. Inspection of pump control panel, VFDs, level controllers, and generator panel for alarms. 6. General inspection of the site, outlet structure, and forebay for trash, security, and any other abnormal conditions. The generator is automatically exercised weekly, by the automatic transfer switch, with no electrical load. The main pumps are not exercised. The sump pump runs regularly and therefore does not require exercising. There is no annual service, maintenance, or testing program currently in place. The City's preparedness for major storm events can be improved by implementing the following additional maintenance, testing, and training practices. A. Pump Station 1. Manually exercise pumps once per month. Pumps can operate safely with no water for a period of 1 to 2 minutes. When operating, make sure the forebay level is below the pump inlet so that damaging vibration and cavitation does not occur. The sump pump will need to be manually operated to remove the water, as its normal off set point in automatic mode is 6- inches above the main pumps suction bell. 2. Clean the pump station forebay and trash rack of debris prior to the beginning of the storm season (before October) and after every storm where the main pumps have to operate. Inspect the interior of each pump bowl for the presence of debris and remove after each storm the pump is called to operate. 3. Change motor oil once every year. 4. Megger motors annually, in July, to determine if there is insulation degradation, which will allow servicing and re- installation prior to the beginning of the storm season. 5. Annually test the natural gas generator with load bank. 6. Test float switch operation annually, just ahead of storm season. 7. Clean level transducer heads once per month. Verify each device is measuring accurately. 8. Trigger alarms once per month to verify alarm dialer is working properly. 9. Check flap gates at the outlet structure to verify that they close properly and can be easily opened prior to storm season. 10. Follow preventative maintenance recommendations contained in the Operation and Maintenance Manual for each piece of equipment. K:\Seal Beach \West End Pump Station 2017\ Audit \Seal Beach Comments 5 417\ Section 5- Maintenance Program 54 2017.docx 5- 1 City of Seal Beach Section 5 West End Pump Station Audit Maintenance Program 8. Storm Drain System 1. Monthly verify catch basin screens are in place. Remove trash accumulation if necessary. 2. Annually video inspect storm drains for debris. Clean storm drains where camera indicates debris accumulation by water jetting. 3. Leave catch basin screens in place year round. During rain, patrol the storm drain system to make sure screens do not become obstructed with debris. Clear debris as necessary. Regular operator training is essential to ensure that equipment is properly maintained and operated. It is recommended that operators receive annual training on all pieces of major equipment. This includes the following: main pumps, VFDs, pump control panels, emergency generator. Training should be conducted by factory trained representatives of the equipment installed. Training should focus on maintenance practices, emergency service, use and programming of electronic control devices, trouble shooting, and general operation. All City staff which operate the station, or are required to be on call to respond to an alarm should receive this training. K: \Seal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17 \Section 5 - Maintenance Program 5 4 2017.docx 5 - 2 SECTION 6 RESPONSES TO COMMUNITY COMMENTS AND QUESTIONS ON MAINTENANCE LOGS WEST END PUMP STATION AUDIT RESPONSES TO COMMUNITY COMMENTS AND QUESTIONS ON MAINTENANCE LOGS The log shows that Pump 1 had 50.8 hours on 1/11/2016 and the same exact number of hours on 12/19/2016. This makes it unclear as to whether this pump had any line testing for almost a year before it failed on 12/22/2016. Pump 1 did not fail on 1212212016, Pump 2 failed. Pump 1 operated properly on that day. There was no entry on the 22nd because the response to the site was not a part of the weekly inspection activity. It is agreed that a notation in the log book should have been made on December 22 "d, indicating that Pump 2 had failed. The entry on January 3, 2017 for Pump 2 should have also noted that the pump was off -line. It was not the City's maintenance policy to exercise the pumps. Therefore, the log is accurate. Pump 1 was not operated for over a year. It is important to note that lack of operation did not cause the pumps to foil. The failures were caused by debris. Pump 2 had successfully operated the week before it foiled, and Pump I started and ran on the 22 "d after Pump 2 had foiled. The City acknowledges that record keeping is important, and will be implementing changes to its policies and practices to ensure more accurate and complete records are maintained. Although lack of operation did not cause the pump failures, we believe exercising the pumps on a monthly basis is a good routine maintenance procedure that should be implemented to help ensure station readiness to operate. 2. Pump 2 had less than 7 life time running hours on it when it failed in the first week of January. Pump 2 failed on December 21, 2016. Pump 1 failed on January 3, 2017. The running hours listed for Pump 2 on January 3, 2017, is 5.4. This is the number of hours on the pump since the new VFD was installed (2015). The total hours for Pump 2 was logged on December 19, 2016 as 48.2 and is the total run time over the life of the pump (since 2007). There was an inconsistency on the logs as to which run time was recorded. The run time meter on the pump control panel is the true running hours for the pump. The run time meter on the VFD is the pump run time since the new VFD was installed in 2015. The maintenance log will be improved so that both run time counters are recorded. 3. What type of motor /controller heaters are installed? The VFD has on internal solid state overload sensing device. Heaters are not part of the overload protection. The pump failures were not electrical in nature, they were caused by debris lodged between the pump propeller and discharge bowl liner, which prevented it from turning. 4. Pump 1 was not exercised between August 14, 2015 and December 19, 2016. Pump 2 appears to have been exercised more but likely less often and less long than pump manufacturers would recommend. Lack of exercise can lead to frozen bearings along the drive shaft, and prevent the pump from restarting. In addition, there is some indication in the pump logs that the drive shaft drip lubrication oil reservoirs had gone dry sometime before January 3, 2017. Exercising of the pumps was not part of the normal maintenance routine for the pump station. Pump 2 was not exercised, it was the lead pump that always started first. The second pump was never needed to run. KASeal Beach \West End Pump Station 2017 \Audit \Seal Beach Comments 5 4 17 \Section 6 - Responses to Community - Maintenance Logs 5 4 2017.docx 6- 1 City of Seal Beach Section 6 West End Pump Station Audit Responses to Maintenance Log The sleeve bearings used in the pump are less likely to freeze from non - operation then other types of bearings, and the manufacturer makes no recommendation for exercising of the pump. However, operating the pumps periodically is a good practice to ensure the station is prepared to operate, and will be added to the City's monthly maintenance routine. As stated in the response to comment number 1, the station did not foil because of lack of operation, it failed because debris became lodged between the propeller and discharge bowl liner. The log does indicate that the oil reservoir for Pump 1 was empty on January 3, 2017. The solenoid by -pass valve on both pumps was open. The pump shaft was therefore being continuously oiled, which was the City's practice to ensure the pump is always adequately lubricated. We agree this should have been observed sooner and the reservoir not allowed to be empty. Pump 2 was off line and inoperable at that time. The oil reservoir should have been filled and the solenoid bypass valve closed and tagged, so that the oil would not be wasted. The oiler solenoid valves are now being used so that the pumps are only oiled when called to run. 5. Pump 1 South shows 50.8 hours from 1/11/2016 to 12/19/2016 with readings of 0.9 on 7/11/2016, 8/15/2016, 8/29/2016, 9/26/2016, 11/7/2016, 11/21/2016, 12/5/2016, 12/19/2016 and then jumps to 4.1 on 1/3/2017. So either someone has been reading the wrong meter or there are possibly two meters, one digital the other mechanical. Pump 2 North shows 4.8 to 6.4 hours from 1/11/2016 to 1/3/217 with overlaying readings of 47.1 to 48.2 hours on 4/11/2016 to 12/19/2016. I do not understand why there is so much oil being used. Typically, your startup sequence would open a solenoid valve a few minutes before the actual pump starts up to allow a few drops of oil to make its way down the pump shaft to the bearings. Nowhere in this report does it indicate if the pumps were turned to indicate any possible bearing binding. Simply removing the top of the hollow shaft motor will expose the adjusting nut and pump shaft that should be easily turned by hand or with the aid of a strap wrench. Since these pumps were on VFDs system they could have been powered up while using the VFD to slowly turn the motor and pump even with no water in the system. I still question why both pumps were on VFDs. You would ramp up one pump "lead" with the VFD until you reached its maximum speed then drop it off and bring your second pump "lag" online "across the wire', and again ramping up the first pump to continue with your required variable flow rate. If the VFD on the lead pump were to fail, you still have one pump able to run without the possible failure of the VFD. There are two sets of running time meters. One is on the pump control panel (Murcal controller). It counts the hours the pump has been operated since it was installed in 2007. The other meter is on the VFD. It counts the hours on the pump since the VFD was installed (2015 and 2016). The run times have been inconsistently recorded in the logs. In the future, both counters (control panel and VFD) will be read and recorded. The City does not regularly exercise the pumps, so the only testing time would be when the VFD's were installed, which would not be captured in the maintenance log. It was the City's practice to leave the oiler solenoid by -pass valve open at all times to ensure the pump's line shaft and bushings were always adequately lubricated. This practice has been changed. The by -pass valve has been closed. The oiler solenoid opens for 2- minute5 prior to a pump starting and stays open during pump operation. It is closed when the pump is shutdown. The logs do not indicate what actions were token to re -start the pumps. Manually turning of the shaft was attempted but failed. The motor is a solid shaft type. KASeal Beach \West End Pump Station 2017\qudlt\Seal Beach Comments 5 417 \Sectlon 6 - Responses to Community - Maintenance Logs 5 4 2017.docx 6- 2 City of Seal Beach West End Pump Station Audit Section 6 Responses to Maintenance Log Having dedicated VFD's is preferred. If a drive fails, the station can be ran normally with pump speed and output modulated to match inflow. Full voltage starting and operation will cause excessive pump cycling, require a larger generator to start and run a pump, and most likely would not have been allowed by SCE for the size of motors installed. 6. 1 have the following observation for the logs. a. They are not date stamped. We do not really know when they were filled out. b. The consistency of the ditto and hash mark would indicate they were done quickly and without any real thought. c. Hash and ditto marks are not acceptable in log books — each entry, even if repetitive, should be completed legibly. d. The hash and ditto marks are not consistent; sometimes an "X ", sometimes ditto, sometimes nothing. e. The test runs of the generator are more legible than the pump tests. Each box is filled instead of using hash marks, but there appears to be hurried entries into the pump logs. f. There is not an area for comments or observations. There is no routing mechanism for critical issues. g. Some entries are scribbled out— protocol for any log is to use a single horizontal line so that the data recorded but changed can give insight into what the technician was thinking before the data was changed. h. Some entries have no technician initials. i. None of the entries have the actual time noted as to when the technicians were on site. j. There is inadequate room for data on some logs and information is written in the margin. The West End Pumping Station is a critical facility. The logbooks present an impression of a country club atmosphere at City Hall. I am skeptical that they are wholly accurate. We agree that the record keeping protocol must be improved and that all operators must be trained on how to properly fill out and maintain the logs. The City will be developing a new log book format and instruction document for completion of forms and maintaining records. K: \Seal Beach \West End pump Station 2017 \Audit \Seal Beach Comments S 4 17 \Section 6 - Responses to Community - Maintenance logs 5 4 2017.docr 6- 3 APPENDIX A CASCADE REPORTS ct� J CASCADE PUMP COMPANY MANUFACTURERS OF AXIAL & MIXED FLOW PUMPING EQUIPMENT January 27, 2017 Flo - Services, Inc. 3010 Floyd Street Burbank, CA 91504 ATTENTION: John Krukowski SUBJECT: Seal Beach (Pump Serial Number: 16970) DOCUMENT TITLE #1234 Page 1 of 5 10107 SOUTH NORWALK BOULEVARD, PO BOX 1767 SANTA FE SPRINGS, CA 90670-0767 P :562.946.1414 F :562.941.3730 The pump was disassembled in the presence of City of Seal Beach, AKM and Flo- Services, Inc. representatives. The components of the pump assembly were checked visually and physically for damage and excessive wear prior to dimensional verification. A. The pump assembly was first disassembled from the elbow. Nothing of remark was found. B. The enclosure tube was removed from the bowl assembly exposing the shaft. There was no visible evidence of damage or excessive wear on the shaft. C. The suction bowl was unbolted from the discharge bowl to expose the propeller for removal. There was a large amount of sump debris build -up around the propeller hub, on the blades and on the bowl liner. The propeller was difficult to turn by hand. D. The propeller and shaft were driven forward to clear the discharge bowl liner. There was no abnormal interference with removal. The shaft slid easily inside the bushings. The bowl liner showed no evidence of excessive wear, damage from FOD or cavitation, but did have small scratches. The baffle plate exhibited a small build -up of corrosion and debris from the sump at the base of the O.D. E. The retaining ring was removed and the shaft slid forward to release the thrust collar and key. F. The propeller was then slid off the shaft and checked for damage. There was no visible distortion of the blades, excessive wear, or damage from impact or cavitation. G. The pump assembly components were dimensionally checked for the following: MADE INTHE USA WWW.CASCADEPUMP.COM HYDRAULIC p SINCE 1948 INSTITUTE PROUD MEMBER ct% J DOCUMENT TITLE #1234 Page 2 of 5 # Inspected Feature Should Be Actual Notes 1 Propeller O.D. 35.990" - 36.000 35.9935" Dimensionally acceptable. 2 Propeller Bore I.D. 2.939" - 2.941" 2.9396" Dimensionally acceptable. 3 Propeller Pitch 880 880 Pitch is consistent on all blades. No deformation. 4 Propeller C /Bore 11.810" - 11.812" 11.788" 5 Propeller Keyway 3/4" x 3/8" 3/4" x 3/8" Porosity in Keyway. No deformation detected. Suction Bowl 2.945" - 2.947" 1" Deep = 2.9458" Dimensionally acceptable. 6 Bushing I.D. (After Press) 5" = 2.9475" No interference with shaft. 8" = 2 949„ Discharge Bowl 2.945" - 2.947" 1" Deep = 2.9457" Dimensionally acceptable. 7 3 = 2.9463" Top Bushing I.D. (After Press) No interference with shaft. 5" = 2.9463" Discharge Bowl 2.945" - 2.947" 1" Deep = 2. 947" Dimensionally acceptable. 8 3 = 2'948 Bottom Bushing I.D. (After Press) No interference with shaft. 5" = 2.947" Tension Nut 2.945" 2.947" 1" Deep = 2.9468" Dimensionally acceptable. 9 3" = 2.9468" Bushing I.D. (After Press) 5" = 2 948„ No interference with shaft. 10 Shaft O.D. 2.937" +.000/ -.002 2.9355" Diameter in round. 11 Shaft Straightness .005 per ten feet .006 at 14' Straightness acceptable 12 Shaft Length 166 - 7/16" 13 Baffle Plate O.D. 11.750" - 11.752" Top (Clean) = 11.760" Bottom (Dirty) = 11.905" When pump 16970 arrived at Cascade Pump, the shaft and propeller were not able to move. During the disassembly of the pump and having the pump lay in a horizontal position, the shaft and propeller broke free and where able to turn. There was no seizure between a bushing and shaft, and there was no evidence of something being wrapped around the propeller. The only evidence of damage on the pump was scratches on the runner bore. This connects with what was seen in the field, when the pump shaft was able to turn '' /< of the way with a lot of resistance. This would indicate that something got jammed in between the propeller and runner bore which caused the propeller to not turn. Cascade is going through normal service repair and providing new bushings throughout the pump, balancing the propeller and assemble using new stainless steel bolts. Per our disassembly and dimensional checks, there were no material or workmanship defects found. Note: While in the field, it was observed that excessive oil was on the pump base. Make sure to check your connection from the oiler to the pump and that the valves are working properly before starting the pump. MADE INTHE USA WWW.CASCADEPUMP.COM CI HYDRAULIC, I SINCE 1948 ✓�J INSTITUTE PROUD MEMBER • Ib Suction bowl and bushing Baseplate & discharge elbow DOCUMENT TITLE #1234 3of5 MADEINTMEUSA WWW.CASCADEPUMP.COM R� HYDRAULIC SINCE 1948 91 INSTITUTE (tOUn EMBER CAPDOCUMENT TITLE #1234 ,� Zl► \ Discharge bowl and bushing Propeller after being polished and re- balanced MADE INTHE USA WWW.CASCADEPUMP.COM HYDRAULIC � SINCE 1948 INSTITUTE PROUD MEMBER CAft%,� J DOCUMENT TITLE #1234 Page 5 of 5 Flanged enclosing tube & Lineshaft Scratches on runner bore MADE IN THE USA WWW.CASCADEPUMP.COM HYDRAULIC SINCE 1948 In INSTITUTE vJ CASCADE PUMP COMPANY MANUFACTURERS OF AXIAL & MIXED FLOW PUMPING EQUIPMENT January 27, 2017 Flo - Services, Inc. 3010 Floyd Street Burbank, CA 91504 ATTENTION: John Krukowski SUBJECT: Seal Beach (Pump Serial Number: 16971) DOCUMENT TITLE #1234 1 of 10107 SOUTH NORWALK BOULEVARD. PO BOX 2767 SANTA FE SPRINGS. CA 90670 -0767 P;562.946.1414 F :562.941.3730 The pump was disassembled in the presence of City of Seal Beach, AKM and Flo- Services, Inc. representatives. The components of the pump assembly were checked visually and physically for damage and excessive wear prior to dimensional verification. A. The pump assembly was first disassembled from the elbow. Nothing of remark was found. B. The enclosure tube was removed from the bowl assembly exposing the shaft. There was no visible evidence of damage or excessive wear on the shaft. C. The suction bowl was unbolted from the discharge bowl to expose the propeller for removal. There was a large amount of sump debris build -up around the propeller hub, on the blades and on the bowl liner. The propeller could not be turned by hand due to a piece of nylon rope that was wedged between the propeller blade and the discharge bowl liner. Once the nylon rope was removed, the propeller was able to spin freely. D. The propeller and shaft were driven forward to clear the discharge bowl liner. There was no abnormal interference with removal. The shaft slid easily inside the bushings. The bowl liner showed no evidence of excessive wear, damage from FOD or cavitation. The baffle plate exhibited a small build -up of corrosion and debris from the sump at the base of the O.D. E. The retaining ring was removed and the shaft slid forward to release the thrust collar and key. F. The propeller was then slid off the shaft and checked for damage. There was no visible distortion of the blades, excessive wear, or damage from impact or cavitation. G. The pump assembly components were dimensionally checked for the following: MADE INTHE USA WWW.CASCADEPUMP.COM HYDRAULIC p SINCE 1948 INSTITUTE PROUD MEMBER t` J DOCUMENT TITLE #1234 Page 2 of 5 # Inspected Feature Should Be Actual Observations Dimensionally acceptable. Note: Approx. 6" long 1 Propeller O.D. 35.990" - 36.000 35.9909" piece of nylon rope found wedged between blade and bowl liner. 2 Propeller Bore I.D. 2.939" - 2.941" 2.9396" Dimensionally acceptable. 3 Propeller Pitch 880 880 Pitch is consistent on all blades. No deformation. 4 Propeller C /Bore 11.810" - 11.812" 11.808" 5 Propeller Keyway 3/4" x 3/8" 3/4" x 3/8" No deformation detected. 1" Deep = 2.949" 6 Suction Bowl 2.945" - 2.947" 5" = 2 9465„ Dimensionally acceptable. Bushing I.D. (After Press) No interference with shaft. 8" 2 947° 1" Deep = 2.9455" Discharge Bowl 2.945" - 2.947" Dimensionally acceptable. 7 Top Bushing I.D. (After Press) 3" = 2.9458" No interference with shaft. 5" = 2 9458„ Discharge Bowl 2.945" - 2.947" 1" Deep = 2.9475" Dimensionally acceptable. 8 Bottom Bushing I.D. (After Press) 3" = 2 948„ No interference with shaft. 5" = 2.948" 1" Deep = 2.947" 2.945" - 2.947" Dimensionally acceptable. 9 Tension Nut I.D. (After Press) 3" = 2 947„ No interference with shaft. 5" = 2 947„ 10 Shaft O.D. 2.937" +.000/ -.002 2.936" Diameter in round 11 Shaft Straightness .005 per ten feet .007 at 14' Straightness acceptable 12 Shaft Length 166 - 7/16" Wrench marks on shaft below coupling 13 Baffle Plate O.D. 11.750" - 11.752" Top (Clean) = 11.761" Bottom (Dirty) = 11.793" When pump 16971 arrived at Cascade Pump, the shaft and propeller were not able to move. During the disassembly of the pump and having the pump lay in a horizontal position, the shaft and propeller broke free after the piece of nylon rope was removed. There was no seizure between a bushing and shaft, and there was no evidence of something being wrapped around the propeller. Cascade is going through normal service repair and providing new bushings throughout the pump, balancing the propeller and assemble using new stainless steel bolts. While in the field, both the lineshaft and pump half coupling were damaged. Both of these items will be replaced. Per our disassembly and dimensional checks, there were no material or workmanship defects found. MADE INTHE USA W WW.CASCADEPUMP.COM HYDRAULIC SINCE 1948 INS„ ITUTE PPODU EMBED �J Baseplate & discharge elbow Propeller DOCUMENT TITLE #1234 3of5 MADE IN THE USA W W W.CASCADEPUMP.COM HYDRAULIC SINCE 1948 Ca INSTITUTE PROUD MEMBER DOCUMENT TITLE #1234 Flanged enclosing tube & Lineshaft Suction bowl and bushing MADE INTHE USA WWW.CASCADEPUMP.COM HYDRAULIC SINCE 1948 In INSTITUTE GPOOO MEMAER QUO-0 DOCUMENT TITLE #1234 Damage on top of lineshaft caused in field by pipe wrench Left Side: Lockring, packing, propeller key, tension nut Right Side: Pump half flanged coupling which was damaged due to pressing off of damaged shaft. MADE INTHE USA WWW.CASCADEPUMP.CCM HYDRAULIC SINCE 1948 T3 INSTITUTE FOUO MEMBER APPENDIX B PUMP CURVES CASCADE pump company ssnw fc springs, rallfomie PUMP TEST DATA Socal Pacific Construction Purchase Order No. 1007- 738 -07 West End - Seal Beach Testing Infamation Dale: 13.Aug -07 Tested By: KAL Pump Serial No.: 10910 Canslsntforvordud: 21411.09 Static Head (9): 211!1 2114 12 Pipe I.D. al Gauge (in): 41.5 Test Motarinfmme0an: 514 RPM 550 Horsepower 1500 Meter Readup C.T. Multiplier Teel No. 4765 48AP Propeller One Stage 500 RPM Propeller 8.85 Pitch Ratio POm n COns"ona Deai9n Speed Gpclry i0fel Dynamic Mead Bowl E/0cienc 8nae None 1e11 75].1 TOW Achat Reading 40203 /93,0 19 (Ituo"ol Mercury) Capacity (loNeaor Mercury) no Velocity 51e0c Dynovid: Speed or Wait input Brake I OA 13.57 S.G. M 1 3 3 a s 6 z OsalOn Speed (fpm) Soo Flout Total Dynamic Hnd ro-I fe 78761 27.17 75].1 TOW Achat Reading 40203 /93,0 19 (Ituo"ol Mercury) Capacity (loNeaor Mercury) Prom's Velocity 51e0c Dynovid: Speed or Wait input Brake I OA 13.57 S.G. Ituml 13.57 S.G. Head 0. Heed & Head 2 Heed Motor Mona r Hare r 1 7.4D 38538 21. 24.43 1.BB 218 2027 510.0 fB6 385.8 374,1 2 43,0 44976 f7.0 1934 1477 Z48 23.58 610.4 108 353,8 33J.1 3 6.60 50285 1210 15,15 2.21 248 10.84 611.0 130 321.7 302.1 1 OAO $4243 70.00 11.29 257 Z48 11.01 511.0 f3, 29aA 2153, 21 5 730 57932 1.30 828 Z93 248 1J.86 6115 121 258.5_ 277.1 6 8.70 61398 130 3.73 3,26 2.48 950 61Lp tOS 225.2 7 _ 1.20 _ 63608 0]0 03,4 Y9 23,8 8.30 6122 50 193.0 1775 M 1 3 3 a s 6 z OsalOn Speed (fpm) Soo Capacity m Total Dynamic Hnd Sn2e Hon of 78761 27.17 75].1 14060 3727 7127 40203 /93,0 19 93016 12.7 361.9 56630 133,8 26.5 69972 93,7 194.2 62091 923 86.4 Bowl Walsh 753,% 20.4% 23.4% e4.8% 223,% TGA-A 57.4% Design Speed (rpm) Capacity Tote Dynamic Hod (in Sake HDrimspame0f 0 0.77 tlA 0 0.00 OA _ 0 DAD g3, 0 0.00 0 D 0.06 ao 0 OA0 ae 0 DAO I OA Carolled by CASCADE pump company mine fe sprang" alifomia PUMP TEST DATA Socal Pacific Construction Purchase Order No. 1007 -738 -07 West End - Seat Beach Testing Information Dale: 20•Aup07 Tested By: KN. Pump Sedal No.: 10971 CeiuWllOr Veolud: 21441.59 Stalk Heed (0): 2032 j 2114 12 Pipe I,O, at Gauge (In): 41.5 Teat Mokr Inkamolum: 614 RPM 500 H9nepower 1600 Mal. Reading C.T. MWUp1kr Teal He. 4771 48AP Propeller One Stage 600 RPM Propeller 0.85 Pitch Ratio Pump,Wg CatllOons Dufnn Speed froral CapacOy (am) iotaf Dynamle Head Bawl EMcifee Brake H.sa ewer 3947 Tdal Aclual Reedkp 49203 11.07 Y948 pnr2ln o1 Newry) Gpadry, OrcAfifi of Newry) Pnaau. Velocity Shlk Dynemk Speed etYyall Input Brats 189.2 13.57. Iii DeM1n 9pead (rpml Soo FiOw Total Dynamic Hutl fl PTaesum 36754 27.47 3947 Tdal Aclual Reedkp 49203 11.07 Y948 pnr2ln o1 Newry) Gpadry, OrcAfifi of Newry) Pnaau. Velocity Shlk Dynemk Speed etYyall Input Brats 189.2 13.57. m 1"73.G. Hatl 5 Head Had a Head m Mai. Mona r Ha owar 3.40 38636 12.00 24.65 1.30 265 26.60 510.1 i9T 406.7 30.73 2 4.50 45464 17.10 19.34 I.al 2.55 23.70 610,5 160 362.6 361A J 650 1M0 16.15 731 2.55 18.91 51110 idf 323.9 381. 4 ,.4 _58265 54243 10.00 11.90 2.67 2.r6 1 .tt 511.2 140 30043 2111.5 9 7.30 5783 760 e. /1 2.03 25S 13.95 Sfi.S 121 270.2 252. 6 630 67773 430 3.73 3.33 2.55 9.82 512.1 100 7252 209.1 6.00 63406 430 0.34 3.93 2.55 642 51 2.3 a 197,3 ta240 Iii DeM1n 9pead (rpml Soo Capacity Total Dynamic Hutl fl Brake Nora 36754 27.47 3947 M549 --2-2.75- 320.9 49203 11.07 Y948 53059 19.37 363A _ 50699 13.33 3.16.7 00313 1.17 1040 02079 612 189.2 N Design Speed (mm1 Capacity f...1 Tetel Dynamic Had flift Brake Horoog= 0 OAO 00 0 0.00 OA 0 0.00 OA 0 OAO 0.0 9 a 0.0 0 0.00 0.0 0 0.00 0.0 Cediflad by APPENDIX C TORQUE CURVES APPENDIX D MOTOR DATA SHEETS MODEL NO. CATALOG NO. PHASE TYPE FRAME NA NA 3 HUEI 6 8P ORDER NO. 20068291 LINE NO. 100 MPI: 100422 HP: 350 POLES: 14 VOLTS: 460 H2: 60 SERVICE FACTOR: 1,15 EFFICIENCY ( %): S.F. 91 FULL 91 3/4 91,2 1/2 89.7 1/4 83.5 POWER FACTOR S.F. 74 FULL 72,7 3/4 68 1/2 58.3 1/4 38.5 NO LOAD 2,9 LOCKED ROTOR i24.5 AMPS: S.F. 560 FULL 495 3/4 396 1/2 314 1/4 255 NO LOAD 234,5 LOCKED ROTOR 2761 NEMA CODE LETTER G NEMA DESIGN LETTER R FULL LOAD RPM 505 NEMA NOMINAL EFFICIENCY ( %) 91 GUARANTEED EFFICIENCY ( %) 89.5 MAX KVAR 158.8 AMBIENT ( °C) 50 ALTITUDE (FASL) 3300 SAFE STALL TIME -HOT (SEC) 30 SOUND PRESSURE (DBA 0 1M) 80 TORQUES: BREAKDOWN(% F.L) 175 LOCKED ROTOR(% F.L.) 60 FULL LOAD(LB -FT) 13630. 4 11- I7:LZ1R1eI-1[1Ff[ li3]Il•7fiTZ^,fPf-�'. - - -. a _�� EMERSON MOTOR COMPANY i IF Y L U EMERSON. ST. LOUIS, MO 'MOTORS M1.iolar ,c ,naloci�s — Emerson Motor Company is a division of Emerson Electric C i. The Emerson logo is a trademark and service mark of Emerson Electric Co. AftacI E® ent "D" LMEE & RO, Inc. May 15, 2017 Mr. Michael S. Ho, P.E. Deputy Director of Public Works /City Engineer City of Seal Beach 211 Eighth Street Seal Beach, CA 90740 1199 South Fullerton Road, City of Industry, CA 91748 Tel: (626) 912 -3391 Fax: (626) 912 -2015 www.LEE- RO.com Subject: Third -Party Peer Review of West End Pump Station Audit Report Opinion Letter Dear Mr. Ho: LEE & RO staff have thoroughly read the West End Storm Water Pump Station Audit Report prepared by AKM and have evaluated AKM's analysis and recommendations found in the report. The attached opinion letter summarizes the findings of our third -party peer review of the West End Storm Water Pump Station Audit Report and is stamped and signed by Lee Badertscher, the lead engineer performing the peer review. We appreciate the City providing us the opportunity be of service to you. Should you have any questions, please do not hesitate to call. Very truly yours, LEE &/ RO, Inc. x — 4 _ Rick Liskow, PMP Principal Engineer Each Opinion Letter ]1PROAl 160 Misc. Small Projects \1160.03 City of Seal Beach West End Storm Water PS Report Review\I PROAG2 Client Correspondence \Opinion Cover Letter.docx The City of Seal Beach West End Pump Station Audit authorized AKM Consulting Engineers (AKM) to: 1. Document the history of the pump station and past studies 2. Review the pump failures in December 2016 and February 2017 and document the chronology of events from December 21, 2016 through February 20, 2017 3. Evaluate the existing equipment and assess their capacities and conditions 4. Review the City's past maintenance practices and provide recommendations for future maintenance of the facilities 5. Respond to questions and comments by the community This was presented by AKM to the City in the West End Pump Station Audit Report (Report). LEE & RO has reviewed the report and has evaluated AKM's analysis and recommendations. This opinion letter presents our findings. Section numbers and headings below are the same as presented in AKM's report. Section 2 - Pump Station History and Past Studies AKM states in the Report that the original West End Pump Station (pump station) design does not meet many of the standards included in the Hydraulic Institute Standards and the Los Angeles County Flood Control District's Pump Station Design Manual. LEE & RO recommends that AKM specifically identify the non - compliant design features and the specific associated standard that applies. Section 3 — Chronology of Events 3.9 Start-up of Rebuilt Pumps, Thursday, February 2, 2017 AKM reports that the minimum speed setting at the pump V FDs was set at 30 Hz, which is 50% of full speed, and that the pump took 60 seconds to ramp up to minimum speed when started. AKM also reports that the ramp up time was decreased to 10 seconds and that the minimum speed setting was increased to 40 Hz, or 67% of full speed. LEE & RO concurs with AKM's recommendations. See discussion on Section 4.4 — VFDs. Section 4 — Eouioment Evaluation 4.1 Pumps AKM opined that the pumps seized as a result of trash and/or debris getting lodged between the propeller and bowl. LEE & RO believes that AKM's evaluation and analysis of the pump failures is accurate. AKM provided a number of recommendations to minimize the chance of future pump failures due to trash and debris. LEE & RO concurs with those recommendations. 4.2 Inlet Storm Drains It is reported that, although the design capacity of the pump station is 200 cfs (10 -year storm), the existing tributary storm drain system cannot convey more than 115 cfs to the pump station. AKM recommends that in order to mitigate flooding, storm drain and catch basin improvements need to be implemented in order to convey the existing capacity to the pump station. Although the pumps can discharge 200 cfs from the pump station, we concur that if the pump station inlet system can convey no more than 115 cfs to the wet well, flooding is likely in a storm event that generates more than 115 cfs of water. Page I 1 LEE & RO, Inc. 1: \PROD \1160 Misc. Small Projects \1160.03 City of Seal Beach West End Storm Water PS Report Review \1 PROJ \G2 Client Correspondence \Opinion Letter.doa 4.3 Motors AKM offered preventative maintenance recommendations for the motors. LEE & RO believes that AKM's recommendations are reasonable. 4.4 Variable Frequency Drives (VFDs) AKM's evaluation of the VFDs concluded that VFD 1 needs to be reprogrammed to eliminate false alarms, both VFDs should be programmed to include the time /date calendar so that an accurate record of alarm events can be maintained, the ramp up time of 60 seconds is too long and increases the chances for debris to become entangled with the pump propeller, and the pump minimum speed setting at 50% of full needs to be increased to help move debris through the pump more quickly. AKM reports in Section 3.9 that the ramp up time was reduced from 60 seconds to 10 seconds. AKM also reports in Section 3.11 that the ramp up time was further reduced from 10 seconds to 4 seconds. The acceleration time limits the current and the deceleration time prevents excessive voltage. Although LEE & RO concurs with these implemented changes, practical experience indicates that shorter times require testing to ensure that over - current and under voltage alarms are avoided at shortened times. 4.5 Control System AKM found that both control panels were operating properly, but that the control panel system is not capable of automatically alternating pump starts. AKM also found that the data captured by the City's SCADA system was inaccurate and recommended that the City's SCADA system consultant implement corrective measures so that the SCADA system operates properly. While we concur with AKM's recommendations, we would like to point out that PLCs are commonly used to control pumps and alternate pump starts. Currently, alternating the pump starts is done manually. We would also recommend that the SCADA radio system be upgraded to allow it to communicate with the maintenance yard. There may be line of sight issues between the pump station and the maintenance yard that make communication difficult. It may be possible to transmit the signals from the pump station to City Hall and then route them from City Hall to the maintenance yard. LEE & RO recommends that the City develop a comprehensive control strategy. 4.6 Standby Generator We concur with the AKM recommendation to test the generator once per year under loaded conditions. 4.7 Trash Rack AKM identified several issues with the trash rack at the pump station. Recommendations were made to either replace the existing trash racks or reduce the size of the openings at the lower levels of the rack and to seal openings at the bottom and on the sides of the existing trash rack. Because trash and debris appears to be a significant problem and the likely reason for the pump failures, it is very important to reduce the amount and size of debris that enters the pump station wet well. We concur with AKM's recommendations to either replace or make improvements to the pump station inlet trash rack to reduce the debris passing through and entering the wet well. Page 12 LEE & RO, Inc. 1: \PROJ \1160 Misc. Small Projects \1160 -03 City of Seal Beach West End Storm Water PS Report Review \1 PROJ \G2 Client Correspondence \Opinion Letter.doa Section 5 — Maintenance Program AKM recommended a number of additional maintenance practices to improve the City's preparedness for a major storm event. We believe that AKM's recommendations for both the pump station and for the storm drain system are reasonable and prudent. AKM also recommends the City to conduct operator training on a regular basis to ensure the equipment is properly maintained and operated. This is a standard practice for most Agencies, Districts and Municipalities. Section 6 — Responses to Community Comments We find nothing problematic with the responses provided to the community comments and questions. LEE & RO, Inc. Lee Badertscher, P.E. Managing Engineer eOpDERTOT�gy�%bc\ W � No. 0 043789 t} EXP. �laTFOF cAl1F���\P� Page 13 LEE & RO, Inc. J: \PROJ \1160 Misc. Small Projects \1160 -03 City of Seal Beach West End Storm Water PS Report Review \1 PROJ \G2 Client Correspondence \Opinion Letter.doa