Minutes of the Special Meeting on URTDSM / SCADA / Communication held on

Size: px

Start display at page:

Download "Minutes of the Special Meeting on URTDSM / SCADA / Communication held on"

Irma Daniels
5 years ago
Views:

3 Minutes of the Special Meeting on URTDSM / SCADA / Communication held on Introduction A Special Meeting on URTDSM / SCADA / Communication was held on at SR-II, PGCIL, Bengaluru. The list of participants is enclosed at Annexure-I. In the 141 st OCC meeting held on , it was noted that there were frequent outage of ICCP links and the ICCP data was not stable. APTRANSCO and TSTRANSCO had expressed concern on the operational and commercial implications due to this. There were other SCADA/Communication issues along-with URTDSM Project Phase-I was nearing completion and it was agreed to take stock of the situation in a special meeting. The agenda was taken up for deliberation. 2. URTDSM 2.1 Status of URTDSM project as on Region Approved Installed Integrated SR-I SS SR-I PMU SR-II SS SR-II PMU Total PMU SR SR-I - 9 PMU Chittor (5) and Srikakulam (4) installation works under progress. SR-I - 5 PMU -Srisailam (4) and Kurnool (1) data validation is not completed. SR-I - 1 PMU -Simhadri (1) - SRLDC requested SR-1 to co-ordinate with NTPC Simhadri to complete the pending installation works. SR-II -2 PMU -Kaiga (2) installation works under progress. SR-II -5 PMU -Arasur (2) and Kalpakkam (3) data validation is not completed. 2.2 Supply and installation of APS- Auxiliary power supply system as part of URTDSM project SRLDC vide letter dated (Annexure-II) had requested to SR-II to expedite the supply Auxiliary Power Supply System. SR-II had intimated that the Auxiliary Power Supply System is under manufacturing and is expected to reach site by 20 th April 2018 and commissioning will be completed by 15 th May 2018 AP and TS also expressed concern regarding delay in supply of Auxiliary power supply system. SR-I confirmed that the APS System is expected to reach site in April 2018 and commissioning was expected by May MOM of URTDSM SCADA - COMMUNICATION Page 1

4 2.3 Status of SAT and SAVT of URTDSM project in SR SRLDC - SAT completed and SAVT is yet to commence. KSEBL - SAT and SAVT completed. KPTCL - SAT and SAVT completed. TNEB - SAT completed and SAVT is yet to commence TSTRANSCO - SAT completed and SAVT has commenced from 1 st March 2018 APTRANSCO - Commissioning works under progress expected to be completed by 31 st March Powergrid Analytical Application installed as part of URTDSM SRLDC vide letter dated (Annexure-III) had informed Smart Grid PGCIL the need to improve the performance of installed Analytical Application to make it useful for real time grid operations SRLDC requested the forum to consider a procedure on the lines of SAT for verifying the working of the Analytical Applications. It was noted that proper feedback could be given for improvement of the Analytical Application Software. All the control centers were requested to use the software and report the deficiencies. PGCIL agreed to make a Web group of all the Control Centers, LD&C, IIT Mumbai, SR-I/SR-II etc for exchanging the information/usage and for deficiencies / rectification etc. It was noted that Chief Engineer, NPC vide letter dated (Annexure-IV) had sought certain information on the Analytics and implementation of PMUs from CTU. 3. SCADA and Communication Issues 3.1 ICCP links failure / not stable APTRANSCO vide letter dated (Annexure-V) had expressed concern on the ICCP data not being received for long durations due to apparent failure of PGCIL communication network and redundancy failure at various Network/Server equipment at SRLDC. Concern was expressed on maintaining the overdrawal /under drawal without the ICCP data and the huge commercial implications. Besides that many 765 kv Powergrid s SS SCADA data is not good and affecting the state CGS drawals. In view of the above, the following was suggested: To discuss SCADA/Communication issues of Powergrid SR1 & SR2, states and also SRLDC s redundancy readiness of ICCP servers/routers. To discuss the issue that the Payments may be linked to availability and healthiness of Powergrid s communication network. SRLDC s ICCP equipment and percentage availability of ICCP data to the State. TSTRANSCO vide letter dated (Annexure-VI) had expressed concern on the ICCP data not being stable for most of the period during the month. On 7 th & 8 th MOM of URTDSM SCADA - COMMUNICATION Page 2

5 February 2018, ICCP data failed due to expiry of SSL certificates of secure ICCP. Again on , the ICCP data failed from 22:00 hrs and it was restored only at 08:30 hrs of The economical load dispatch operation solely lies on ICCP data in real time including safety and security of the grid. It was pointed out that the loss of real time data for long time periods affects the CGS drawal severely and attracts penalties. To avoid such problems in future there was a necessity to discuss these situations in detail and make a resolution. SRLDC observation on ICCP failure are given below: Secure ICCP certificates expired throughout SCADA systems maintained by M/s GE (T & D) in India. ICCP data exchange unaffected for Karnataka & TN SLDCs (connected on non secure ICCP) SRLDC SCADA team tried to resolve issue with known methods Upon non restoration of ICCP data exchange to/from affected SLDCs, issue was reported to GE Noida and ticket open for problem resolution by SRLDC. ICCP data exchange restored in morning hours starting with AP, Kerala, Telangana & Puducherry. Data restored in non secure mode. ICCP certificates were restored by GE on 15 th Feb SRLDC suggested that affected SLDCs need to raise issue with M/s GE (T&D) directly for timely resolution and coordinate for early resolution. Further all were requested to analyse the problem before arriving at any conclusion. SRLDC observation on 23 rd Feb 2018 (22:30 hrs onwards) ICCP failure is given below: ICCP communication with AP & TS went down. Multiple fibre cuts in CTU system were reported. PGCIL was alerted in real time. Links were restored in morning hours (around 08:30 Hrs), followed by ICCP restoration. Subsequently following was taken up jointly by SRLDC & APSLDC/TSSLDC : AP & TS SCADA team informed SRLDC that in certain situations (ICCP server 2 being main at SRLDC, and ICCP server 1 being main at AP/TS end), ICCP data is not reporting even when the ICCP links are up. AP & TS SCADA teams claimed problem due to misconfiguration at SRLDC servers/routers/firewalls. FPRs raised by SRLDC for resolution of TS & AP ICCP issues. Issue resolved for both AP & TS ICCP by attending erroneous entries in AP firewall & TS routers. Subsequently following was taken up jointly by SRLDC & APSLDC/TSSLDC for ICCP Resolution : AP ICCP Resolution steps: AP internal firewall mis-configuration rectified. AP ICCP application error rectified. MOM of URTDSM SCADA - COMMUNICATION Page 3

6 AP SISCO security outdated certificate configured with new certificates. TS ICCP Resolution steps: Encapsulation PPP enable in TS WR1 router for serial link interface. Deleted all the static routing from both TS WR routers and TS ICCP servers which was enforced for SRLDC ICCP LAN. APTRANSCO expressed concern on frequent ICCP failures recently. This failure was likely due to communication link and other issues. Due to high CGS schedule, ICCP data is very important for state. Due to failure of ICCP, AP SLDC control room is not aware of its OD/UD. After secure license updation one of ICCP server it was not able to communicate on ICCP protocol. SCADA personnel should be available in shifts to solve any issue related to SCADA. APTRANSCO requested to waive off penalty for UD/OD, which happened due to ICCP failure. It was pointed out that many communication channel has been configured for ICCP data communication between AP SLDC and SRLDC. M/s Tejas Engineers have configured 6 No of communication channel with same ID, which creates problem in troubleshooting after link outage. Many changes have been done in communication network by Powergrid (M/s Tejas), without information of AP Telecom. Expiry date of all licenses to be taken from M/s GE. APTRANSCO requested to check the healthiness of server and link redundancy for ICCP data transfer. Recently complete redundancy test was done for server, Router, link and found to be OK. TSTRANSCO expressed similar concern as stated by APTRANSCO. ICCP failure outage duration on was longer, which causes penalty for states due to OD/UD. KPTCL informed that ICCP with SRLDC failed on On prima facie analysis it was analyzed that patch updation was the issue. Subsequently on further analysis it was found that number of points being handled in ICCP database was huge and it had created problem. Matter was resolved on priority basis with the help of OEM, M/s ABB Engineers. During above outage real time scada data available on web server provided by SRLDC helped KPTCL in grid operation. TANTRANSCO requested to waive off penalty for UD/OD, which happened due to ICCP failure. APTRANSCO / TSTRANSCO stressed on 24 hrs shift at SRLDC /SR-I / SR-II for SCADA and communication issues in similar lines to the SLDCs. SRLDC pointed out that 24 hrs support was being extended from office and through remote login. It was confirmed that there were some mis-configuration at AP int-firewall. It was rectified by M/s GE team upon ticket raised for rectification. It was confirmed that there were some mis-configuration at TS ICCP router. The same was attended by M/s GE Engineers on FPR raised by SRLDC. It was noted that ICCP failure on was due to power supply interruption of equipment at Hoody substation of KPTCL. SRLDC stated that data is extremely important for reliable and secure grid operation and all the concerned entities needed to resolve the issue at their end and also coordinate with other entities. MOM of URTDSM SCADA - COMMUNICATION Page 4

7 SRLDC suggested AP & TS to actively raise all SCADA/communication related complain to GE/CTU/STU and not leaving the matter only to SRLDC for resolution. SR-I & SR-II pointed out that all ICCP link failures may not be due to communication. ICCP problem occurred on was not due to fibre cut in Munirabad/Raichur area. After discussion the following was agreed: Expiry date of all licenses would be furnished by M/s GE to the respective Control Centers. It was noted that respective Control Center would ensure that M/s GE take pro-active action to renew the licenses before the expiry. As pointed out by KPTCL that there should not be any license renewal issues during the project cycle of the project and it needed to be ensured by Control Centers (through M/s GE). License for external firewall is going to expire in three months. M/s GE has to update the certificates in ext firewall of AP. M/s GE to take necessary action. M/s GE site engineer at respective Control Centers to escalate the matter to BO- Noida, if they are unable the resolve the issue at their level. Further, inter-action between M/s GE site engineers at SLDCs and M/s GE site engineers at SRLDC needs to be strengthened. Changeover to ICCP data communication through Backup SRLDC could be explored during such conditions which would reduce data outage duration. All the ICCP links (Main/Back up CC of SLDCs to Main/Back up CC of SRLDC) to be checked daily to ensure healthiness. All SLDCs to carry out HA test once in every month to a check SCADA/Communication system healthiness. All states to actively raise all SCADA/communication related complain to M/s GE/CTU (M/s Tejas) /STU and not leaving the matter only to SRLDC for resolution. It was noted there was no provision for DSM waiver during ICCP failure as per extant Regulations. Updated communication network diagram to be submitted by SR-I and SR-II to all SLDCs & SRLDC. Further any changes done in the communication network needed to be informed to all. 3.2 Frequent data outage (Central Sector & State Sector) SRLDC presented the detail of data outage from to (Annexure-VII). It was pointed out that most of the outage was due to communication failure. SRLDC requested Powergrid to provide all fibre cut/equipment outage information in real time. TSTRANSCO expressed concern that due to frequent failure of Powergrid station (Nizamabad, Warangal etc.), TS drawl gets affected and real time CGS drawl becomes erratic. SR-I & SR-II stated that all data outages may not be due to communication issues. MOM of URTDSM SCADA - COMMUNICATION Page 5

8 SR-I & SR-II stated that DCPC healthiness also needed to be checked. For this SRLDC replied that all DCPC patches etc. has been done and at present one of DCPC at Narendra is having the problem, which is being looked by M/s GE for faster resolution After deliberations the following was agreed: PGCIL to share their observations for communication failure on the instances of data outage beyond 8 hrs (refer annexure) put up by SRLDC. PGCIL to share all fibre cut/equipment outage information in real time. It was suggested to take all measure to minimize down time of data 3.3 KKNPP analog data not reporting It was noted that presently Analog data issue has been resolved after Analog Card (as supplied by PGCIL SR2) replacement and is under observation. The forum opined that if the problem persists, it would be prudent for KKNPP to procure a new RTU. 3.4 Data outage after commissioning In earlier meeting it was agreed to cross check with site daily values for newly commissioned elements for a period of two week duration from commissioning. 3.5 Data discrepancy The latest data discrepancy list compiled by SRLDC was presented in the forum (Annexure-VIII). The forum opined to delete the points which are not feasible to be telemetered. SRLDC agreed to look into the issue. 3.5 RTU/SAS Maintenance/Upgradation All states/powergrid/cgs to explore option of replacement of old RTUs and AMC for SAS, as getting spare is difficult for old RTU. The issue has already been discussed earlier and AMC need to be provided for all RTU/SAS gateways by CTU/STU. A life/functional assessment report and action plan by PGCIL/ISGS on old RTUs would be prepared and submitted to SRLDC/SRPC. 3.6 Backup Control Centre Backup SCADA functionality testing frequency needs to be done once in 6 months to ensure high availability of data/preparedness to operate from Back up CC in event of any contingency. Compliance report may be furnished to SRLDC. Periodical testing was being carried out by SRLDC and KSEBL. TN stated that communication network is being strengthened to Backup CC. Once same is completed, TN will do the testing. APTRANSCO informed that communication link has to be established from AP CC at Vijaywada to Back Up CC of SRLDC at Delhi. Powergrid & AP confirmed to establish the same by MOM of URTDSM SCADA - COMMUNICATION Page 6

9 PGCIL would be floating new combined Open Tender for Backup SLDCs for AP (Tirupati), TS (Warangal) along-with few other states. After discussions, it was noted that backup control centre establishment for AP & TS need to be expedited. 3.7 ICCP with NTAMC PGCIL confirmed that as pointed out in 32 nd TCC/33 rd SRPC meeting, due to cyber security concerns data would not be extended to SRLDC from NTAMC ICCP. It was noted that functional requirement of the requisite data by SRLDC would remain, which needed to be ensured by PGCIL. SRLDC would furnish the list of telemetry points that have to be extended to PGCIL within 10 days, and within 1 month PGCIL would come out with a detailed plan to be completed within 6 months (October 2018) 3.8 Modification in EMS package TSTRANSCO Earlier TSTRANSCO informed that they had received estimate of Rs 47 lakhs and the contract was being firmed up. TSTRANSCO requested PGCIL to assist in finalizing the EMS contract through M/s GE. SR-I stated that TSTRANSCO to interact directly with M/s GE without PGCIL involvement. SRPC requested that PGCIL to render assistance to TSTRANSCO if possible. 3.9 Status review of compliance of approved protocol of handling scada & communication issues ( communication meeting) In the earlier meeting, it had been noted that RTU not reporting/data failure would be communicated to the concerned entity/user with a copy to PGCIL. User would take lead role in getting the issue resolved. User/SRLDC would ensure necessary verification at their ends as explained in the training. Parallely, PGCIL would look into communication issues. User/SRLDC would appraise PGCIL about the verification at their end for PGCIL to sort out the communication issues. Official ID for SCADA issues to be created for each entity/user involved for easier communication and proper delivery of information. SRLDC pointed out that there were instances of delay in attending the faults wrt the time schedule and requested all the constituents to attend to the faults within the agreed time schedules. It was noted that if the issue is not getting resolved within 8 hrs (3 hrs for ICCP) the following action would be taken: Shri A P Gangadharan, GM(AM), SR-II and Shri S Ravi, GM(AM), SR-I would be informed for necessary follow up. The issue would be escalated with M/s G E (both by SLDCs and SRLDC) and with M/s Tejas (by SR-I & SR-II). MOM of URTDSM SCADA - COMMUNICATION Page 7

10 In case the issue was as not resolved within 24 hrs physical involvement of OEM (M/s GE and M/s Tejas) would be ensured through effective coordination by user / PGCIL/ SRLDC. Sustained efforts/involvement of all the concerned entities to be ensured till the issue is resolved In case of hardware replacement / major rectification it would be completed by 72 hrs Sparing of Fibre in Central Sector Project TANTRANSCO again requested for sparing of fibre in Central Sector Project. It was noted that the issue was already deliberated in the 30 th TCC/31 st SRPC meetings held on 24/ wherein, the following was noted: PGCIL had informed that in most of the FO, there were no spare fibres available. Six fibres are available for grid management purposes. The balance 18 fibres in the state or central sector are left to the entities on how to utilize. As for Central sector, 18 fibres are being leased out to M/s Powertel as per CERC formula which covered issues like sparing of fibres, cost sharing (50% being shared by M/s Powertel), ROW compensation (Rs.3000/km) etc. For grid operation purpose, bandwidth on 6 fibres could be given to the states. M/s Powertel may not take all the 18 fibres in some of the lines but those fibres are being leased out anywhere. Keeping in view of the non availability of spare fibres in most of FO links and the approved MoU with M/s Powertel the scheme of allocation the fibres was dropped. SR-II stated that request for sparing of fibre in new FO links could be communicated for consideration VC issues TSTRANSCO pointed out that Video Conference issue at TSSLDC. There was disturbance in Video Conference at TS end when connected to SRLDC. SRLDC confirmed as per AMC Engineer no issue was observed at SRLDC end and software upgradation is pending at TS end. After deliberations, TSSLDC and SRLDC were requested to coordinate and sort out the issue Talcher RTU It was noted that earlier proposed meeting was not held due to overlapping holidays. It was noted that a meeting with NTPC Talcher, SR2 ULDC and SRLDC for the same in is proposed 3 rd /4 th week of March SR-II was kindly requested to address the communication issue between Talcher-II & HVDC. NTPC as a User needed to be proactive in this. 4. SE (O), SRPC thanked all for the interactive deliberations and requested the Entities for concerted/sustained action to close the issues highlighted in the Meeting. MOM of URTDSM SCADA - COMMUNICATION Page 8

SOUTHERN REGIONAL POWER COMMITTEE BENGALURU. Record Notes of the Meeting to discuss disruption of power supply affecting BMRCL services

SOUTHERN REGIONAL POWER COMMITTEE BENGALURU Record Notes of the Meeting to discuss disruption of power supply affecting BMRCL services A Meeting to discuss disruption of power supply affecting BMRCL services