RE: Docket Electric Infrastructure, Safety, and Reliability Plan FY 2013 Responses to Commission Data Requests Set 2

Transcription

1 Luly Massaro FY 2012 Electric ISR Plan December 20, 2010 Thomas R. Teehan Senior Counsel March 12, 2012 VIA HAND DELIVERY & ELECTRONIC MAIL Luly E. Massaro, Commission Clerk Rhode Island Public Utilities Commission 89 Jefferson Boulevard Warwick, RI RE: - Electric Infrastructure, Safety, and Reliability Plan FY 2013 Dear Ms. Massaro: Enclosed are ten (10) copies of National Grid s responses to the Commission s Second Set of Data Requests in the above-captioned proceeding. Thank you for your attention to this transmittal. If you have any questions, please feel free to contact me at (401) Very truly yours, Enclosure Thomas R. Teehan cc: Service List Steve Scialabba Leo Wold, Esq.

2 The Narragansett Electric Company d/b/a National Grid Commission 2-1 Request: Please provide a copy of pole agreements between National Grid and Verizon, specifically including cost responsibilities for poles with joint ownership. Response: Attached are the joint pole ownership agreements between Narragansett Electric (National Grid) and Verizon, including the Intercompany Operating Procedures. Prepared by or under the supervision of: Jennifer L. Grimsley

3 Attachment COMM 2-1 Page 1 of 54

4 Attachment COMM 2-1 Page 2 of 54

5 Attachment COMM 2-1 Page 3 of 54

6 Attachment COMM 2-1 Page 4 of 54

7 Attachment COMM 2-1 Page 5 of 54

8 Attachment COMM 2-1 Page 6 of 54

9 Attachment COMM 2-1 Page 7 of 54

10 Attachment COMM 2-1 Page 8 of 54

11 Attachment COMM 2-1 Page 9 of 54

12 Attachment COMM 2-1 Page 10 of 54

13 Attachment COMM 2-1 Page 11 of 54

14 Attachment COMM 2-1 Page 12 of 54

15 Attachment COMM 2-1 Page 13 of 54

16 Attachment COMM 2-1 Page 14 of 54

17 Attachment COMM 2-1 Page 15 of 54

18 Attachment COMM 2-1 Page 16 of 54

19 Attachment COMM 2-1 Page 17 of 54

20 Attachment COMM 2-1 Page 18 of 54

21 Attachment COMM 2-1 Page 19 of 54

22 Attachment COMM 2-1 Page 20 of 54

23 Attachment COMM 2-1 Page 21 of 54

24 Attachment COMM 2-1 Page 22 of 54

25 Attachment COMM 2-1 Page 23 of 54

26 Attachment COMM 2-1 Page 24 of 54

27 Attachment COMM 2-1 Page 25 of 54

28 Attachment COMM 2-1 Page 26 of 54

29 Attachment COMM 2-1 Page 27 of 54

30 Attachment COMM 2-1 Page 28 of 54

31 Attachment COMM 2-1 Page 29 of 54

32 Attachment COMM 2-1 Page 30 of 54

33 Attachment COMM 2-1 Page 31 of 54

34 Attachment COMM 2-1 Page 32 of 54

35 Attachment COMM 2-1 Page 33 of 54

36 Attachment COMM 2-1 Page 34 of 54

37 Attachment COMM 2-1 Page 35 of 54

38 Attachment COMM 2-1 Page 36 of 54

39 Attachment COMM 2-1 Page 37 of 54

40 Attachment COMM 2-1 Page 38 of 54

41 Attachment COMM 2-1 Page 39 of 54

42 Attachment COMM 2-1 Page 40 of 54

43 Attachment COMM 2-1 Page 41 of 54

44 Attachment COMM 2-1 Page 42 of 54

45 Attachment COMM 2-1 Page 43 of 54

46 Attachment COMM 2-1 Page 44 of 54

47 Attachment COMM 2-1 Page 45 of 54

48 Attachment COMM 2-1 Page 46 of 54

49 Attachment COMM 2-1 Page 47 of 54

50 Attachment COMM 2-1 Page 48 of 54

51 Attachment COMM 2-1 Page 49 of 54

52 Attachment COMM 2-1 Page 50 of 54

53 Attachment COMM 2-1 Page 51 of 54

54 Attachment COMM 2-1 Page 52 of 54

55 Attachment COMM 2-1 Page 53 of 54

56 Attachment COMM 2-1 Page 54 of 54

57 The Narragansett Electric Company d/b/a National Grid Commission 2-2 Request: Please provide a copy of the current Terms and Conditions related to customer Contributions in Aid of Construction ( CIAC ). Response: The Company s Terms and Conditions for Distribution Service, R.I.P.U.C. No (See Section 3), Policy 1, Line Extension Policy for Individual Residential Customers, Policy 2, Line Extension Policy for Residential Developments and Policy 3, Line Extension and Construction Advance Policy for Commercial, Industrial and Existing Residential Customers are included as Attachment 1 Commission 2-2. Prepared by or under the supervision of: Jeanne A. Lloyd

58 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 1 of 26 R.I.P.U.C. No Sheet 1 Canceling R.I.P.U.C. No A THE NARRAGANSETT ELECTRIC COMPANY TERMS AND CONDITIONS FOR DISTRIBUTION SERVICE The following Terms and Conditions where not inconsistent with the rates are a part of all rates. The provisions of these Terms and Conditions apply to all persons, partnerships, corporations or others (the Customer) who obtain local distribution service from The Narragansett Electric Company (the Company) and to companies that are nonregulated power producers, as defined in Rhode Island General Laws. All policies, standards, specifications, and documents referred to herein have been filed with the Rhode Island Public Utilities Commission (Commission) and Division, and such documents and any revisions have been filed at least 30 days before becoming effective. Compliance by the Customer and nonregulated power producer is a condition precedent to the initial and continuing delivery of electricity by the Company. Service Connection 1. The Company shall furnish on request detailed information on the method and manner of making service connections. Such detailed information may include a copy of the Company s Specifications for Electrical Installations booklet, as may be amended from time to time, a description of the service available, connections necessary between the Company s facilities and the Customer s premises, location and access of service connection facilities and metering equipment, and Customer and Company responsibilities for installation of facilities. The Customer shall wire to the point designated by the Company, at which point the Company will connect its facilities. In addition, the Customer s facilities shall comply with any reasonable construction and equipment standards required by the Company for safe, reliable, and cost efficient service. For a service meeting Company requirements (which requirements are set forth on the Company s website at the Company may also permit this connection to be made by a licensed electrician in good standing with the authority having jurisdiction, as required by applicable law, and who is registered with the Company, provided, however, that the Company gives no warranty to the Customer, express or implied, as to the knowledge, training, reliability, honesty, fitness, or performance of any electrician registered with the Company for this purpose, and the Company shall not be liable for any damages or injuries caused by any electrician who may be used for such purpose. Application for Service 2. Application for new service or alteration to an existing service should be made as far in advance as possible to assure time for engineering, ordering of material, and construction. Upon the Company s reasonable request, the Customer shall provide to the Company all data and plans reasonably needed to process this application. S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\Terms & Conditions doc

59 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 2 of 26 R.I.P.U.C. No Sheet 2 Canceling R.I.P.U.C. No A Line Extensions [Overhead (OH) & Underground (UG)] 3. The Company shall construct or install overhead or underground distribution facilities or other equipment determined by the Company to be appropriate under the following policies: Line Extension Policy for Residential Developments, Line Extension Policy for Individual Residential Customers, and Line Extension and Construction Advance Policy for Commercial, Industrial and Existing Residential Customers. Whenever it is necessary to provide service and a Customer requests the Company to extend or install poles, distribution lines or other service equipment to the Customer s home, premises or facility in order to supply service, the Company will furnish the necessary poles, wires, or equipment in accordance with the Company s Line Extension and Construction Advance Policies on file with the Commission. Except as provided in the Policies, all such equipment, poles, and wires shall remain the property of the Company and be maintained by it in accordance with the Policies. To the extent that any Company property needs to be located on private property, the Company will require the Customer to furnish a permanent easement. Attachments 4. Any individual or organization who requests an attachment to distribution facilities, utility poles, or along any span between such poles, shall comply with the Company s specifications and policies governing the type of construction, metering, attachment fees, easements, permissions and electrical inspections required. Outside Basic Local Distribution Services 5. Customers requesting the Company to arrange for Customer facility outages or additional maintenance or construction not normally part of basic local distribution service will be notified in a reasonable timely manner by the Company that the customer shall be required to pay these the Company s costs of reasonably meeting the request. Acquisition of Necessary Permits 6. The Company shall make, or cause to be made, application for any necessary street permits, and shall not be required to supply service until a reasonable time after such permits are granted. The Customer shall obtain or cause to be obtained all permits or certificates, except street permits, necessary to give the Company or its agents access to the Customer s equipment and to enable its conductors to be connected with the Customer s equipment. Service to Out-Building 7. The Company shall not be required to install service or meter for a garage, barn or other outbuilding, so located that it may be supplied with electricity through a service and meter in the main building. S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\Terms & Conditions doc

60 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 3 of 26 R.I.P.U.C. No Sheet 3 Canceling R.I.P.U.C. No A Customer Furnished Equipment 8. The Customer shall furnish and install upon its premises such service conductors, service equipment, including circuit breaker if used, and meter mounting device as shall conform with specifications issued from time to time by the Company, and the Company will seal such service equipment and meter mounting device, and adjust, set and seal such circuit breaker, and such seals shall not be broken and such adjustments or settings shall not be changed or in any way interfered with by the Customer. The Customer shall furnish and maintain, at no cost to the Company, the necessary space, housing, fencing, and foundations for all equipment that is installed on its premises in order to supply the Customer with local distribution service, whether such equipment is furnished by the Customer or the Company. Such space, housing, fencing, and foundations shall be in conformity with the Company s specifications and subject to its approval. Up-Keep of Customer Equipment 9. The Customer s wiring, piping, apparatus and equipment shall, at all times, conform to the requirements of any legally constituted authorities and to those of the Company, and the Customer shall keep such wiring, piping, apparatus and equipment in proper repair. Installation of Meters 10. Meters of either the indoor or outdoor type shall be installed by the Company at locations to be designated by the Company. The Company may at any time change any meter installed by it. The Company may also change the location of any meter or change from an indoor type to an outdoor type, provided that the cost of the change shall be borne by the Company except when such change is pursuant to the provisions of Paragraph 11. Upon the reading of the Company s meter all bills shall be computed. If more than one meter is installed, unless it is installed at the Company s option, the monthly charge for local distribution service delivered through each meter shall be computed separately under the applicable rates. Unauthorized and Unmetered Use 11. Whenever the Company determines that an unauthorized and unmetered use of electricity is being made on the premises of a Customer and is causing a loss of revenue to the Company, the Company may, at the Customer s expense, make such changes in the location of its meters, appliance and equipment on said premises as will, in the opinion of the Company, prevent such unauthorized and unmetered use from being made. Definition of Month 12. Whenever reference is made to month in connection with electricity delivered or payments to be made, it shall mean the period between two successive regular monthly meter readings or estimated meter readings, the second of which occurs in the month to which reference is made. S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\Terms & Conditions doc

61 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 4 of 26 R.I.P.U.C. No Sheet 4 Canceling R.I.P.U.C. No A If the Company is unable to read the meter when scheduled, the necessary billing determinants may be estimated. Bills may be rendered on such estimated basis and will be payable as so rendered. Payment Due Date -- Interest Charge 13. All bills shall be due and payable upon receipt. Bills rendered to customers, other than individually metered residential customers, on which payment has not been received by the Avoid Interest Date as shown on the bill, shall bear interest, at the rate of 1¼% per month on any unpaid balance, including any outstanding interest charges, from the date of receipt until the date of payment. The Avoid Interest Date corresponds to the next normal bill preparation date. Bills disputed in good faith by a Customer will not be subject to the late payment charge until after the dispute is resolved. Customer payment responsibilities with their nonregulated power producer will be governed by the particular Customer/nonregulated power producer contract. Payments made through the Company for electricity purchased from a nonregulated power supplier will be applied first to any Narragansett charges or arrearages. Returned Check Fee 14. A $15.00 Fee shall be charged to the Customer for each check presented to the Company that is not honored by the financial institution. This fee shall be applicable only where the check has been dishonored after being deposited for a second time. Seasonal Customers 15. Seasonal Customers are those using local distribution services between June lst and September 30th only, or those using local distribution services principally between June lst and September 30th and incidentally or intermittently during the rest of the year. Deposit and Security 16. The Company may require a cash deposit or other collateral satisfactory to it as security for prompt payment of the Customer s indebtedness to the Company. The rate of interest shall be adjusted on March lst annually. The interest rate in effect in any year shall be based on the average rate over the prior calendar year for 10-year constant maturity Treasury Bonds as reported by the Federal Reserve Board. Payments for Line Extensions 17. The Company may require a Customer to pay for all or a portion of the cost of extending or installing poles, distribution lines, or equipment to the Customer s home, premises or facility, consistent with the terms of the Company s Line Extension and Construction Advance Policies on file with the Commission. S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\Terms & Conditions doc

62 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 5 of 26 R.I.P.U.C. No Sheet 5 Canceling R.I.P.U.C. No A Lighting Service Charge 18. The Company may assess a Lighting Service Charge of $ for Company services rendered in response to a Customer request in support of Customer equipment where the condition, service or connection is unrelated to the performance of facilities owned by the Company. A Lighting Service Charge per each occurrence will be assessed to the Customer on their subsequent bill. Determining Customer s Demand 19. The demand is the maximum rate of taking electricity. Under ordinary load conditions it will be based upon one or more fifteen-minute peaks as herein defined. A fifteen-minute peak is the average rate of delivery of electricity during any fifteen-minute period as determined by any suitable instrument chosen by the Company. In the case of extremely fluctuating load, however, where the demand based on the average over fifteen minutes does not fairly represent the maximum demand imposed by the Customer, the demand will be based upon the instantaneous peak or the peak for a shorter period than fifteen minutes. Such measurements will be made by any suitable instrument chosen by the Company. The demand which is billed to the Customer is determined according to the terms of the appropriate tariffs approved by the PUC from time to time. Customer Changing Rates 20. The Customer may change from the rate under which he is purchasing electricity to any other rate applicable to a class of service which he is receiving. Any change, however, shall not be retroactive, nor reduce, eliminate or modify any contract period, provision or guarantee made in respect to any line extension or other special condition. Nor shall such change cause such service to be billed at any rate for a period less than that specified in such rate except during the first year of electric service to any Customer. A Customer having changed from one rate to another may not again change within twelve months or within any longer contract period specified in the rate under which he is receiving electric service. Discontinuance of Service 21. Subject to the Rules and Regulations of the Commission, the Company shall have the right to discontinue its service upon due notice and to remove its property from the premises in case the Customer fails to pay any bill due the Company for such service, or fails to perform any of its obligations to the Company. For restoration of service after such discontinuance, a reconnection charge of $38.00 will be made. Right of Access 22. The Company shall have the right of access to the Customer s premises at all reasonable times for the purpose of examining or removing the Company s meters, and other appliances and equipment. During emergency conditions, the Company shall have the right of access to the S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\Terms & Conditions doc

63 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 6 of 26 R.I.P.U.C. No Sheet 6 Canceling R.I.P.U.C. No A Customer s premises at all hours of the day to make conditions safe and/or to restore service. Safeguarding Company Equipment 23. The Customer shall not permit access for any purpose whatsoever, except by authorized employees of the Company, to the meter or other appliances and equipment of the Company, or interfere with the same, and shall provide for their safe keeping. In case of loss or damage of the Company s property, the Customer shall pay to the Company the value of such property or the cost of making good the same. Temporary Service 24. A temporary connection is local distribution service which does not continue for a sufficient period to yield the Company adequate revenue at its regular local distribution service rates to justify the expenditures necessary to provide such a connection. The Company may require a Customer requesting a temporary connection to pay the full amount of the estimated cost of installing and removing the requested connection, less estimated salvage value, in advance of the installation of the connection by the Company. In addition, the customer shall pay the applicable regular local distribution service and, if applicable, basic or standard offer service rates. Limitation of Liability for Service Problems 25. The Company shall not be liable for any damage to equipment or facilities using electricity which damage is a result of Service Problems, or any economic losses which are a consequence of Service Problems. For purposes of this paragraph, the term Service Problems means any service interruption, power outage, voltage or amperage, fluctuations, discontinuance of service, reversal of its service, or irregular service caused by accident, labor difficulties, condition of fuel supply or equipment, federal or state agency order, failure to receive any electricity for which the Company has contracted, or any other causes beyond the Company s immediate control. However, if the Company is unable for any reason to supply electricity for a continuous period of two days or more, then upon the request of the Customer, the Demand Charge, if any, shall be suspended for the duration of such inability. The Company shall not be liable for damage to the person or property of the Customer or any other persons resulting from the use of electricity or the presence of the Company s appliances and equipment on the Customer s premises. Limitation on Use of Electricity - Auxiliary & Temporary Local Distribution Service 26. Local distribution service supplied by the Company shall not be used to supplement or relay, or as standby or back up to any other electrical source or service except under the provisions of the Back-Up Service Rate, unless the Customer shall makes such guarantees with respect to the payment for such local distribution service as shall be just and reasonable in each case. Where such local distribution service is supplied, the Customer shall not operate its generation in S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\Terms & Conditions doc

64 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 7 of 26 R.I.P.U.C. No Sheet 7 Canceling R.I.P.U.C. No A parallel with the Company s system without the consent of the Company, and then only under such conditions as the Company may specify from time to time. Company Right to Place Facilities on Customer Property 27. The Company has the right to place on a Customer s property facilities to provide and meter electric service to the Customer. Company Right to Request a Guarantee 28. Whenever the estimated expenditures for the services or equipment necessary to deliver electricity to a Customer s premises shall be of such an amount that the income to be derived therefrom at the applicable rates will, in the opinion of the Company, be insufficient to warrant such expenditures, the Company may require a Customer to guarantee a minimum annual payment or commitment for a term of years, or to pay the whole or a part of the cost of such equipment. Fluctuating Load & Harmonic Distortion 29. In certain instances, extreme fluctuating loads or harmonic distortions which are created by a Customer s machinery or equipment may impair service to other Customers. If the fluctuating load or harmonic distortion causes a deterioration of the Company s service to other customers, the Company shall specify a service arrangement that avoids the deterioration and the Customer owning or operating the equipment that causes the fluctuation or distortion shall pay the cost to implement the new service arrangement together with applicable taxes. Customer Tax Liability 30. The Company shall collect taxes imposed by governmental authorities on services provided or products sold by the Company. It shall be the Customer s responsibility to identify and request any exemption from the collection of the tax by filing appropriate documentation with the Company. Customer/Supplier Relationship 31. For electricity supplied by nonregulated power producers, the Company is a local distribution service provider of electricity supplied by others. When such electricity is supplied and delivered to the Company s local distribution supply point, the Company then performs a delivery service for the electricity. Ownership of such electricity lies with either the nonregulated power producer or Customer, as per the specific agreement between the Customer and the nonregulated power producer. In no case shall the Company be liable for loss of electricity. Customer Notice and Right to Appeal 32. Where practicable, the Company will give the Customer reasonable notice of actions taken S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\Terms & Conditions doc

65 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 8 of 26 R.I.P.U.C. No Sheet 8 Canceling R.I.P.U.C. No A pursuant to these Terms & Conditions. The Customer shall have the right to appeal, pursuant to the Division s Rules of Practice and Procedure, all action taken by the Company hereunder. Effective: October 15, 2010 S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\Terms & Conditions doc

66 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 9 of 26 THE NARRAGANSETT ELECTRIC COMPANY POLICY 1 LINE EXTENSION POLICY FOR INDIVIDUAL RESIDENTIAL CUSTOMERS When an individual residential customer ( Customer ) requests that a distribution line be extended to serve such customer s home whether over private property, along common way or along a public way, the terms of this policy shall apply. This policy applies only to the installation of electric service by Narragansett Electric Company ( Company ). The Customer should contact other utilities to determine the utilities requirements governing the provision of their service and whether any costs and/or requirements are to be the responsibility of the Customer. 1. Installation of Overhead Distribution Line The Company will provide a regular overhead 120/240 volts, single phase, 3 wire service up to a capacity limit of 50 kva for the Customer. The Company will determine the route of the distribution line in consultation with the Customer. 2. Distance of Overhead Distribution Line Allowed Without Charge The Company will provide up to two poles and two spans of overhead distribution line needed to serve the Customer plus a service drop (that does not require a carrier pole) to the Customer s home free of charge. 3. Overhead Line Extension If more than two poles and two spans of overhead distribution line are required to serve the Customer s home, the Customer will pay an Overhead Installation Charge, as determined below. The Overhead Installation Charge will be equal to the number of feet of distribution line (beyond two poles and two spans) required to serve the Customer s home, multiplied by the Overhead Cost Per Foot (as defined in section 9 below), plus the applicable tax contribution factor. When overhead service is requested, the Company shall be responsible for: i. installing (or having others install), owning (individually or jointly) and maintaining (individually or jointly) all poles, primary and secondary wires, transformers, service drops, meters, etc. that, in its opinion are required to provide adequate service; ii. designating the location of all Company owned equipment, excluding streetlights, and the service entrance and meter location(s); S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\NECO Policy 1.doc

67 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 10 of 26 Policy 1 Page 2 of 5 iii. blasting and tree trimming and removal along public ways; the Company may charge the Customer the cost of such blasting and tree trimming and removal if, in the Company s opinion, such cost is excessive; excessive cost shall be defined as the type of work which requires the Company to contract with a third party to remove ledge through blasting or to trim trees for the purposes of clearing the space needed for the line work. The Customer, at no cost to the Company, shall be responsible for: 4. Payment Terms i. blasting and tree trimming and removal on private property, including roadways not accepted as public ways by the municipality, in accordance with the Company s specifications and subject to the Company s inspection. For Overhead Installation Charges up to $6,000, the Customer will be required to pay the entire amount before the start of construction. If the Overhead Installation Charge is greater than $6,000, the Customer will have the option to either pay the entire amount before the start of construction, or pay $6,000 before the start of construction and pay the amount in excess of $6,000 under a payment plan. The term of the payment plan will be based on equal payments of at least $75 per month until the amount in excess of $6,000 is paid in its entirety. The term of the payment plan is not to exceed a period of five (5) years or sixty (60) months. The amount collected under the payment plan will include interest at the rate of interest applicable to the Company s customer deposit accounts. 5. More Than One Customer Where overhead service is requested by more than one Customer for the same line, the Overhead Installation Charge will be prorated among those Customers, based on the amount of line attributable to each Customer. (The calculation of the Overhead Installation Charge shall allow for a credit equal to the Overhead Cost Per Foot of two poles and two spans for each Customer). 6. Customer Added After Initial Construction If a new Customer (or group of customers) is supplied service from facilities constructed under this policy, and if such service begins within five (5) years from the date of the first payment received by the Company from the original Customer or group of Customers, the Company will require such new Customer(s) to make prorated contribution to payment of the balance of the Overhead Installation Charge. Any contribution received from a new Customer will be used to proportionately reduce the balance owed by the initial Customers(s). In addition, a credit of two poles and two spans per customer will be applied against the remaining balance. However, no refunds will be paid if the credit exceeds the balance. 7. Change of Customer S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\NECO Policy 1.doc

68 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 11 of 26 Policy 1 Page 3 of 5 The Customer must agree, as a condition for the line extension monthly payment terms, that if he/she sells, leases or otherwise transfers control and use of the home to another individual ( New Occupant ), and such New Occupant opens a new account with the Company, the Customer will obtain an agreement from such New Occupant to pay the remaining balance as prescribed in the agreement of the Overhead Installation Charge that would have been owed by the Customer at that location. Otherwise, the Customer will remain personally liable for the balance owed. The Company reserves the right to place a lien on the property until such time that the obligation is fulfilled. 8. Underground Lines If the Customer requests an underground distribution line in lieu of the standard overhead line, the Company will give reasonable consideration to the request. If the Company believes that there are technical complications, safety issues, engineering concerns, or other reasonable concerns regarding the feasibility and/or maintenance of an underground system in the given circumstances, the Company may decline to provide underground service. If the Company agrees to an underground service, the Company will estimate the cost of providing the underground line to the home, using a predetermined underground cost per foot ( Underground Cost Per Foot ). The Customer will be required to pay an Underground Charge equal to: (A) the Company s estimated cost of installing the underground line (based on Underground Cost Per Foot); minus (B) an amount equal to the Overhead Cost Per Foot of two poles and two spans; plus (C) a tax contribution factor based on the value of donated property and/or any cash contribution. When the above results in a negative number, there shall be no Underground Charge. The Underground Charge shall be paid by the customer in advance of the Company s construction (even if it exceeds $6,000) and is nonrefundable if the line is built. The Customer will be responsible for removal of ledge, trenching, backfilling in accordance with the Company s construction standards and/or the Information & Requirements for Electric Service as published by the Company from time to time, and shall comply with codes and requirements of legally constituted authorities having jurisdiction. In addition, the Customer will be responsible for: i. providing, prior to the start of the Company s construction, all applicable documents and site plans required for the Company to prepare design S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\NECO Policy 1.doc

69 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 12 of 26 drawings and easements for its facilities to be installed on private property; Policy 1 Page 4 of 5 ii. iii. iv. providing and installing all required foundations (except for Company owned street light foundations), handholes, manholes, grounding systems, secondary cable, all conduit including spacers, glue and pulling strings, etc. as indicated on the Company s plan and related construction documents; installing foundations, provided by the Company, for Company owned street lights; v. supplying copies of all invoices, when requested, indicating manufacturer and part number for all such equipment listed above; equipment that is not approved shall not be used without the prior written consent of the Company; vi. vii. installing, owning, and maintaining all secondary services and service conduit from the Company s equipment to the designated meter location(s); and turning over ownership of the conduit system, excluding the service conduit, to the Company upon inspection and acceptance of the conduit system by the Company. When underground service is requested, the Company shall be responsible for: i. supplying a list of approved manufacturers and their part numbers for equipment to be supplied by the Customer; ii. iii. iv. designating the location of all Company owned equipment, excluding street lights, and the service entrance and meter location(s); providing Company owned street light foundations; providing, installing, owning and maintaining the transformer, Company owned street lights, meter and primary cable; v. making all connections to Company equipment; and vi. inspecting the underground conduit system and equipment foundations installed by the Customer, prior to backfilling. 9. Publication of Current Per Foot Costs The Overhead Costs Per Foot and Underground Costs Per Foot for new construction S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\NECO Policy 1.doc

70 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 13 of 26 Policy 1 Page 5 of 5 shall be as calculated by the Company and placed on file with the Public Utilities Commission. These costs are included in the attachment to this policy. 10. Tree Trimming The Customer will be responsible for all necessary tree trimming on private property. Tree trimming along public ways and common ways will remain the responsibility of the Company but may cause additional charges to be billed to the Customer if the type of work requires the Company to contract with a third party to trim trees for the purposes of clearing the space needed for the line work. 11. Line Extension Agreement The Company will require the Customer to sign a Line Extension Agreement setting forth the terms of this policy and any other terms that the Company deems are reasonably necessary in connection with the installation line to the Customer s home, provided that such terms are not inconsistent with the terms expressed in this policy. 12. Temporary Service This policy shall not apply to lines constructed for temporary service, unless the Company, in its sole discretion, deems it appropriate in the given circumstances of each case. 13. Winter Moratorium on Underground Construction From the period of December 15 to April 1, the Company may decline, in its sole discretion, to install any underground facilities. 14. Easements The Company will, as a condition on the installation of the service, require the Customer to provide the Company with an easement (drafted by the Company) for all facilities located on private property. 15. Customer Request to Upgrade Service When, in the Company s opinion, the Company is required to upgrade its distribution line, or any associated equipment, in order to accommodate a Customer s upgrade of the existing main switch to the Customer s premises, the terms of Policy 3 shall apply. Effective: February 1, 2006 S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\NECO Policy 1.doc

71 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 14 of 26 Policy 1 Attachment Narragansett Electric Company Per Foot Costs for Line Extensions For Individual Residential Customers The Per Foot Costs referred to in the above policy are as follows: Underground Cost Per Foot: $22.68 Overhead Cost Per Foot: $15.03 These costs are effective until the Rhode Island Public Utilities Commission is notified in writing of any changes (with a copy of the written notice provided to the Rhode Island Builders Association). Effective: September 1, 2006 S:\RADATA1\RATE ADMINISTRATION\Tariffs_Current\Narragansett\NECO Policy 1.doc

72 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 15 of 26 THE NARRAGANSETT ELECTRIC COMPANY POLICY 2 LINE EXTENSION POLICY FOR RESIDENTIAL DEVELOPMENTS When a developer, contractor, builder or other customer ( Developer ) proposing to construct a residential development or individual homes requests that distribution lines be constructed to serve the development or homes, the terms of this policy shall apply. This policy applies only to the installation of electric service by Narragansett Electric Company ( Company ). The Customer should contact other utilities to determine the utilities requirements governing the provision of their service and whether any costs and/or requirements are to be the responsibility of the Customer. 1. Installation of Overhead Distribution Lines The Company will provide a regular overhead distribution line to the development or individual homes designed to provide regular residential service to each home proposed in the project. The Company will determine the route of the line in consultation with the Developer. The Developer shall wire to the point designated by the Company, at which point the Company will connect its facilities. In addition, the Developer s facilities shall comply with the Company s construction standards and/or the Information & Requirements for Electric Service as published by the Company from time to time and shall comply with codes and requirements of legally constituted authorities having jurisdiction. 2. Distance of Overhead Distribution Line Allowed Without Charge The Company will provide 150 feet of overhead distribution line, not including the secondary service drop, per each house lot free of charge. 3. Overhead Line Extension If the number of centerline feet of overhead distribution line required to serve the development ( Required Line Distance ) is greater than the Allowed Distance of 150 feet per House Lot, then there will be a charge to the Developer for the overhead line extension for the additional feet ( Overhead Installation Charge ). The additional charge shall be paid by the Developer in advance of the Company s construction. The Overhead Installation Charge will be equal to the Overhead Cost Per Foot times the number of feet in excess of the Allowed Distance of 150 feet per House Lot (plus applicable tax contribution factor). When overhead service is requested, the Company shall be responsible for: i. installing (or having others install), owning (individually or jointly) and maintaining (individually or jointly) all poles, primary and secondary wires, transformers, service drops, meters, etc. that, in its opinion are required to provide adequate service; S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 2.doc

73 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 16 of 26 Policy 2 Page 2 of 5 ii. iii. designating the location of all Company owned equipment, excluding streetlights, and the service entrance and meter location(s); blasting and tree trimming and removal along public ways; the Company may charge the Customer the cost of such blasting and tree trimming and removal if, in the Company s opinion, such cost is excessive; excessive cost shall be defined as the type of work which requires the Company to contract with a third party to remove the ledge through blasting or to trim trees for the purposes of clearing the space needed for the line work. The Developer, at no cost to the Company, shall be responsible for: i. blasting and tree trimming and removal on private property, including roadways not accepted as public ways by the municipality, in accordance with the Company s specifications and subject to the Company s inspection. The Overhead Cost Per Foot will be a predetermined cost per foot as calculated by the Company. The Overhead Installation Charge is nonrefundable if the line is built. 4. Underground Lines A Developer may request an underground distribution line in lieu of the regular overhead line. If requested, however, the Company will estimate the cost of providing the underground line to the development using a predetermined underground cost per foot ( Underground Cost Per Foot ). The Developer will be required to pay an Underground Charge equal to: (A) the difference between the estimated underground construction cost (based on Underground Cost Per Foot) and the estimated construction cost for a regular overhead line (based on the Overhead Cost Per Foot); plus (B) the Overhead Installation Charge, if any, that would have been paid for an overhead line in the development as calculated in Section 3 above; plus (C) a tax contribution factor based on the value of donated property and/or any cash contribution. The Underground Charge shall be paid by the Developer in advance of the Company s construction and is nonrefundable if the line is built. The Developer will be responsible for removal of ledge, trenching and backfilling in accordance with the Company's construction standards and/or the Information & Requirements for Electric Service as published by the Company from time to time and shall comply with codes and requirements of legally constituted authorities having jurisdiction. In addition, the Developer will be responsible for: S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 2.doc

74 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 17 of 26 Policy 2 Page 3 of 5 i. providing, prior to the start of the Company s construction, all applicable documents required for the Company to prepare design drawings and easements for its facilities to be installed on private property; ii. iii. iv. providing and installing all required foundations (except for Company owned street light foundations), handholes, manholes, grounding systems, all conduit including spacers, glue and pulling strings, etc. as indicated on the Company s plan and related construction documents and in accordance with the Company s specifications; installing foundations, provided by the Company, for proposed street lighting based on a plan approved, in writing, by a Municipality, which includes agreement by that Municipality to accept responsibility for payment of the lights once the lights are energized; supplying copies of all invoices, when requested, indicating manufacturer and part number for all such equipment listed above; equipment that is not approved shall not be used without the prior written consent of the Company; v. installing, owning, and maintaining all secondary services and service conduit from the Company s equipment to each designated meter location; and vi. turning over ownership of the conduit system, excluding the service conduit, to the Company upon inspection and acceptance of the conduit system by the Company. When underground service is requested, the Company shall be responsible for: i. developing the plan to provide underground electric service; ii. iii. iv. supplying a list of approved manufacturers and their part numbers for equipment to be supplied by the Customer; designating the location of all Company owned equipment, excluding street lights, and the service entrance and meter location(s); providing Company owned street light foundations; v. providing, installing, owning and maintaining all transformers, Company owned street lights, primary and secondary cable, except services; vi. making all connections to Company equipment; and S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 2.doc

75 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 18 of 26 Policy 2 Page 4 of 5 vii. inspecting the underground conduit system and equipment foundations installed by the Customer, prior to backfilling. 5. Publication of Current Per Foot Costs The Overhead Costs Per Foot and Underground Costs Per Foot for new construction shall be as calculated by the Company and placed on file with the Public Utilities Commission. These costs are included in the attachment to this policy. The Company also will provide such Overhead and Underground Costs Per Foot and the method of calculating the applicable tax contribution factor to anyone who inquires. If the Company changes the Overhead and Underground Cost Per Foot or method of calculating the tax contribution factor, it will notify the Commission in writing and provide a copy of the written notice to the Rhode Island Builders Association, if in existence. The Company will not increase the Overhead or Underground Costs Per Foot by more than 10% per year without specific approval from the Commission and advance notice to the Rhode Island Builders Association, if in existence. 6. Developer Provides Plans and Documentation The total number of house lots proposed to be constructed ( House Lots ) will be provided in advance to the Company by the Developer (prior to the Company building the distribution line), along with an electronic copy (in a format acceptable to the Company) of the subdivision plan approved by the planning board in the applicable community. The Company may require the Developer to provide, in advance, the following: (A) (B) (C) (D) (E) (F) a copy of the approval of the planning board for the subdivision; a copy of all permits and approvals that have been obtained for constructing the development; the name and address of the bank or credit union providing financing for the development, including a contact person and phone number; a copy of a city/town-approved street light proposal for the development. If installation is requested after construction is completed, the actual, incremental cost of installing the street lights may be borne by the city/town if the tariff does not collect all costs of construction. a schedule or Developer s best estimate for the construction of homes in the development; and if requested by the Company, such other reasonable information that may be requested to confirm the viability of the development. S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 2.doc

76 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 19 of 26 Policy 2 Page 5 of 5 7. Building the Distribution Line in Segments The Company may, in its own discretion, construct the distribution line in segments, rather than all at once in the proposed development. 8. Line Extension Agreement The Company will require the Developer to sign a Line Extension Agreement setting forth the terms of this policy and any other terms that the Company deems are reasonably necessary in connection with the installation of a distribution line in the development, provided that such terms are not inconsistent with the terms expressed in this policy. 9. Winter Moratorium on Underground Construction From the period of December 15 to April 1, the Company may decline, in its sole discretion, to install any underground facilities. 10. Easements The Company will require the Developer to provide the Company with easements (drafted by the Company) for all facilities to reach and serve the development. Effective: February 1, 2006 S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 2.doc

77 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 20 of 26 Policy 2 Attachment Narragansett Electric Company Per Foot Costs for Line Extensions For Residential Developments The Per Foot Costs referred to the above policy are as follows: Underground Cost Per Foot: $22.68 Overhead Cost Per Foot: $15.03 These costs are effective until the Rhode Island Public Utilities Commission is notified in writing of any changes (with a copy of the written notice provided to the Rhode Island Builders Association). Effective: September 1, 2006 S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 2.doc

78 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 21 of 26 THE NARRAGANSETT ELECTRIC COMPANY POLICY 3 LINE EXTENSION AND CONSTRUCTION ADVANCE POLICY FOR COMMERCIAL, INDUSTRIAL AND EXISTING RESIDENTIAL CUSTOMERS The terms of this policy shall apply when a commercial, industrial or non-residential (a real estate development which is not an approved subdivision of single-family homes) customer ( Customer ) requests that a distribution line and/or other facilities ( New Facilities ) necessary to properly supply electricity to the Customer s facilities be installed. This policy applies only to the installation of electric service by Narragansett Electric Company ( Company ). The Customer should contact other utilities to determine the utilities requirements governing the provision of their service and whether any costs and/or requirements are to be the responsibility of the Customer. The terms of this policy shall also apply to an individual residential customer whose upgrade of the existing main switch to his/her premises will, in the Company s opinion, require the Company to upgrade its distribution line or associated equipment. In applying this policy, the Company will estimate any additional incremental revenue that may be realized as a result of the upgraded service for the purposes of determining whether a Construction Advance is required from the residential customer. 1. Amount of Overhead Distribution Provided without Charge If the New Facilities being requested by the Customer consists of an overhead, single phase, secondary voltage distribution line extension that does not exceed two poles and two spans of line, the Company will provide the poles and spans of line needed to serve the New Facilities plus a service drop (that does not require a carrier pole) free of charge to the Customer. Otherwise, the costs of all poles and spans of line determined by the Company as needed to serve the New Facilities will be included in the cost component of the Construction Advance Formula described below. 2. Estimated Revenue Before undertaking the construction of the New Facilities to serve the Customer, the Company will estimate the annual incremental revenue to be derived by the Company under the local distribution service rates from the installation of the New Facilities. 3. Construction Advance The Company will determine the facilities required to meet the distribution service requirements of the Customer. Facilities in excess of those required to meet the distribution service requirements of the Customer are outside the scope of this policy and may entail additional payments from the Customer. In accordance with the formula below (the Formula ), the Company shall determine S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 3.doc

79 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 22 of 26 Policy 3 Page 2 of 6 whether a payment by the Customer of a Construction Advance shall be required. The Construction Advance shall be paid by the customer in advance of the Company s construction. where Construction Advance (A) = [ C - [ D x M ] K ] A= Construction Advance paid to the Company by the Customer. C= The total estimated cost of construction for facilities required exclusively to meet the distribution service requirements of the Customer. This cost includes capital and non-capital costs and the Company s liability for tax required on the value of the material and labor provided by the Customer. Where these new or upgraded facilities are not solely to provide service to the Customer, the Company shall appropriately apportion these costs. D= For a single customer, the estimated annual Distribution Revenue derived from the Customer within the first year following the completion of the Company s construction of facilities; or for developments, the estimated additional annual Distribution Revenue derived from those new customers in the development anticipated to be supplied directly with electric service within one year from the commencement of the delivery of electricity to the first customer in the development. M= 0.5, the revenue apportionment factor. K= The annual carrying charge factor, expressed as a decimal. Where the calculation of (A) results in a positive number, a Construction Advance in the amount of (A) shall be required from the Customer. Where the calculation of (A) results in a negative number, (A) shall be considered to be zero. Where the calculation of (A) results in a Construction Advance of $500 or less, the payment of the Construction Advance will be waived. The Company shall exercise good faith in making each estimate and determination required above. Any revenues from Transmission Service, the Non-Bypassable Transition Charge, Standard Offer Service, Last Resort Service, and the Demand Side Management Charge shall be excluded from this calculation. The Construction Advance in the formula shall be further adjusted to include a tax contribution factor on the cash value of the Construction Advance, excluding the value of the tax contribution on any donated property received from the Customer. This tax contribution factor shall be paid in full by the Customer prior to the start of construction. S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 3.doc

80 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 23 of 26 Policy 3 Page 3 of 6 4. Refund Whenever the Company collects a Construction Advance from the Customer, the Customer has the option to request the Company to perform a one-time recalculation of the Construction Advance payment using actual construction costs and actual Distribution Revenue to determine if a refund of all or a portion of the original payment is warranted. The request for the one-time review may be made at any time between twelve and thirty-six months after commencement of the delivery of electricity. To determine the refund, the Formula shall be modified as follows: C= The actual cost of construction. If the actual cost of construction exceeds the estimate, then the estimated cost of construction shall be used. This cost includes capital and non-capital costs and the Company s liability for tax required on the value of the material and labor provided by the Customer. Where these new or upgraded facilities are not solely to provide service to the Customer, the Company shall appropriately apportion these costs. D= The actual annual Distribution Revenue for the most recent twelve months. M= 0.5, the revenue apportionment factor. K= The annual carrying charge factor, expressed as a decimal. If a lower or negative (A) results from applying the Formula as so modified, and if, in the Company s opinion, a risk does not exist regarding either a future reduction in the level of the Customer s usage or the collectability of the Customer s account, then the Company shall refund a portion of or the entire calculated Construction Advance, or the full cost of construction, without interest. In no case shall the amount refunded exceed the original Construction Advance (A); nor shall the review result in additional payments from the Customer. If a refund is made, the Company will refund the appropriate portion of any tax contribution factor at the current tax rate. 5. Overhead Line Extension When overhead service is requested, the Company shall be responsible for: i. installing (or having others install), owning (individually or jointly) and maintaining (individually or jointly) all poles, primary and secondary wires, transformers, service drops, meters, etc. that, in its opinion are required to provide adequate service; ii. designating the location of all Company owned equipment, excluding streetlights, and the service entrance and meter location(s); S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 3.doc

81 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 24 of 26 Policy 3 Page 4 of 6 iii. blasting and tree trimming and removal along public ways; the Company may charge the Customer the cost of such blasting and tree trimming and removal if, in the Company s opinion, such cost is excessive; excessive cost shall be defined as the type of work which requires the Company to contract with a third party to remove ledge through blasting or to trim trees for the purposes of clearing the space needed for the line work. The Customer, at no cost to the Company, shall be responsible for: i. blasting and tree trimming and removal on private property, including roadways not accepted as public ways by the municipality, in accordance with the Company s specifications and subject to the Company s inspection. The Company may, at its discretion, construct the distribution line in segments rather than all at once in the proposed development. 6. Underground Lines (A) (B) If the Customer requests an underground distribution line in lieu of the standard overhead line, the Company will give reasonable consideration to the request. If the Company believes that there are technical complications, safety issues, engineering concerns, or other reasonable concerns regarding the feasibility and/or maintenance of an underground system in the given circumstances, the Company may decline to provide underground service. If the Company agrees to underground service, the Customer will be responsible for removal of ledge, trenching and backfilling in accordance with the Company s construction standards and/or the Information & Requirements for Electric Service as published by the Company from time to time and shall comply with the codes and requirements of legally constituted authorities having jurisdiction. In addition, the Customer will be responsible for: i. providing, prior to the start of the Company s construction, all applicable documents and electronically formatted site plans required for the Company to prepare design drawings and easements for its facilities to be installed on private property; ii. providing and installing all required foundations (except for Company owned street light foundations), handholes, manholes, grounding systems, secondary cable, all conduit including spacers, glue and pulling strings, etc. as indicated on the Company s plan and related construction documents and in accordance with the Company s specifications; S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 3.doc

82 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 25 of 26 Policy 3 Page 5 of 6 iii. iv. Installing foundations, provided by the Company, for Company owned street lights; supplying copies of all invoices, when requested, indicating manufacturer and part number for all such equipment listed above; equipment that is not approved shall not be used without the prior written consent of the Company; v. retaining ownership of transformer foundations and grounding systems, and all secondary cables and conduit on private property, excluding Company owned street lighting; and vi. turning over ownership of the conduit system, excluding the secondary conduit, to the Company upon inspection and acceptance of the conduit system by the Company. When underground service is requested, the Company shall be responsible for: i. developing the plan to provide underground electric service; ii. iii. iv. supplying a list of approved manufacturers and their part numbers for equipment to be supplied by the Customer; designating the location of all Company owned equipment, excluding street lights, and the service entrance and meter location(s); providing Company owned street light foundations; v. providing, installing, owning and maintaining all transformers, primary cable, related primary equipment, Company owned street lights, and meters; vi. vii. making all connections to Company equipment; and inspecting the underground conduit system and equipment foundations installed by the Customer, prior to backfilling. 7. Winter Moratorium on Underground Construction From December 15 to April 1, the Company may decline, in its sole discretion, to install any underground facilities. 8. Easements The Company will require the Customer to provide the Company a permanent easement (drafted by the Company) for all facilities to reach and serve the New Facilities. S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 3.doc

83 Attachment 1 - Commission 2-2 RIPUC Docket No FY 2013 Electric Infrastructure Safety and Reliability Plan Page 26 of 26 Policy 3 Page 6 of 6 9. Additional Payment When, in the Company s opinion, significant engineering is required to determine the method of service or prepare construction estimates, the Company will estimate the cost of such engineering. The Company may charge the Customer this cost before engineering begins. If construction is undertaken, this payment will be applied to any required Construction Advance. If construction is not undertaken, the Company will refund any balance not spent. If no Construction Advance is required, the entire additional advance payment will be refunded. Effective: February 1, 2006 S:\RADATA1\Tariffs_Current\Narragansett\NECO Policy 3.doc

84 The Narragansett Electric Company d/b/a National Grid Commission 2-3 Request: How does the Company account for CIAC in Schedule WRR-1? Response: Historically, over 90% of Contributions in Aid of Construction (CIAC) are received within the "New Business", "Damage/Failure" and "Public Requirements" categories. A significant portion of the budgets for these mandatory categories, including the Blanket Projects and Reserves for Emergent Projects, are based on net trends in spending, and therefore, the CIACs embedded in the estimated total spending and capital additions for the fiscal year cause a reduction to those figures. This serves to reduce the overall Electric ISR revenue requirement. Prepared by or under the supervision of: William Richer

85 The Narragansett Electric Company d/b/a National Grid Commission 2-4 Request: Where National Grid has indicated that CIAC may not be applied in the same fiscal year as the expenditure, please explain how it is later credited to the project. Response: Section 5 of the Electric ISR filing, Exhibit 1 JLG, Section 5 Revenue Requirement, page 2 of 7 states: It is important to note that these proposed amounts will be trued up to actual investment activity after the conclusion of the respective FY, with rate adjustments for the revenue requirement differences incorporated in future ISR filings. As described in the Company s response to Commission 2-3, estimated Contributions in Aid of Construction (CIAC) are embedded in the estimate of total spending and capital additions for the fiscal year as a reduction to the overall spending and additions for that year, which serves to reduce the overall ISR revenue requirement. If the CIAC is received in the year of the project spending, it will be reflected as a net reduction to the overall actual spending in that year for the true up referenced above. If the CIAC is received in the year prior to the spending or the year subsequent to the spending, the CIAC will be reflected as a net reduction to overall actual ISR spending in the year that it is received for the true up referenced above. Prepared by or under the supervision of: William Richer

86 The Narragansett Electric Company d/b/a National Grid Commission 2-5 Request: Referencing page 43 of the 2013 Electric ISR Proposal, the Company discusses budgeting for major storms. Please explain the relationship between this budgeting and fund availability from the Storm Fund. Response: Based on the reference above it would appear that this question relates to the following statement: Major Storms Each year, the Company carries a budgeted project for major storm activity that affects the Company s assets. While the actual spend in this category may vary greatly, this reserve, based on average trends over the past several years, allows the Company to avoid removing other planned work from the capital program when replacement of assets due to weather is required. The storm fund is designed to normalize the impact on customer rates that results from incurring qualifying incremental Operation and Maintenance expense that is typically incurred during major storm preparation and restoration activities. It is not designed to mitigate capital costs that can be incurred during major storm events. Therefore, any recovery associated with capital investments included in the 2013 Electric ISR Proposal would be separate and apart from the storm fund. Prepared by or under the supervision of: Jennifer L. Grimsley

87 The Narragansett Electric Company d/b/a National Grid Commission 2-6 Request: Referencing page 45 of the 2013 Electric ISR Proposal and page 35 of the Responses to Division DR 2-4, please provide the age of each of the battery systems and chargers for which the Company has information. Response: Please see the graph below for the age profile for the Company s batteries and chargers. Count Distribution Substation Battery and Charger Age Profile Rhode Island Year of Installation Prepared by or under the supervision of: Jennifer L. Grimsley Batteries Chargers

88 The Narragansett Electric Company d/b/a National Grid Commission 2-7 Request: Referencing page 60 of the 2013 Electric ISR Proposal, please explain how the Emergent Reliability Project Reserve was derived. Response: On page 60 of the 2013 Electric ISR Proposal, the Emergent Reliability Project Reserve is described as follows: This reserve replaces the Pockets of Poor Performance Strategy. This reserve will be used to fund projects that are identified by the review of localized reliability issues. The goal is to identify and correct repeat device interruptions and to help identify future reliability hotspots and support the timely correction of localized problems before they become larger issues. The Company is placing $250,000 in this reserve for these projects in FY The Emergent Reliability Project Reserve is derived to ensure that sufficient capital is available to begin the planning and design work associated with reliability projects work identified during the fiscal year, but not included in the 2013 Electric ISR proposal. Typically, capital fixes for smaller local reliability issues are not known at the time the budget is built, yet they are still necessary and important for the Company to correct. In the past, the Company had the Pocket for Poor Performance (PPP) project budgeted for such items. It was determined that instead of the PPP project, the Company would budget a reserve for emergent or smaller reliability issues which arose and need to be engineered or constructed in a relatively short time frame but were not in the original budget/plan. The Emergent Reliability Project Reserve was set up for this purpose and was estimated at approximately 10% of the average year s non-program reliability project spend of $2.5M. Prepared by or under the supervision of: Jennifer L. Grimsley

89 The Narragansett Electric Company d/b/a National Grid Commission 2-8 Request: Please provide a copy of the KEMA Consulting Study referenced on page 60 of the 2013 Electric ISR Proposal. Response: The Company was actively engaged with KEMA in 2005 to review the costs and benefits of developing Enterprise Level Substation Systems Integration (ELSSI). The ELSSI is an Enterprise Systems Architecture for substation data, including the architecture within the substation, the communications infrastructure, and the enterprise virtual data mart that would make substation data accessible to applications and users throughout National Grid. The Company has determined that the study done by KEMA Consulting did not contain the finding that SCADA systems, when used to monitor and control the distribution feeder breakers, can provide a 15 percent to 20 percent reduction in average customer outage duration (CAIDI) when compared with a similar feeder that is not equipped with SCADA facilities. Following the review, the Company chose not to implement an enterprise solution; however, it did pursue the expansion of the Company s EMS system. Although the scope of the engagement with KEMA in 2005 greatly exceeded the expansion of the Company s EMS system, reports and presentations from that engagement are attached, as Attachment COMM 2-8 (a), Attachment COMM 2-8 (b), and Attachment COMM 2-8 (c). As a part of the review, the potential for improvements in CAIDI was mentioned. In Attachment COMM 2-8 (a), an expected improvement of 10 minutes was thought possible for Targeted Dispatching. A ten-minute improvement in RI would equate to a 14 percent improvement compared to the Company s 2011 results. However, this improvement was based on not only EMS (SCADA monitoring and control) but also other technologies including fault locating. As such, the 15 to 20 percent improvement for EMS alone is likely overestimated. Prepared by or under the supervision of: Jennifer L. Grimsley

90 The Narragansett Electric Company d/b/a National Grid Attachments COMM 2-8 (a), (b), (c) Please be advised that Attachment COMM 2-8(a) is 46 pages, Attachment COMM 2-8(b) is 160 pages, and Attachment COMM 2-8(c) is 103 pages. Due to the voluminous nature of these attachments, the Company is providing the Commission with three (3) copies and three (3) CD-ROM s containing Attachments COMM 2-8 (a), (b), (c).

91 K E M A, I N C. Attachment COMM 2-8(a) Page 1 of 46 Enterprise Level Substation System Integration Final Presentation Findings and Recommendations National Grid USA, Westborough, MA March 30, :00-11:00am

92 Attachment COMM 2-8(a) Page 2 of 46 K E M A, I N C. Introductions Ralph Masiello, Senior VP John McDonald, Executive Consultant Bob Uluski, Senior Principal Consultant Lee Willis, Senior Principal Consultant Mark Knight, Senior Principal Consultant John Bourguignon, Senior Principal Consultant Not Present Chuck Schreader, Senior Principal Consultant Scott Mix, Senior Principal Consultant 2

93 Attachment COMM 2-8(a) Page 3 of 46 K E M A, I N C. Agenda Background & Summary of Project NGRID State Today Issues & Opportunities ELSSI Vision and Plan Examples of Best Practices Summary of Potential NGRID Costs & Benefits Recommendations 3

94 Attachment COMM 2-8(a) Page 4 of 46 K E M A, I N C. Background Purpose of study was to explore ways that NGRID can better exploit IEDs and System Integration Expanded into a high level comparison of NGRID T&D Engineering, capital Asset Management, operations to best practices KEMA Interviewed 25 NGRID Departments and 110 Personnel State of IED penetration, communications, Enterprise usage of data; overview of maintenance / planning practices KEMA summarized gaps to Best Practice & estimated costs and Benefits of targeted gap closure KEMA recommends some short term efforts to pluck low hanging fruit What the Study Was NOT Analysis of State of T&D Infrastructure Analysis of NGRID Planning, Maintenance, & Operations Practices Business Process Analysis 4

95 Attachment COMM 2-8(a) Page 5 of 46 K E M A, I N C. Background State of the Industry setting the table 90%+ utilities implementing IEDs Extracting only 15% of the benefits 85% remaining to be tapped condition, performance, etc. key indicators that drive the decisions that business users make everyday! More and more utilities are initiating studies like this National Grid Position Atypical - IEDs being deployed very slowly new substations and ad hoc electro-mechanical (EM) replacements Typical - Enterprise IT Architecture to exploit IEDs is incomplete 5

96 Attachment COMM 2-8(a) Page 6 of 46 K E M A, I N C. Where is National Grid USA? Many good initiatives already going on, several of which can be leveraged. For example; Wavewin, PI, IED-based standard architecture, initiatives to ID outage trends. Low penetration of IEDs and uneven adoption of integration architecture No Focused Plan for IED Penetration Continuous condition monitoring not being done today; nor is easy retrieval of equipment historical loadings, etc. Asset Management not supported by statistical data SCADA penetration is below industry best practice. Lack of distance to fault information Many Maintenance / Operations procedures designed around EM relays and lack of continuous condition monitoring. Makes statistical analysis and project portfolio optimization difficult Low system observability 6

97 Attachment COMM 2-8(a) Page 7 of 46 K E M A, I N C. Outage Trending Widely Desired 7

98 Attachment COMM 2-8(a) Page 8 of 46 K E M A, I N C. Low SCADA Penetration 8 Combined T & D

99 Attachment COMM 2-8(a) Page 9 of 46 K E M A, I N C. Substation / IED Refit Rate CURRENT RATE, APPROX 5 NEW SUBSTATIONS PER YEAR FUTURE RATE, APPROX NEW SUBSTATIONS PER YEAR 9

100 Attachment COMM 2-8(a) Page 10 of 46 K E M A, I N C. Why the Big Change? 10 Just like compound interest, any rollout that lasts longer than the life of an IED will incur compound rollout costs. The Oval Fitting electro-mechanical devices for 10 more years, with a service life of 40 years. Need to replace in 50 years. Support is unavailable (to a greater extent) in 30 years. At 5 new stations per year: deployment will take 260 years will refit a total of 11,480 stations the first stations will get refitted 18 times over 50 "standard" designs theoretical but unrealistic not even making minimum payment

101 Attachment COMM 2-8(a) Page 11 of 46 K E M A, I N C. All the Costs, None of the Benefits Current IED Deployment Rates Will NOT: Improve reliability Produce cost reductions Increase operations efficiency Allow process improvements But Current Rates Will: Guarantee multiple technology families deployed across regions / districts Data silos will persist Prolong existence of EM relays Which in turn prolongs existence of RTU equipment difficult to obtain non-operational data 11

102 Attachment COMM 2-8(a) Page 12 of 46 K E M A, I N C. Design Cycle Design cycle is duration of a specific architecture (e.g. LPS03) 5-6 year design cycle for integrated substation feels about right. Shorter cycle means constant change Longer cycle does not keep pace with technology This is about technology, but not for technology s sake Need to utilize the available cycle time for IED rollout into sufficient substations. Optimize compound rollout costs 12

103 Attachment COMM 2-8(a) Page 13 of 46 K E M A, I N C. Data Requirements Matrix Example 13

104 Attachment COMM 2-8(a) Page 14 of 46 K E M A, I N C. ELSSI Vision COMMUNICATIONS USER LEVELS USER FUNCTIONS DATA MART 14

105 Attachment COMM 2-8(a) Page 15 of 46 K E M A, I N C. ELSSI Communications Vision ELSSI Communications Vision PC Engineering PC PC Engineering PC PC Historical Data Engineering PC Server Operational Data Path PC Engineering PC Server Data Server Firewall Corporate WAN Router Firewall Data Server Data Warehouse EMS Environment Firewall Operational Path Frame Relay Strategy Non-Operatioinal Path Remote Access Path Telemetric Telemetric Environment v via internet web browser Data Concentrator PC Local HMI Router Firewall 56Kbps Remote Access thru Corp. WANPDA (ex. VPN, RAS) Laptop IED IED IED IED IED IED IED 15 Feeder Environment Substation Environment

106 Attachment COMM 2-8(a) Page 16 of 46 K E M A, I N C. Communication Paths From Substation Two second data to SCADA system (operational data extracted using industry standard protocol such as DNP3) On demand data to utility information server or data warehouse (non-operational data extracted using IED vendor s proprietary ASCII commands) Remote access from remote site to isolate a particular IED (also called pass through or loop through )

107 Attachment COMM 2-8(a) Page 17 of 46 K E M A, I N C. Communication Paths From Substation (continued) Utility Enterprise Connection SCADA Data to MCC Historical Data to Data Remote Dial-In to IED Warehouse Substation Automation Applications IED Integration Via Data Concentrator/Substation Host Processor IED Implementation Power System Equipment (Transformers, Breakers)

108 Attachment COMM 2-8(a) Page 18 of 46 K E M A, I N C. Local vs. Enterprise Data Marts Local historian at substation level is a component of the Substation Automation System (e.g., PC with local substation HMI and historical data archiving) and is designed for Data Mart integration Ability to push data From substation to enterprise Data Mart based on time, demand or event triggered Enterprise Data Mart can pull data from local Data Mart in substation

109 Attachment COMM 2-8(a) Page 19 of 46 K E M A, I N C. Today Substation System Network Topology Fitch Road GPS Clock IRIG B Local I/O Quantum PLC F.O Mux Modbus DNP F.O Repeater RTU Modem Leased Telephone EMS Secondary Substation Computer Modbus Primary Substation Computer Satec Meters Modbus Terminal Server DNP ETHERNET Telephone Dialup SEL 2032 SEL 500 Series Terminal Server SEL 351 Terminal Server DNP SEL 421 FTP DNP FTP NXTPhase L-PRO WaveWin Comm Processor 19 ASCII

110 Attachment COMM 2-8(a) Page 20 of 46 K E M A, I N C. Future Vision Substation System Network Topology Local I/O PLC Modem Leased Telephone EMS GPS Clock IRIG B Data Concentrator (Includes WaveWin Comm Server) Frame relay or other Network Corporate Network Local HMI Satec Meters Modbus Terminal Server Router FRAD ETHERNET Terminal Server (Non-Operational Data) Terminal Server Terminal Server DNP FTP DNP FTP SEL 500 Series SEL 351 SEL 421 NXTPhase L-PRO 20 ASCII

111 Attachment COMM 2-8(a) Page 21 of 46 K E M A, I N C. Islands of Information Islands of information make it difficult to improve crossdepartment processes. National Grid currently performs analytical studies on various combinations of data. An integrated virtual data mart for operations (field) data would permit multi-dimensional analysis to uncover more information about root cause etc. 21

112 Attachment COMM 2-8(a) Page 22 of 46 K E M A, I N C. Data Mart Vision Integrate different business areas, multiple technologies and provide access to different levels of users. 22

113 Attachment COMM 2-8(a) Page 23 of 46 K E M A, I N C. Data Mart Build on existing investments in systems. Based on service definition of sharing data. Start with definition of terms, detailed user data requirements, and identification of key data services. Develop a service definition methodology and build a pilot. Benefit of being a one stop shop. 23

114 Attachment COMM 2-8(a) Page 24 of 46 K E M A, I N C. Best Practice Examples Comparable utilities are exploiting IEDs to achieve improved performance (reliability) at lower costs (Capital and O&M) Fault location and dispatching Condition Based Maintenance Asset Management Productivity improvements These utilities have a broad strategy for deploying and integrating IEDs and IT Systems to achieve business objectives 24

115 Attachment COMM 2-8(a) Page 25 of 46 K E M A, I N C. LADWP Data Integration Project CORPORATE INFORMATION SYSTEM ENERGY CONTROL CENTER LEGACY CONTROL SYSTEM METALLIC COMMUNICATION CUSTOMER STATION (28 ea) 25 GENERATING STATION TRANSMISSION RECEIVING STATION ( 21 ea) SUBTRANSMISSION DISTRIBUTING STATION (126 ea)

116 Attachment COMM 2-8(a) Page 26 of 46 K E M A, I N C. LADWP Problems 25-year old obsolete & exhausted SCADA/RTU Poor reactive power control Labor intensive processes Inability to rapidly detect & diagnose trouble Inability to manage integrated resources Information islands Inability to realize full benefits of modern devices Aging, low bandwidth, failure-prone telecom 26

117 Attachment COMM 2-8(a) Page 27 of 46 K E M A, I N C. Program Scope CORPORATE INFORMATION SYSTEM FIBER -OPTIC WAN COMMUNICATION ENERGY CONTROL CENTER CUSTOMER STATION (28 ea) AMR BACK-UP ECC RADIO TOWER ECSU Project WIRELESS DEVICES 2700 capacitors 250 switches 7000 fault recorders Replace the SCADA/AGC master Build a back-up control center Automate 179 substations Fiber-optic WAN connect all stations Install 10,000 wireless micro-rtus Automate reactive power control Enterprise Information System access to Energy Control System GENERATING STATION TRANSMISSION RECEIVING STATION ( 21 ea) SUBTRANSMISSION DISTRIBUTING STATION (126 ea) DISTRIBUTION FEEDERS (1600 ea) 27 Reengineer related human processes

118 Attachment COMM 2-8(a) Page 28 of 46 K E M A, I N C. Project Benefits Financial Reduce reactive power flow Improve O&M efficiencies Reengineer work processes 28 Reliability Maintain power system control integrity Reduce Customer Average Interruption Duration Index (CAIDI) by 10 minutes on monitored circuits Improve maintenance to key power system components Reduced reactive power = Reduced emissions Reduced emissions = Improved air quality Environmental

119 Attachment COMM 2-8(a) Page 29 of 46 K E M A, I N C. Project Benefits Information Provide critical information for engineering and system planning Develop an information history to improve maintenance management Provide an information bridge to the corporate enterprise 29

120 Attachment COMM 2-8(a) Page 30 of 46 K E M A, I N C. Asset Management & Condition Based Maintenance Usage and Environmental History Updated Manufacturer s Data and Industry Data Continuous Condition Observations EQUIPMENT PERFORMANCE AND FAILURE PROBABILISTIC MODEL Scheduled / Event Driven Condition Observations Maintenance History Updated Equipment Performance and Failure Probability Distribution Functions Alarming & Scheduling Systems To ASSET MANAGEMENT 30

121 Attachment COMM 2-8(a) Page 31 of 46 K E M A, I N C. Asset Management & Condition Based Maintenance Probabilistic Equipment Performance Models Resource Constraints Financial Objectives & Constraints Maintenance Alternatives / Strategies Replacement Alternatives SYSTEM PERFORMANCE MODEL PORTFOLIO OPTIMIZATION Financial / Technical Performance 31 Risk

122 Attachment COMM 2-8(a) Page 32 of 46 K E M A, I N C. Cost/Substation Estimates Installed Cost Breakdown Transmission GIS Transmission Open Air Distribution SA Hardware $ 92,500 $ 92,500 $ 92,500 Communication facilities $ 2,500 $ 2,500 $ 2,500 SA Software $ 42,000 $ 42,000 $ 37,000 SA Vendor Services $ 99,500 $ 99,500 $ 99,500 IED Installation and Commissioning $ 825,000 $ 855,000 $ 530,000 $ 1,061,500 $ 1,091,500 $ 761,500 O&M Costs for Substation Facilities Transmission GIS Transmission Open Air Distribution SA Hardware $ 2,775 $ 2,775 $ 2,775 Communication facilities $ 12,000 $ 12,000 $ 12,000 SA Software $ 4,200 $ 4,200 $ 4,200 SA Vendor Services $ - $ - $ - IED Installation and Commissioning $ 24,750 $ 25, $ 43,725 $ 44,625 $ 34, * Costs Have to be Adjusted for Eliminated Costs Associated with RTUs and EM Relays. Includes labor costs for installation and maintenance.

123 Attachment COMM 2-8(a) Page 33 of 46 K E M A, I N C. Projected Benefits and Cost Per Substation Area Initial $ Annual $ Benefit/Yr. $ B/C Implement IED Foundation $1,000 $44??? Targeted Dispatch + $4 $1 $5 3.5 IED-driven CBI & CBM+ $13 $1 $ Dynamic Equipment Ratings $25 $3 $ Intelligent Alarming $1 $1 $1.9 Total $1,043 $50 $

124 Attachment COMM 2-8(a) Page 34 of 46 K E M A, I N C. Your Results Could Differ National Grid will get different results: results could be better or worse than KEMA s estimates. T&D System design, loading, ownership policies Asset base age and condition Current and past maintenance and inspection practices Regulatory and environmental factors Workforce and labor cost issues Customer demographics and need Corporate strategy and its priorities 34 Results at these other utilities led KEMA to recommend that National Grid look into these opportunities in detail.

125 Attachment COMM 2-8(a) Page 35 of 46 K E M A, I N C. Targeted Dispatch and Allied Benefits Progress Energy Carolinas. Since M con. mtrs., 405 subs, 44K mi of D, 6K mi. of T Rated #1 distribution operations in 6 of past 10 years Industry leading OMS and WMS SAIDI is about 110 min. now Progress Florida (Projected/being implemented approved by regulator) 1.3 M con. mtrs., 435 subs, 34K mi of D, 6K mi. of T Lags state but not nation in SAIDI (79 min. in 2004) Aging system, no automation strategy until 2004 Item Initial Annual Annual Ben. B/C Use of Subst. IEDs to improve distribution restoration all $1,666 $285 $1, Credited with 7 minutes total SAIDI improvement Item Initial Annual Annual Ben. B/C Use of Subst. IEDs to improve distribution restoration all $3,760 $610 $2, Expected 3-5 minutes total SAIDI 35 Roughly half of the benefits come from after-the-fact analysis and response. - Tree trimming - Line maintenance

126 Attachment COMM 2-8(a) Page 36 of 46 K E M A, I N C. IED-driven CBI and CBM Central U.S. - projection 1.5 partial years bear out projected results in all ways 1.5 M con. mtrs., 380 subs, 31K mi of D, 5KM of T J.D. Power Cust. Satisfaction, # 1 once in last 5 years Pretty good CBM system, finetuned in 00 Item Initial Annual $4,000 $1,200 $1,840 $450 $9,500 $685 $3,800 $1,200 $1,785 Existing CBM System + ( 00) IED-driven CBI years 1-2 IED-driven CBI projected final CBI-driven CBM (delta figures) IED-driven CBI and CBM total $14,300 * Includes early hits but early costs ** Expected to improve due to asset base aging model Annual Ben. B/C $2, $ * $1, $2, $1, ** 36 Central U.S. - nearly 5 years of results.37m con. mtrs., 130 subs, 13K mi of D, 1.5K mi. of T Top performer among small utilities Maybe best planning in U.S. Co-planned with workforce attrition plan CBI: CBM: Item Initial Annual Annual Ben. B/C Targeted IED- CBI&M as done $2,780 $380 $ Targeted IED- CBI&M planned $2550 $440 $ Systmwd IED-CBI&M (est) $12,850 $2080 $3, Condition Based Inspection Condition Based Maintenance

127 Attachment COMM 2-8(a) Page 37 of 46 K E M A, I N C. Dynamic Equipment Ratings & Contingency Usage Item Initial Annual Annual Ben. B/C Australian Wind Farm Tie $425 $125 $1, Large US Utility Pool Tie $1,400 $450 $ Southern US best 3 lines $2,800 $750 $7, best 10 $29,000 $7,000 $15, European Cable Project $1,800 $217 $ South American Intertie $1,850 $200 $1, Results change considerably when evaluated probabilistically 37

128 Attachment COMM 2-8(a) Page 38 of 46 K E M A, I N C. Selective Intelligent Alarm Use These are the results from a smaller utility in the western US Item Initial Annual Annual Ben. B/C Targeted Alarms as done $2,780 $380 $1, Targeted Alarms as plan $2,550 $440 $1, Systemwide plan $12,850 $2,080 $3, Good targeting is the key 38

129 Attachment COMM 2-8(a) Page 39 of 46 K E M A, I N C. Lessons Learned from Other Utilities Substantial evidence that ELSSI provides a positive benefit to a T&D utility Good results come from integrated exploitation of multiple benefit areas Targeting to specific sites and equipment categories is a key to high B/C ratio Serendipitous uses add about 10 15% to benefits after initial implementation 39

130 Attachment COMM 2-8(a) Page 40 of 46 K E M A, I N C. Projected Benefits and Cost (per substation) Area Initial $ Annual $ Benefit/Yr. $ B/C Implement IED Foundation $1,000 $44??? Targeted Dispatch + $4 $1 $5 3.5 IED-driven CBI & CBM+ $13 $1 $ Dynamic Equipment Ratings $25 $3 $ Intelligent Alarming $1 $1 $1.9 Total $1,043 $50 $ Enterprise IT costs are not included above (approx $15K - $50K per station)

131 Attachment COMM 2-8(a) Page 41 of 46 K E M A, I N C. Snapshot: What We ve Done, What We Need to Do Next Phase 1 Phase 2 Phase 3 Vision Design Build/Implement Objectives Opportunities Benefits & Costs Plan to Proceed Develop ELSSI Architecture Develop CBM strategy Address fault location needs Address intelligent alarming Develop ELSSI-based CBM plan Integrate Wavewin strategy Tap secondary benefits Initiative Initiative Initiative Initiative Vision Feb/Mar 2005 April 2005 (FY2006) to?? FY2006-{5-6 YEARS} 41

132 Attachment COMM 2-8(a) Page 42 of 46 K E M A, I N C. R1: Develop ELSSI Technical Architecture Build on Existing Initiatives Substation Substation Integration Architecture Enterprise Establish Data Models, Interoperability, IT Architecture Formalize Process for Reviewing IED Technologies on a Determined Time Scale (e.g. 5 yrs) Wavewin SCADA OSISoft PI Rationalize IED, Communications, and IT Strategies in NE and NY 42

133 Attachment COMM 2-8(a) Page 43 of 46 K E M A, I N C. R2: Analyze T&D Infrastructure for Targeted CBM and AM Strategy Development Develop Rigorous Estimates of Potential Savings Specific to NGRID Build Base Asset Model Compatible with Maximo Deployment Developed to Support Asset Management Initiative Develop ELSSI Deployment Strategies to Support Integrate AM portfolio planning and data mart with IED rollout strategy 43

134 Attachment COMM 2-8(a) Page 44 of 46 K E M A, I N C. R3: Consider Targeting a District for Rollout of Distribution IEDs Achieve a critical mass for dispatching maintenance operations 44

135 Attachment COMM 2-8(a) Page 45 of 46 K E M A, I N C. R4: Accelerate Existing Initiatives Transform Wavewin into an Enterprise application Move forward with NY EMS replacement 45

136 Attachment COMM 2-8(a) Page 46 of 46 K E M A, I N C. Thank You Now the real work begins

137 Attachment COMM 2-8(b) Page 1 of 160 Enterprise Level Substation Systems Integration Phase II Report National Grid USA 25 Research Drive Westborough, Massachusetts August 26, 2005

138 Attachment COMM 2-8(b) Page 2 of 160

139 Attachment COMM 2-8(b) Page 3 of 160 Executive Summary KEMA conducted Phase I of the Enterprise Level Substation Systems Integration (ELSSI) study for National Grid in January - March of One of the leading recommendations of that study was that National Grid should act quickly to develop an Enterprise Systems Architecture for substation data, including the architecture within the substation, the communications infrastructure, and the enterprise virtual data mart that would make substation data accessible to applications and users throughout National Grid. National Grid requested a plan for Phase II of ELSSI to develop that Architecture and this report represents the results of that work, performed during May - July The architecture presented in this report is a straw-man architecture that is representative of what the final ELSSI architecture will look like, but it is not the implementation design. It has enough detail to identify the meaningful architectural components required and is mapped to a representative set of current software vendors that supply these type of functions in order to generate a better cost model. This cost model was then used as part of a benefit cost analysis exercise. Increasing the availability of IED data has many potential benefits and the wider integration of operations data has many potential benefits for National Grid. The nature of utility businesses will no doubt change over time and the nature and use of data will also change over time. Therefore benefits will also have temporal variations based on economic and regulatory conditions, making it necessary to develop a business case for each new benefit area over time. In developing these business cases there needs to be a method for enabling the benefits to be realized. In order to enable this to be achieved across multiple functional areas an architectural framework is imperative, which specifies data elements, methods, and guidelines for how to implement ELSSI. This report describes that architectural framework. The enterprise architecture was generally well received in the final presentation of this work and includes two suggested implementation work packages for National Grid which can be phased to support a pilot implementation at a lower initial cost. The substation architecture was developed with the following objectives in mind: Enable National Grid to fully exploit the wealth of information contained in existing and future substation IEDs Enable National Grid personnel to access operational and non-operational data from substation IEDs without having to travel to the substation Promote data sharing and minimize the existence of data silos and duplication of data National Grid i Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

140 Attachment COMM 2-8(b) Page 4 of 160 Executive Summary Enable National Grid to accomplish high payback application functions by providing access to IED data required to support these functions The design characteristics of the ELSSI substation facilities may differ somewhat for transmission substations and distribution substations due to differences in the physical, functional, and performance requirements at these two classes of substations and various factors affecting these characteristics were considered. The architecture takes account of multiple functional data paths for operational data, nonoperational data, and pass-through. It also looks at various possible configurations for key components, supports the current LPS initiatives underway by TISC and makes several recommendations. The architecture was well received during the final presentation of this work. The architectural framework is easier to embrace when the potential benefits are more visible, so KEMA has also provided National Grid with one outline business case focused on asset management and CBI/CBM for critical (bulk) transmission substations. This is described in an accompanying report and also includes a tool for evaluating various benefit cost analyses that can be developed by National Grid using the KEMA provided tool as a starting point. The purpose of this business case was to develop a case using a repeatable model that can continue to be used over time to develop future business cases as and when National Grid considers increased deployment of IEDs. This work was performed in National Grid s Syracuse, Northborough and Westborough offices. Due to the high IT component and inter-departmental reach of the substation data, National Grid s IT department played a significant role in review of the integration architecture through several meetings and workshops. Considerations were made for existing National Grid processes and governance of projects as well as external standards such as NERC CIP. KEMA performed the Benefit Cost Analysis (BCA) for ELSSI using the Sandy Pond substation as the specific example to be analyzed. Sandy Pond was chosen by National Grid for the analysis for its convenient location and because of the criticality of DC link at that station. However, it came out during the BCA activity that replacement energy costs, cost of congestion (a major factor in other 345kV station BCA methodology) and transmission reliability are not quantities with direct economic impacts to the National Grid T&D business. Thus the BCA had to focus on avoided O&M costs, repair costs, and deferred capital replacement costs. Estimates of the societal economics are computed and shown separately. The greatest amount of debate during development of this tool were the parameters used and values that National Grid felt comfortable with. While there are many tangible benefits that KEMA National Grid ii Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

141 Attachment COMM 2-8(b) Page 5 of 160 Executive Summary believes will accrue from ELSSI deployment, the BCA focused on a few classes of equipment and a few relatively straightforward economic benefits. Both the methodology and the order of magnitude of assumed parameters are consistent with other published work performing BCA for substation automation and transformer monitoring. After working with NGRID and meeting twice and presenting three iterations of the BCA tool, KEMA have provided a flexible, parameter driven tool that reflects numbers agreed to between KEMA and National Grid. The analysis failed to find great benefits (relative to cost) in continuous monitoring for widespread IED deployment, but found enough light at the end of the tunnel to continue the search for additional benefits. Many soft benefits in the form of avoided costs were quantified that show clear value. A final option considered was a monitoring-only solution for key bulk power transformers and breakers, which lends itself to a strong benefit cost ratio yet avoids the costs of installing large numbers of additional equipment and structures, such as costly dual control houses, as required by NPCC and included in initial modeling that involved extensive deployment of IEDs. The modeling of other 345 and 230 kv stations is generic and does not account for differences in criticality, impact on congestion, or age and condition of equipment. The BCA is therefore indicative but hardly precise. There are six major recommendations and fifteen other recommendations described in this report. The major recommendations are summarized below. The other recommendations can be found in section 6, and are cross-referenced to the body of the report where the recommendations are made. The major recommendations are: Determine areas where improved CBI/CBM could have greatest benefits and prioritize substations, equipment, and specific monitoring technologies where these benefits would have most impact for early implementation of IEDs. Extend the cost benefit model used in Phase II and perform different scenario analysis to evaluate sensitivity of the model with respect to individual costs, benefits and current capabilities. Evaluate a target list of specific stations so that the variability in equipment age, station configuration, and station impact on system costs are understood. The EMS and Gas SCADA design should be tasked with ensuring support of ELSSI objectives and Enterprise Architecture. National Grid iii Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

142 Attachment COMM 2-8(b) Page 6 of 160 Executive Summary The broad class of automation equipment should itself be considered as a set of assets subject to asset management and portfolio optimization in an integrated and holistic way. Currently transmission substations have had a high focus but distribution substations and distribution feeders have many locations where IED data would be beneficial. KEMA recommends that National Grid conduct an ELSSI visioning and planning exercise aimed at distribution stations and feeders to see what reliability and economic benefits are possible. Also, ELSSI for distribution would be able to take advantage of existing NGRID initiatives around digital substations and the BCA might be more attractive. KEMA recommends an ELSSI pilot that builds on the work of TISC as well as the results of ELSSI Phases I and II. The scope of this pilot and the justification and benefits are explained in greater detail in Section 6.3. There is no issue of whether IEDs will get deployed at National Grid. They are currently being deployed in new substations and are also being used in one for one replacement for electromechanical devices. The biggest unanswered question is the rate at which IEDs ought to be deployed. The benefit to cost ratio discussed in the accompanying BCA report is based on what is currently understood about use of IED data within the industry. A slow rate of deployment could result in the use of many different IED types but with relatively few of each type being used. As discussed in the Phase I report any rollout that lasts longer than the life of an IED will incur compound rollout costs, not unlike paying compound interest on a credit card bill. An optimum range would seem to be about new stations per year (from Phase I), which will take years for the replacements. This is a big increase over the approximate rate of 5 substations per year and incurs significant additional cost. Yet if you are going to do it anyway it surely makes sense to focus on ways to minimize the costs and increase the benefits whatever the BCA ratio. Increased use of IEDs is the direction that the industry is headed. Since the benefits will change over time and increase with better understanding of the data KEMA strongly recommends that National Grid develop a pilot project to implement some of the Enterprise Architecture recommendations and create an environment where limited IED data integration is performed based on the currently deployed IEDs and that this is treated as an exercise in exploring improved algorithms and mechanisms for realizing greater National Grid iv Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

143 Attachment COMM 2-8(b) Page 7 of 160 Executive Summary benefits. This knowledge can then be used in conjunction with the BCA tool to develop new benefit cost models for IED deployment. National Grid v Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

144 Attachment COMM 2-8(b) Page 8 of 160 Table of Contents 1. Introduction Purpose of Document Organization of Report Objectives, Scope and Benefits Business Objectives and Scope Phase I Objectives and Scope Phase II Objectives and Scope Scope of Systems Benefits and Condition Based Maintenance Inspection and Maintenance Reliability Improvements through Avoided Failures Reliability Improvements via use of Fault Location in Trouble Dispatching Dynamic Ratings Asset Management Decisions Asset Life Cycle Management and Life Extension Productivity Improvements via Integration of IT Systems and Databases Improvement of Engineering Functions Architectural Goals, Constraints and Values Architecture Drivers and Constraints Base Architecture on Existing Technologies Reuse National Grid s Enterprise Integration Architecture Map Functional Requirements to Existing National Grid Owned Vendor Products Ensure Scalability for Large Number of Substations Provide Component Base Architecture Base Cost Estimate on Solid Straw-man Architecture Cyber Security ELSSI Substation & System Architecture Analysis and Recommendations Overview System Context View System Component View Software Component View System Architecture View System Integration View Deployment View Architectural Significant Flows Recommended Approach System Context View System Component View Software Component View System Architecture View System Integration View National Grid i Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

145 Attachment COMM 2-8(b) Page 9 of 160 Table of Contents Deployment View Data View Use Case Scenarios and Sequence Diagrams Data Flows and Data Management Data Flow Definitions IED Data Volume and Timing Meta Data Fact Model Data Requirements Matrix Update Substation Architecture Phase I SA Vision Substation Classes Specific Design Issues Interface to EMS SCADA Cyber Security Substation Operator Interface Hardwired Inputs Hardwired Outputs Handling of Sequence Control Logic Communication Facilities and Protocols Implementation Considerations Opportunities and Benefits To Roll-out or Not to Roll-out, That is the Question Discussions from Phase I Implementation Considerations On-going and Planned Projects Technology and Platform Considerations Legacy Systems Capabilities and Limitations Migration Strategies Communications Requirements Security Requirements Recommendations and Issues Major Recommendations Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # National Grid ii Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

146 Attachment COMM 2-8(b) Page 10 of 160 Table of Contents 6.2 Summary of Other Recommendations Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # Recommendation # ELSSI Pilot Details Other Issues Appendix A...A-1 Appendix B... B-1 Appendix C... C-1 List of Exhibits: Exhibit 2-1: Phase II Scope Exhibit 2-2: High Level ELSSI Architecture Exhibit 2-3: Benefits Exhibit 2-4: Stress Effects and Transformer Failure Exhibit 3-1: Cost and Duration for BR1 and BR Exhibit 3-2: Scope of BR1 and BR Exhibit 4-1: ELSSI Project Scope Exhibit 4-2: System Component Overview Exhibit 4-3: Software Stack Diagram Exhibit 4-4: Strawman System Architecture Exhibit 4-5: System Integration View Exhibit 4-6: Deployment Diagram- Data Ingest Exhibit 4-7: Deployment Diagram- Data Usage Exhibit 4-8: System Repositories Exhibit 4-9: Sequence Diagram Legend Exhibit 4-10: System Operations Sequence Diagram Exhibit 4-11: Event Investigation Sequence Diagram National Grid iii Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

147 Attachment COMM 2-8(b) Page 11 of 160 Table of Contents Exhibit 4-12: Distribution Planning Sequence Diagram Exhibit 4-13: Reliability Analysis Sequence Diagram Exhibit 4-14: CBM/CBI Sequence Diagram Exhibit 4-15: Maintenance Sequence Diagram Exhibit 4-16: Dynamic Ratings Sequence Diagram Exhibit 4-17: MDS Use Case Diagram Exhibit 4-18: ELSSI System Context Diagram Exhibit 4-19: High Level ELSSI Business Data Flows Exhibit 4-20: Data Ingest DFD Exhibit 4-21: Data Usage DFD Exhibit 4-22:IED Data Element Sizing Assumptions Exhibit 4-23: Generic IED Characteristics Exhibit 4-24: Generic Transmission and Substation IED Usage Patterns Exhibit 4-25: Range of System Level Loadings Exhibit 4-26: Storage Summary Exhibit 4-27: Initial Fact Model Exhibit 4-28: Pass Through Approach Exhibit 4-29: Enterprise Level Data Warehouse Exhibit 4-30: ELSSI SA Configuration for Approach Exhibit 4-31: ELSSI SA Configuration for Approach Exhibit 4-32: Possible Interim Approach Exhibit 4-33: SA Architecture for PC-Based Local HMI Exhibit 4-34: Trip/Close Pushbuttons on SEL 351S Relay Exhibit 5-1: Data Retrieval Network Diagram with Network Bridge Exhibit 5-2: Typical IED Characteristics Exhibit 5-3: Background of Sizing of Data Elements Exhibit 5-4: Typical Transmission Substation Data Offered to WAN Exhibit 5-5: Typical Distribution Substation Data Offered to WAN Exhibit 5-6: Data Bandwidth Requirements for IED Penetration Ranges-Transmission Substations Exhibit 5-7: Data Bandwidth Requirements for IED Penetration Ranges-Distribution Substations Exhibit 5-8: Comparison of Network Based vs. Point-to-Point Circuits Exhibit 5-9: Typical Access Configuration from the Substation to the Corporate Network Exhibit 5-10: Interim Substation Architecture Exhibit 5-11: Basis of 3% Yearly IED Penetration Exhibit 5-12: Typical TDM over SONET Configuration Exhibit 5-13: Effects of Penetration of IED Substations on the Corporate WAN Exhibit 6-1: ELSSI Pilot Overview List of Tables: Table 2-1: Transformer Failure Mechanisms & Detection Methods National Grid iv Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

148 Attachment COMM 2-8(b) Page 12 of Introduction 1.1 Purpose of Document Continuing on from the results and recommendations of Phase I (ELSSI Vision), National Grid engaged KEMA to proceed with further ELSSI activities to determine how ELSSI, through deployment of IEDs, would impact the architecture of substations and also to develop an architecture and cost for the data integration. This work extended to include analysis of where ELSSI could potentially deliver benefits to National Grid s business through a benefit cost analysis combined with analysis of potential roll out strategies. The benefit cost analysis shows favorable avoided O&M, repair and deferred capital replacement cost benefits; therefore KEMA believes ELSSI should proceed further. If ELSSI did not show a favorable benefit:cost ratio based on identifiable hard benefits alone the associated soft benefits and business process improvements could still make the pursuit of ELSSI as an enabler of change a feasible choice. This document explains the ELSSI architectures for substations and the (enterprise) data and explains the benefit cost analysis modeled after the Sandy Pond substation. Along with these major artifacts, KEMA has developed a supporting set of artifacts that support IED-based data collection and automation and are described in this report. 1.2 Organization of Report This report titled Enterprise Level Substation Systems Integration, Phase II: Systems Architecture represents the results of the second phase of a multi-phase initiative to develop the substation and data mart architectures which will support the operational and non-operational data use. This work was performed from May through July 2005 by KEMA in National Grid s Syracuse, Northborough and Westborough offices. Phase II represents the development of several products that can be used by National Grid to revise the design of substations, initiate the detailed design of a data mart, identify what the required supporting enterprise infrastructure will cost and review the benefits of ELSSI based upon its impact on Sandy Pond and other substations. The main body of this report is structured as follows: Section Name Description Executive Summary A brief executive summary of the findings and recommendations is provided in this section. Section 1 Introduction The introduction provides a description of the organization of the final report, the methods deployed and the references used. National Grid 1-1 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

149 Attachment COMM 2-8(b) Page 13 of 160 Section Name Description Section 2 Objectives, Scope and Benefits This section describes the objectives for this phase of ELSSI, the scope of work and brief overviews of ELSSI benefits and the architecture. Section 3 Section 4 Section 5 Section 6 Architectural Goals, Constraints and Values ELSSI Substation & System Architecture Analysis and Recommendations Implementation Considerations Recommendations and Issues Benefit Cost Analysis This section describes the factors that provide form to the ELSSI architecture in terms of industry, National Gridspecific and practical guidance. This section describes the ELSSI architecture for National Grid in terms of substations and the integration of IED data. This section contains deployment considerations (including technology considerations) that covers systems, communications, security, etc., as well as projects already being considered by National Grid and benefit opportunities. This section summarizes the various sections of this report and highlights some issues that cropped up during the course of this work. The Benefit Cost Analysis (BCA) provides a summary analysis of the benefits of deploying ELSSI including costs associated with refurbishing, the data mart costs and upgrades to communications typical of those required for 345 kv substations at National Grid. The BCA is provided under separate cover. The main report is supported by a substantial amount of material that was captured during this work and which will serve as the basis for use in subsequent phases. All of the appendices are contained on CD- ROM. The structure of the appendices and the disposition of the contents of each section are as follows: Section Name Description Appendix A Data Requirements Matrix This section contains the Data Requirements Matrix (DRM) as gathered from the interviews in Phase I, but updated to reflect National Grid s feedback and recommended edits during Phase II. The DRM also includes the prioritization based upon specific parameters (e.g. need, etc.). This version represents the second of what normally takes three passes to complete the data requirements. The DRM is provided under separate cover. National Grid 1-2 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

150 Attachment COMM 2-8(b) Page 14 of 160 Section Name Description Appendix B IED Templates This section contains the IED templates to ensure that the desired operational and non-operational data are identified for future use when maximized and simplify extraction capabilities. The IED templates are provided under separate cover. Appendix C Relevant Industry Standards Descriptions of relevant industry standards for communications and NERC CIP, for example, are provided as background information for this report. National Grid 1-3 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

151 Attachment COMM 2-8(b) Page 15 of Objectives, Scope and Benefits 2.1 Business Objectives and Scope Opportunities to mine information are becoming increasingly available in ways that will enable applications to be developed to use IED data and to improve operations within National Grid. Abilities to more quickly and more accurately diagnose faults for protection engineers and operations engineers, as well as providing smarter analysis tools for asset managers are becoming more and more viable hence National Grid s interest in the Enterprise Level Substation System Integration project, which KEMA has been pleased to participate in. The overall objective of National Grid s integrated substation design efforts to date has been to reduce the total life-cycle cost of the secondary systems within substations while increasing their functionality. However, as the number of substations deployed with this integrated technology has increased, and substation information has become more readily available, attention needs to be given to the way that the available information is managed. The objective of the work to date (Phase I, Phase II) has been to develop and provide National Grid with an overall vision, strategy and the architecture plans for implementing an enterprise level substation integration system. The scope of this work has included consideration of data acquisition from Substation Intelligent Electronic Devices (IEDs) as well as from other IEDs on the distribution system, though the primary focus has been substation IEDs due to their physical proximity and relative ease of integrating with communications infrastructure. All levels of the electric and gas delivery system were considered: transmission, sub-transmission, and distribution. It has included the communication requirements for National Grid to design a communications architecture to connect to those IEDs, the collection of data, the storing of the data, the integration requirements needed to support the systems, the data and the IEDs as well as potential application interfaces and benefit cost analyses. 2.2 Phase I Objectives and Scope A key objective of Phase I was to gather information from users and potential users of information from IEDs and how they might use data that they do not currently have access to, as well as to determine which data were presently used by those same people. This was achieved by a single round of user interviews that were preceded with a questionnaire relating to IED data for National Grid staff to complete prior to the interview process which also included aspects such as how people gathered that data (e.g., manual, automated). National Grid 2-1 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

152 Attachment COMM 2-8(b) Page 16 of 160 KEMA developed a conceptual plan on how to move National Grid s integration efforts to the enterprise level through visions for data, a virtual data mart and for substation design (number of logical data paths etc.). In order to deploy the type of integrated solution proposed, a gap analysis was required to describe the data and to look for synergies with existing data. This was explored in more detail during Phase II. The recommended integration and automation functions were then prioritized after a benefit cost analysis based on payback, strategic value, and other business concerns. In November 2004, before this project started, the desired end result that came out of discussions on this topic was that National Grid could come away with a view on whether further work and analysis was required or whether there were more synergies for structuring operating data or whether it was acceptable to leave things as they were. Phase I clearly found that there were more synergies for structuring not only operational data but also non-operational data from IEDs and integration of these data with other internal National Grid data sources. 2.3 Phase II Objectives and Scope The objective of the Phase II project was to develop a reference system architecture for enterprise-level integration of substation data at National Grid. This included developing a vision for future substation architecture (see Section 4.5), including the roles of components such as the Human Machine Interface (HMI), Programmable Logic Controllers (PLC), Remote Terminal Units (RTU), Intelligent Electronic Devices (IED) and Substation Data Concentrators (SDC). Also, the project further assessed business priorities and benefits for data integration, reviewed status and capabilities of existing target systems for information integration, and evaluated potential implementation options to achieve the anticipated benefits. The IEDs currently deployed (and planned for future deployment at National Grid) were also reviewed along with the lists of data points and files that these devices could capture. Due to the high IT component and inter-departmental reach of the substation data, National Grid s IT department played a significant role in review of the integration architecture through several meetings and workshops. Considerations were made for existing National Grid processes and governance of projects as well as external standards such as NERC CIP (see Appendix C). Functional requirements for telecommunications requirements were also developed and included review of the TISC document Functional Requirements For Integrated Access Devices Supporting Communications With Station Equipment - Draft. Section has more information on communications. Also a benefit cost analysis was developed based on the Sandy Pond 345 kv substation in Ayer, MA. This analysis looked at the effects of deploying IEDs at Sandy Pond and also used Sandy Pond as a representative 345 kv/115 kv substation to analyze the cost and benefits for deploying IEDs at more substations. This analysis can be found in Section 1. National Grid 2-2 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

153 Attachment COMM 2-8(b) Page 17 of 160 Exhibit 2-1outlines the high level scope of the Phase II work: Exhibit 2-1: Phase II Scope 2.4 Scope of Systems The following diagram shows the high level overview of the ELSSI architecture. There are basically two types of inputs that the architecture must support: field devices such as IEDs, SERs, DFRs, Meters and Telemetric devices and existing operations and maintenance repositories. The field device information takes one of two paths into the ELSSI set of data repositories. The real-time operational data goes through the existing operations network to EMS/SCADA and then sent to enterprise EMS/SCADA (PI 1 ) historian for easy corporate access via the ELSSI components. The non-operations data (both files and data points) is collected by the data acquisition component, which routes the point data to the ELSSI DB and the file data to the content manager. The existing historical and related operational and maintenance data repositories will have two paths into the ELSSI architecture as well. ELSSI includes a federated database component, which provides the virtual data warehouse. This means the end users and applications will have access to all of the data within the ELSSI repositories as well as of the related operational data without moving any of the related operational data. The second path for existing historical and related operational and maintenance systems is to just load the information into the ELSSI DB using the existing set of ETL (extract, transform, load) 1 For the purpose of the enterprise architecture design, KEMA has assumed that there will, in the future, be a single repository type for all SCADA data. This report (see Section 5.2) assumes that this repository will be OSIsoft s PI. National Grid 2-3 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

154 Attachment COMM 2-8(b) Page 18 of 160 processes and tools developed and owned by National Grid and used to build the existing corporate data warehouse and business data marts. The standard substation automation architecture and the enterprise system architecture are described in detail in Section 4. These architectures reuse the existing National Grid architecture designs and components as much as possible; the architectures maximize the use of commercially available products and the architectures have been designed based on KEMA s knowledge of the domain space s products, functions and architecturally limitations to minimize the amount of investment in new software development. The approach taken is to assume configuration of these products will be required and the majority of the estimated software development will be using National Grid s infrastructure tools to integrate ELSSI with both existing National Grid business systems and the proposed new analysis systems that would be enabled with the ELSSI information. The diagram uses color codes to indicate categories of components: Orange Data inputs to be handled by ELSSI Blue Data collection processes Green Data storage repositories Exhibit 2-2: High Level ELSSI Architecture National Grid 2-4 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

155 Attachment COMM 2-8(b) Page 19 of Benefits and Condition Based Maintenance The benefits of implementing ELSSI at National Grid will enable the realization of benefits derived by utilizing and integrating operational and engineering information to achieve savings in maintenance, operations, and investment planning. Implementation of ELSSI provides potential benefits to many of National Grid s departments. Exhibit 2-3: Benefits T&D Performance Enhancement Environmental Reliability SAIDI / SAIFI Delivery Capacity Values Asset Mgmt ROI Risk/Reward Mgmt Planning Engineering T&D Operations System Ops Control Center Distribution Mgmt Enterprise- Level Information Integration Maintenance Customer Services Data Data Substation Substation Substation Generation Customers A list of the potential benefits from implementing ELSSI is as follows: Cost savings via condition based inspection and maintenance based on individual assets vs. a priority based inspection and maintenance system based on time and critical number (current priority based system does not include cost of maintenance) Reliability/cost improvements as a result of more effective maintenance for the same budget Reliability improvements via avoided failures 2 through continuous monitoring Reliability improvements via use of fault location in trouble dispatching 2 There are no existing track records to quote as representative examples but KEMA believes that the effect of continuous monitoring must have a measurable reduction in failure rates from the results of National Grid analysis of the data collected from this monitoring and from 25 years experience using critical numbers to drive maintenance. National Grid 2-5 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

156 Attachment COMM 2-8(b) Page 20 of 160 Increased capacity, deferred capital expenditures, and operating efficiencies/cost savings via use of dynamic ratings Improved asset management decisions based on hard statistical data and demand management profiles Improved asset life cycle management and life extension Productivity improvements via integration of IT systems and data bases; error reductions via elimination of manual and redundant steps Improvement of engineering functions Cost savings via conscious technology management and standardization Compliance with emerging standards (reliability, cyber security) A description of the major benefits follows Inspection and Maintenance Some utilities take advantage of experience with certain equipment types to lengthen or shorten fixed time intervals for equipment maintenance based on performance, primarily reliability or maintainability information. National Grid is an example of a utility that has long since embraced condition based maintenance however KEMA believes that with the use of timely information on operational and equipment-specific condition through an ELSSI system, the maintenance time intervals can be optimized to their fullest extent. For example, a circuit breaker that has not operated often might not need the same maintenance as one that operates frequently. However, depending on the frequency of operation a breaker that has not operated often might actually require additional inspections. Also, the magnitude of the current that the breaker interrupts might influence the decision to perform maintenance. An example of current National Grid practice is to inspect breakers that have not operated for a long time to determine if the breaker is capable of correct operation. For a breaker that has operated recently (and correctly) and is in good condition this is not an issue. This is a good example of how the discipline of condition based maintenance is very analytical and is improved by having good data on which to base decisions. Having near real-time information on these parameters allows maintenance to be performed when needed. Combining these parameters with the actual performance of the breaker to recent events, such as opening and closing times, can help drive a maintenance program to initiate specific tasks on critical items that could be prone to failure. National Grid 2-6 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

157 Attachment COMM 2-8(b) Page 21 of 160 The goal is to be sure to allocate resources to the most important tasks based on the best information available. In practice, such a program usually accomplishes this goal while reducing overall maintenance costs (as compared to the previous practices) making the program more effective as well as less expensive Reliability Improvements through Avoided Failures Equipment condition monitoring (ECM) devices installed as part of ELSSI implementation could provide the ability to detect many failure modes prior to the occurrence of an in-service failure. This benefit can be extremely significant with older equipment at critical locations on the system, where failure can be costly. Many on-line monitoring devices are available today to increase the ability to detect these failures. Priorities typically are given to power transformers and circuit breakers since they are usually the focus of the most expensive and disruptive failures to the system when they occur. However, other units in the substation can be monitored also, including CCVT s, voltage transformers, free-standing CT s, surge arresters, batteries, and other devices. Parameters that can be continuously monitored by commercially available devices for transformers include: Dissolved-gas-in-oil (single gas or multiple gas) Moisture-in-oil Temperature (ambient, oil, winding, etc.) Current (steady-state, short circuit, transient, harmonic) Voltage (steady-state, transient, harmonic) Cooling equipment status Bushing integrity (sum-current or voltage sensing) Partial discharge (acoustic or electrical) Parameters that can be continuously monitored for circuit breakers include: Operating time (opening or closing) Compressor operations National Grid 2-7 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

158 Attachment COMM 2-8(b) Page 22 of 160 Hydraulic or pneumatic system pressure levels SF 6 pressure levels Oil condition Individual or accumulated current levels during operation (including I 2 t values) Voltages (steady-state or transient) IEDs can be used to capture the above data, compute trends, and be used in software to provide useful analysis to warn the user of undesirable events and conditions. It can also be used to automate maintenance schedules to make the program more effective (see above). Much of the interest in the reliability benefit involves the ability to predict transformer failures. Table (at the end of this section) illustrates the types of failure mechanisms that can occur in transformers, and how they can be detected. Monitoring devices are currently available that, with applicable software, can detect many of the failure modes using sensors available for measuring dissolved gases, moisture, vibration, partial discharge, current, temperature, and voltage. Algorithms for assessing conditions related to thermal, dielectric, and arcing fault risks are well developed. It is generally assumed that current techniques will only detect about 50 to 60 % of the total failures, but activity is underway in several areas to increase this percentage. One failure mode that is still an important subject of analysis for detection algorithms is external fault (or through-fault) initiated mode. It is known that influencing factors for this failure mode include shortcircuit magnitude and frequency and insulation degradation. Exhibit 2-4 illustrates the degradation of a transformer mechanical strength over time. Short circuits cause significant drops in the withstand curve, and insulation degradation have gradual effects. Stresses encountered (fault levels and loads) generally increase as systems grow and mature. When the two curves meet, the through-fault failure occurs via a turn-to-turn winding failure. Algorithms for quantifying these factors are still being developed, but the ability to calculate stresses and provide a general condition of the strength of the transformer is available. 3 Derived from EPRI Report # , Power Transformer Maintenance and Application Guide, September 2002 National Grid 2-8 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

159 Attachment COMM 2-8(b) Page 23 of 160 Exhibit 2-4: Stress Effects and Transformer Failure Damage from faults Stress Withstand Capacity Stress Encountered Stress Maintenance Life-ending failure Through faults Time Reliability Improvements via use of Fault Location in Trouble Dispatching When faults occur on the T&D system, crews are usually dispatched to the area of the outage based on available protective device information provided through SCADA systems. Having additional power system disturbance data, including relay targets, fault location and magnitude, specific alarm information, etc., more effective dispatching can be performed by sending the precise personnel and other resources to the exact location of the trouble. This reduces the outage duration, labor cost (including overtime), and consequentially the lost revenue and other outage costs. An example of this at National Grid is the recent integration of data from Wavewin and TCM to locate short line sections close to a fault so that crews can be rapidly dispatched to locations close to the fault and avoid the cost (in some circumstances) of dispatching a helicopter to fly the line. This also provides a look-up inspection more likely to spot insulator problems that the look-down inspection provided from an aerial inspection Dynamic Ratings The dynamic rating of a transformer is indicative of its real-time capability to handle continuous or emergency loads based on existing and past ambient temperatures, internal oil and winding temperatures, load cycle, specific transformer design tolerances, and other parameters. Knowledge of this capability can be extremely valuable especially for situations involving short-term emergency conditions, where loads might be curtailed with significant cost or reliability implications. ELSSI could include software that National Grid 2-9 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

160 Attachment COMM 2-8(b) Page 24 of 160 would calculate an allowable rating and duration that the system operator could use to make more informed decisions about the operation of equipment in such scenarios. Such information is also valuable in forecasting the need to implement a capital project to increase the capacity of the substation to handle increased loads Asset Management Decisions Asset management decisions, such as equipment upgrades and replacements, could be made more accurately with the addition of information from an ELSSI architecture as recommended in this report. Historical operation and maintenance information including condition monitoring data, combined with statistical information on loading and other system parameters, could be used to calculate the risks associated with various asset management strategies. Priorities for spending could also be made more accurately with such information Asset Life Cycle Management and Life Extension Knowledge of the equipment condition, past performance, and historical loading and operations could be used to determine the remaining life of the equipment, future maintenance requirements, and ultimately the economic decision making criteria for retirement and life extension alternatives. The ability to perform these decisions more accurately is becoming increasingly important as the equipment population increases in age Productivity Improvements via Integration of IT Systems and Databases Many utilities have multiple IT systems and databases that are not well integrated. Different protocols are often used for the various devices that are utilized by the different departments in a typical utility. An advantage of the recommended ELSSI architecture is the integration of these systems so that the combined data sources can be used for more purposes and realize benefits otherwise unachievable. This should result in productivity improvements in various data collection and analysis functions Improvement of Engineering Functions Actual field measurements obtained from substation IEDs benefit the design and engineering functions related to: system protection, power factor monitoring and control, phase balancing, circuit reconfiguration and load balancing, load forecasting, and outage trending. National Grid 2-10 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

161 Attachment COMM 2-8(b) Page 25 of 160 Table 2-1: Transformer Failure Mechanisms & Detection Methods Category Factors Aging Mechanism Degradation Failure Mode Detection Method Acidity of Oil Thermal deterioration Insulation mechanical & Dielectric breakdown Oil quality Temperature Voltage Current Mechanical & Electrical Cycling Non-seismic Vibration Ohmic heating winding & core losses Contaminants Moisture & contaminants Contaminants; moisture Gas pockets or contaminants Low level contaminants External fault or surge Deteriorated lubricants Voltage; contaminants Loose components Thermal deterioration; cracking Thermal deterioration of lubricants Thermal decomposition of oil Bushing dielectric withstand Oil sludging or dilution Ionization and corona breakdown Electromagnetic forces on conductors/core High friction between moving parts Pitting or erosion of auxiliary device contacts Cyclic wear/fatigue electrical Dielectric strength & electrical connection insulation Binding & high friction Dielectric strength; sludge; gas formation Surface or internal insulating properties Oil dielectric strength Insulating material dielectric Movement or dislocation of windings and/or core Wear Electrical component wear Wear of components; loss of tolerance Dielectric breakdown Seized motor bearings Localized dielectric breakdown External flashover Localized dielectric breakdown Partial dielectric Turn-to-turn short circuit Binding of tapchanger, pump, fan motor, or valves Misoperation of LTC; loss of cooling components Binding; component deformation Oil quality & dissolved gas analysis Vibration analysis Oil quality, moisture monitors Visual observation; current or voltage monitoring Oil quality Dissolved gas analysis, PD sensors Acoustics, vibration, or relay activation Vibration PD sensors, DGA Vibration or acoustics National Grid 2-11 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

162 Attachment COMM 2-8(b) Page 26 of Architectural Goals, Constraints and Values 3.1 Architecture Drivers and Constraints KEMA approach s to developing an enterprise integration architecture is much more than just using the latest enterprise application integration technologies such as following a service orientated architecture (SOA) strategy or using the latest IT technologies such as Web Services. KEMA strongly believes that in order to come up with a solid enterprise architecture, one must lead or start with the business strategic objectives, then develop business and IT technical governance policies that guide how National Grid will roll out and adopt new business systems, and establish architectural principles that new systems and integrations should adhere to. This holistic approach gives National Grid the visibility they need into the set of business, technical and organizational changes required to get to their desired future state. Meetings were held early on in Phase II with National Grid to review internal governance procedures for project development/approval and data warehousing strategy. The straw-man architecture detailed in Section 4 was developed taking these into account in the following approach. Base the architecture on existing technologies Reuse National Grid s enterprise integration architecture Map functional requirements to existing National Grid owned vendor products Ensure scalability for large numbers of substations Provide a component based architecture that supports incremental business releases that stand on their own and provide value to National Grid Base the cost estimate on a solid straw-man architecture Base Architecture on Existing Technologies National Grid believes in a low risk approach to deploying new systems. The preference is to buy commercial products to support business needs rather than designing, developing and maintaining in house business systems. Based on KEMA s experience with the product vendors in this domain space, none of the vendors have an enterprise class turnkey solution for this work. KEMA has taken the approach to configure and extend the enterprise class products that support the communication requirements of communicating with potentially hundreds of substation data concentrators which may have as many as IEDs connected to each of them. National Grid 3-1 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

163 Attachment COMM 2-8(b) Page 27 of 160 All of the components within the substation architecture are based on existing technologies that have a proven industry installed track record. All of the components within the enterprise architecture, except for the federated database component, are comprised of existing technologies that have a track record at National Grid. The storage components are a RDBMS, a content management system and an EMS historian. The required systems are either currently deployed at National Grid or National Grid has used similar technologies when deploying similar operational and decision support systems Reuse National Grid s Enterprise Integration Architecture National Grid takes a pragmatic approach when it comes to integrating existing business systems with new systems. The ELSSI straw-man integration architecture is based on using National Grid s pragmatic hybrid approach to integration and is proposing to use National Grid s existing infrastructure tools, which include: Seebeyond s egate to develop the required web services for the new functionality and interfaces for the new data ingest set of functionality not currently commercially available IBM s DataStage to develop the data QA/QC and cleansing routines and data quality reports, to load the historical point data, the current operations and maintenance point data and the point data from the various field devices into the ELSSI data warehouse IBM s Websphere Federated Database Information Integrator to provide the existing client applications client/server ODBC/JDBC type of access to both existing operational and maintenance systems as well as the new ELSSI non-operational point data and potentially to provide access to non relational sources such as the proposed ELSSI content manager and the existing enterprise EMS historian (assumed to be PI in the long term) IBM s Websphere MQ to provide the queuing and persistence required for asynchronous guaranteed messaging Map Functional Requirements to Existing National Grid Owned Vendor Products The list of vendor products that make up the straw-man architecture is all currently owned by National Grid and provides the majority of the functionality required to build out the ELSSI architecture: Oracle s RDBMS Documentum s Content Management Server National Grid 3-2 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

164 Attachment COMM 2-8(b) Page 28 of 160 OSIsoft s PI LiveData s RTI Server IBM s DataStage IBM s Websphere MQ Business Objects Crystal Reports SoftStuf s Wavewin Novatech s Orion The only proposed new technology has been used previously by National Grid in an earlier version: IBM s Websphere Federated Database Information Integrator (aka Data Joiner, DB2 II) Ensure Scalability for Large Number of Substations The goal of the architecture is to ensure National Grid can rollout the substation automation equipment to 35%-60% of its substations. To support the data volumes and data latency requirements as specified in detail in Section 4, KEMA recommends using proven enterprise level data storage components: Oracle, Documentum, PI as well as enterprise class data ingest components such as LiveData to ensure the architecture will scale over time as the number of automated substations grow. Another key to ensuring scalability is to design and state the requirements now for the data ingest server to be able to take advantage of both multi-cpu servers and potentially GRID-based application servers. This means requiring the future product and/or developed software to use the application server s functionality for supporting resource management, multi-threading and partitioning of the components. The analysis vendors in this domain space are developing data analysis, data filtering and data reduction software. So while currently it is a human intensive job to review each of the non-operational files that get generated, KEMA s proposed design is to support the automation of the first pass screening process so rather than needing to go through 10,000 s of files for a large event, the analysis engines will have generated the list of files for further review and reduce the set of files that need to be looked at by humans substantially. National Grid 3-3 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

165 Attachment COMM 2-8(b) Page 29 of Provide Component Base Architecture The goal was to provide a component based architecture that supports incremental business releases that stand on their own and provide National Grid value. KEMA has proposed two business releases BR1 and BR2 to implement this architecture BR1 ELSSI Data Warehouse, ELSSI Content Manager, ELSSI Data Browser, ELSSI Data Classify Component, ELSSI Data Loader Component, ELSSI Content Loader Component and the ELSSI Federated Database Component This release will support the existing field devices being collected by Wavewin as well provide an enterprise storage repository that contains all current data files and data points. It will provide an end-user ad-hoc browser for the enterprise set of data repositories. National Grid can choose how much existing data should be loaded in the ELSSI data warehouse as well as gauge how many existing systems should be integrated via the ELSSI federated database component to support analysis across functional systems. From a pragmatic point of view, the existing SoftStuf s Wavewin product can provide all of the data ingest functionality that National Grid needs for its current number of automated substations. The proposed ELSSI data ingest component can be deferred for a few years if National Grid so desires since it is designed for higher data volumes. This gives the market place a chance to catch up as vendors develop new capabilities as well as providing National Grid a chance to build out the ELSSI data repository, data browsing and review and prototype high-end analytical packages to look at the operational and nonoperational data BR2 ELSSI Data Ingest Component This release will support integrating the substation data concentrators with the newly deployed ELSSI enterprise storage components. In Section 5, KEMA recommends not to immediately commence the full proposed deployment plan as outlined in two business releases but rather, since so much of the architecture is based on commercial products that National Grid already owns, to put together a pilot ELSSI project that prototypes as much of the ELSSI enterprise components as National Grid needs in order to do engineering and analysis using the operational and non-operational data that has been loaded into the proposed ELSSI prototype Base Cost Estimate on Solid Straw-man Architecture KEMA would like to reiterate that the proposed straw-man architecture detailed in Section 4 is not the recommended detailed ELSSI architecture for deployment. It is a representative architecture for driving a cost and deployment estimate. KEMA is proposing that National Grid does both prototyping of the various architecture components and perform vendor RFP evaluations based on the requirements gathered during the pilot prototyping project to develop the detailed ELSSI architecture that should be deployed. National Grid 3-4 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

166 Attachment COMM 2-8(b) Page 30 of 160 Based on KEMA s understanding of the business requirements, the National Grid IT guidelines and KEMA knowledge of the problem domain market space, this report shows a straw-man architecture that is detailed enough to drive both a reasonable cost model and to develop an implementation plan. The implementation plan developed by KEMA was built as a true schedule so it is fully linked based on technology dependencies and is loaded with resources. The resources were assigned in a fashion that minimizes the amount of National Grid resources that are required to deploy ELSSI enterprise architecture. Deployment Schedule: 19 months, $3.9M BR 1: 14 months, $2.6M BR 2: 13 months, $1.3M Exhibit 3-1: Cost and Duration for BR1 and BR2 National Grid 3-5 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

167 Attachment COMM 2-8(b) Page 31 of 160 Exhibit 3-2: Scope of BR1 and BR2 3.2 Cyber Security The primary goal of the Cyber Security aspect of the ELSSI project is to protect the critical functions from misuse leading to reliability issues and outages, inadvertent or unauthorized release of data, and damage to equipment from improper configuration or use. While security issues will always take a second seat to the operational issues associated with data access and remote management, the security issues always need to be considered in any decision process undertaken. The NERC CIP standards deal primarily with Bulk Electric System reliability issues and how the cyber assets associated with operationally critical assets need to be protected (although the current standards are subject to some scope creep and interpretation). The NERC Cyber Security standards have always been a minimal floor upon which additional actions can be layered; therefore, additional, prudent cyber security actions can and should be applied where they make sense, i.e., consistent training, consistent rules for password construction and maintenance, and application to assets that National Grid considers critical such as some distribution circuits, even if they are generally out of scope to a NERC standard. National Grid 3-6 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

168 Attachment COMM 2-8(b) Page 32 of 160 New substation design will likely be used in critical locations (whether the NERC or National Grid definition); therefore the design should include required and/or prudent security for consistent procurement, implementation, training, documentation, etc. Business requirements should always drive the security requirements. Business requirements dictate the protection of critical functions. Security should not be implemented to just to do it. When implementing security, be careful of improper application of security technologies, e.g., active scanning of critical computers (causing malfunctions or reboots), or automatic lockout of operator accounts after failed password attempts. Consideration must be given for alternative mechanisms, such as passive analysis rather than active scan to test for vulnerabilities on a live running system; and, visual verification of operator function inside 24x7 control room. Where there are not strong business requirements, or operational needs, such as maintenance and engineer access, it is possible to implement stricter requirements for password lockout, monitoring, and logging, since the use is generally non-operational, and additional security and all of its implications can be more tolerated. All other factors being equal, the higher security, reliability, or integrity data location should control what data is being transmitted, when it is being transmitted, and to what devices the data is being transmitted. This is essentially a corollary the Biba information flow model, which describes information flows from one integrity domain to another. The Biba model prevents data flowing from one integrity level to a higher integrity level, but allows information to flow from a higher integrity level to a lower one, thus protecting the data integrity of the higher integrity environment. The Biba model does not specifically describe the initiation direction of the information flow, but it is easier for the high integrity environment to control the initiation, as long as the target of the information flow is known and consists of a relatively small number of nodes. In the ELSSI case, the substation environment is the higher integrity level (since it is the primary source of the data), and the data warehouse is the lower integrity level. Thus, the use of push technology ensures that the integrity of the substation environment is maintained. See Section for additional discussion of push versus pull of data. Additionally, by affording the substation control over how often it communicates, and how much data it sends, the reliability and performance of the substation automation environment is maintained. A properly configured substation environment will not overload its processor or networks when sending data, which is a possibility if the requests are unscheduled and uncontrolled from outside the environment. As a side benefit, the substation environment can provide near-real-time transmission of data to the data warehouse in response to an operational event. Once the data transmission issues are resolved, there still remains the remote access to substation equipment for purposes of maintenance and authorized modification of settings. This maintenance access can be controlled by limiting the number of sources of remote access, for example, by the use of an access server (or servers). An access server works by requiring the relay technician to authenticate to the access National Grid 3-7 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

169 Attachment COMM 2-8(b) Page 33 of 160 server, and then using the access server to log into the substation environment. This authentication provides an additional level of access control (i.e., identifying the specific technician), as well as providing an audit trail to determine which substation devices were accessed, and when they were accessed. In the Microsoft Windows environment, for example, a Citrix server can perform this function. These concepts allow the implementation of relatively simple rules for the perimeter firewall: Allow outbound connections from the substation only to the data warehouse environment, for data transport protocols (e.g., ftp, DNP, proprietary IED protocols) Allow inbound connections to the substation only from the access server environment, for maintenance protocols (e.g., ssh, telnet) Deny all else In addition to the rules, both successful and denied accesses should be logged. Successful accesses provide an audit trail of when and what has been accessed; and denied accesses provide an initial indication of an attempted intrusion into the network. Both of these log types are required for compliance with the NERC CIP standards. The security, integrity and confidentiality of the access could be augmented by the use of an authenticated VPN, to further control and restrict what environments could connect to each other, and to protect the data from observation while in transit. In general, an access method that encrypts the access passwords of the substation equipment should be used, since the same access passwords could be used locally at the substation to access the equipment, and potentially modify settings without providing an audit trail. In order to provide control over the access into the substation environment, a firewall must be implemented between the external network and the internal substation network. The NERC standards to not impose any requirements on the quality of the firewall used, only that the firewall function be used. Firewalls generally fall into two classes: proxy and stateless. A Proxy firewall is generally more secure, but is very specific to the protocols and functions used. The proxy firewall may be written to allow specific subsets of commands to specific authenticated individuals, for example, read-only commands, to one set of authenticated users, while configuration modification comments to a different set of authenticated users. Since proxies are very specific to protocol implementations and versions, they are difficult to write and maintain, and are not used extensively. Stateless firewalls, on the other hand, inspect each packet at the address and protocol level to determine if the communication is allowed. Given the example above, a telnet (or ssh) protocol packet would be allowed to be initiated from the access server to the a node in the substation environment, but no inspection of the specific configuration commands within the packet would not necessarily be performed. Stateless firewalls range from relatively simple National Grid 3-8 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

170 Attachment COMM 2-8(b) Page 34 of 160 routers containing access lists (such as a Cisco router), or may be a more sophisticated dedicated firewall device. Some more traditional firewalls also provide termination and configuration of VPNs. Either firewall type, when properly configured and monitored, will meet the requirements of the NERC CIP standards. Remote access to the substation environment for the purposes of changing settings is more a policy issue than a technology issue. A number of questions must be discussed and answered by National Grid before providing remote access. These questions include: What technology prevents someone accessing an IED remotely for purposes of accessing data from changing a setting? If a setting is changed, what is in place to detect the change? How long does the detection process take to find the change? What happens in the mean time? If available, a proxy server (coded in the access server) could provide a technical solution to some of these questions. National Grid will need to internally discuss these issues and perform a risk/benefit analysis to determine whether to allow remote access, and if so, how to monitor the actions performed. A communication multiplexer in combination with a router and/or firewall can provide shared use of a common communication medium such as a SONET network. This provides operational, non-operational, corporate access, and other uses (such as security and camera) on a single communication pipe (of high speed) as long as the multiplexer and router/firewall provide the required isolation between data streams. The NERC CIP Standards require that changes must be tested before implemented in live systems. In the substation automation environment, this is especially true for firmware updates or patches. The specific requirements for testing are less well defined for configuration or parameter changes (some changes may not need to be tested or be able to be tested). A change management process, including approval to implement the change, must be implemented. This process should include a description of the test and implementation process, as well as a general description of thee kinds or classes of changes that do not need to be tested, or cannot be tested in an isolated environment. Communications technology is currently a driver for NERC CIP Standards compliance. Routable protocol (e.g., IP) connections into a substation bring the systems into scope; while non-routable protocols offer an exemption. This exemption is a stopgap measure, however, since most implementations will be forced into migrating to routable protocols to provide additional services (access to non-operational data, access to relay settings, and increased SCADA or other operational data requirements). Non-routable protocols are not inherently more secure, they are simply more self-containing if the communications line is compromised. Encryption of traditional serial communication has been offered as a potential solution, but industry is reluctant to implement due to a number of issues, including increased overhead, costs, and inability to diagnose problems. The industry does not generally perceive a problem with operational data communication that encryption can solve within the constraints imposed by the existing communication implementation. National Grid 3-9 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

171 Attachment COMM 2-8(b) Page 35 of ELSSI Substation & System Architecture Analysis and Recommendations 4.1 Overview The ELSSI enterprise architecture is defined in this document using a set of architectural views: context view, system component view, system architecture view, software component view, deployment view and a data view. Each of these views emphasizes a different aspect of the ELSSI enterprise architecture. This architecture is termed a strawman architecture. It is representative of what the final ELSSI architecture will look like, but it is not meant to be the ELSSI implementation design. As a representative design it has enough detail to identify the meaningful architectural components required but it is not a recommendation of the final solution set of vendors or the exact number and content of the automated integrations to be implemented with the ELSSI data repositories. It is mapped to a representative set of current software vendors that supply these type of functions in order to generate a better cost model for what it takes to develop an enterprise architecture that will support the deployment of substation automation devices for 20% or more of the National Grid substations. Subsets of the architecturally significant use cases are mapped against the architecture components using sequence diagrams to highlight the different integration issues that the proposed enterprise architecture must address System Context View The system context diagram (Exhibit 4-1, page 4-6) denotes the scope of the ELSSI project. Depicted on this diagram are all of the existing systems at National Grid that could potentially be integrated with some information from the ELSSI set of data repositories to provide additional value for National Grid business users. This integration could be automated interfaces or manual interfaces. This diagram merely identifies the need for information flow, not the mechanism to implement it. The high level data flows are labeled with the representative categories of data that either will be available from the ELSSI data repositories or will be used in conjunction with the ELSSI information and made accessible to all interested users via the new enterprise integration and deployment infrastructure being recommended to support ELSSI: data collection, data classification, data access, data correlation, user interface and data management requirements to maximize the value of the information being collected in the field System Component View The system component diagram (Exhibit 4-2, page 4-9) denotes the major functional components required for the system. The diagram shows the new enterprise level components as well as the components from the operational environments in the control center and in the substations that will be integrated with the enterprise components. National Grid 4-1 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

172 Attachment COMM 2-8(b) Page 36 of Software Component View The software component diagram (Exhibit 4-3, page 4-11) shows the various products software stacks and how they are layered when deployed together. The main objective of the software component view is to see which components from which of the software vendor products will be used to comprise the ELSSI enterprise architecture and to get a breakdown of how much new development from an integration point of view will be required to enable the business functionality required by the ELSSI project System Architecture View The system architecture diagram (Exhibit 4-4, page 4-12) shows the major software components and their configuration required for the system. The diagram shows the major third party software components running on each type of system component as well the significant software components that will need to be developed to support the ELSSI vision. The diagram shows where new development is required and which software components should be used to build the components to stay consistent with how National Grid deploys new applications, new infrastructure technologies and integrates with existing business systems System Integration View The system integration diagram (Exhibit 4-5, page 4-14) displays the major software components and their required components for integration with other ELSSI introduced services, existing business applications and potential ELSSI user interface. The diagram shows the major software components running on each type of system component as well the significant software components required to support the integration of the application, server or service with new services and existing applications. The diagram shows where new development is required and which software components should be used to build the components to stay consistent with how National Grid integrates new applications and new infrastructure technologies with existing business systems Deployment View The deployment views (Exhibit 4-6 and Exhibit 4-7) maps the software components to their hardware components. It shows the required communications between the hardware components Architectural Significant Flows The architectural significant flows are the subset of use cases that illustrate the integration requirements as known and understood currently by the ELSSI project team. These include enough of the use cases to put the integration requirements in context of the overall ELSSI business objectives. National Grid 4-2 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

173 Attachment COMM 2-8(b) Page 37 of Recommended Approach The ELSSI Enterprise Architecture will consist of the components as depicted in Exhibit 4-2. The ELSSI data warehouse server, the ELSSI content management server, the ELSSI federated database integration server, the ELSSI data ingest server, the ELSSI ad-hoc browser, the substation security appliance and the substation data concentrator. The architecture is centered on how to ingest data from the substation data concentrators and provide enterprise access for all the data being generated by the IEDs, power quality meters, non-revenue meters, digital fault recorders and sequence of event recorders. In addition to providing access to substation non-operational data, the ELSSI architecture provides the infrastructure to join with other operational and maintenance systems in order to correlate asset related information with the non-operational information. In the spirit of enterprise system architecture methodologies such as the Zachman Framework, C4ISR, IDEF and the wealth of structured analysis methodologies, each of the views below formally models a set of the requirements that the ELSSI project must address System Context View The system context diagram shows the scope of the ELSSI project in terms of which existing National Grid systems could potentially be interfaced with. This is not an implementation diagram, but is meant to convey the broad potential usage of the ELSSI set of data repositories once they have been built and are being populated. The rectangular boxes are the existing National Grid business systems that will provide information either by directly being loaded into the ELSSI data warehouse or by being integrated with the suite of data repositories via the federated database information integrator component. Whether the systems are having their data loaded into the ELSSI data warehouse or are integrated into the ELSSI architecture with the federated database information integrator component, the National Grid business users will have a wealth of operational and non operational data that can be analyzed and correlated without having to spend time trying to figure out which system contains the piece of information they want. Each data flow is labeled with a high level semantic name that represents the type of information that is being sent to one of the ELSSI data repositories or that is being extracted from one of the ELSSI data repositories. While reviewing the ELSSI system context diagram, it is good to keep in mind which business systems or business processes would be enhanced by correlating their information with: the nonoperational point data managed by the ELSSI data warehouse, the operational data managed by the enterprise EMS data historian, or which could benefit from looking at the non-operational files that are managed by the ELSSI content management system. When it comes time to implement ELSSI, each of these interfaces will need to be further detailed to their enumerated set of attributes that need to be shared between the systems. National Grid 4-3 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

174 Attachment COMM 2-8(b) Page 38 of 160 There are a few source systems that haven t been architecturally connected to the ELSSI project. The first is the current repository for subtransmission facilities in New York. KEMA understands that these are currently in the MapInfo GIS and are anticipated to be migrated to the Smallworld GIS where the rest of the National Grid transmission and distribution network facilities are managed. The others are field devices that provide valuable non-operational type of information. These have been generalized to 900MASS type of devices and though they provide useful non-operational information, the architecture to bring in all of these types of devices has not been analyzed. At first glance, if these are just front-ended with a data concentrator, the existing architecture will support them with little effort. The issue is building out the various communication mechanisms necessary to communicate with these field devices to enable the collection of their transient non-operational information. In many cases the operational values are already being sent to the National Grid EMS systems for use by control center personnel System Component View The following subsections describe the components of the ELSSI enterprise (not substation) architecture ELSSI Data Warehouse Server This component provides the following basic functions to the architecture: a) RDBMS A relational database that manages the non-operational data points that are being collected from the IEDs. This database will also be used to manage the meta-data required to provide efficient data retrieval from the content management system. This database will also be used to contain a copy of the operational data point values that need to be analyzed and correlated with other data values managed by other National Grid business systems in other RDBMS systems ELSSI Content Management Server This component will provide a repository of documents and their histories to be accessible directly by both ELSSI end-users and other applications. This component supports storing files in multiple formats so the non-operational files can be stored in both their native format and the IED industry standard COMTRADE format (see Appendix C). Over time as users collect information from the field in terms of images, wave forms (COMTRADE) and power quality format (PQDIF) files will be automatically cataloged and stored in the content management system. Ad-hoc tools will be provided for users to manually add files of potential interest to the repository. Initially the following types of files will be managed by the content management system. a) IED Wave forms files collected by the data concentrators when abnormal events occur. b) Power Quality PQDIF power quality files collected by the power quality meters continuously. National Grid 4-4 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

175 Attachment COMM 2-8(b) Page 39 of 160 c) IR images infrared images collected by power quality group in the field and then manually downloaded into the ELSSI content management system. National Grid 4-5 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

176 Attachment COMM 2-8(b) Page 40 of 160 Exhibit 4-1: ELSSI Project Scope EVENT OSCILLOGRAPH FILES, SOE FILES DEVICE CONDITION, HISTORY, TEMP, FAULT LOCATIONS CIRCUIT READINGS PEAK USAGE CASE STUDY RESULTS PEAK USAGE, TEMP, LOAD CIRCUIT CONFIG INSIDE FENCE HISTORICAL ASSET COSTS DEVICE CONDITION, HISTORY, TEMP RELIABILITY (ESRI) DEVICE CONDITION, HISTORY, TEMP LOAD F/CASTING PROFILING METER USAGE TLM INTERVAL METER READINGS DISTRIBUTION SWITCH LOGS ASRS DEVICE OUTAGE CODES, OUTAGE, DEVICE IDS, CLEARING DEVICES GATE STATION READINGS, REGULATOR READINGS, METER READINGS DEVICE LOCKOUTS, MOMENTARIES, CONDITIONS, FAULT LOCATIONS RELAY VALUES, HISTORY PROFILING METER USAGE LINE CONDITION, HISTORY, TEMP, CONFIGURATIONS, DEVICE CONDITION, HISTORY, TEMP LINE CONDITION, HISTORY, TEMP, LINE OPERATING STATES, FAULT LOCATIONS, MAGNITUDE DEVICE OPERATING HISTORY, LINE FAULT LOCATION, FAULT MAGNITUDE POWERON INTERVAL METER READINGS RECLOSING OPERATIONS GEMS AIMMS EMS / SCADA EMS HISTORIAN FIST DPS (CYME) SMLWORLD RAM (CYME) FEEDPRO TOA TCM WAVEWIN TRANS. PLANNING MV90 14 DAY FORECASTED SCHEDULES, CURRENT OUTAGES DEVICE OPERATING HISTORY, INTERVAL ERS METER READINGS PQ TIMESERIES READINGS, PEAK LOADS METER REPOS/Y LOSS OF LIFE RATING EQUIPMENT RATINGS ASPEN DEVICE SETTINGS IDS LINE CONDITION, HISTORY, TEMP, CONFIGURATIONS, DEVICE CONDITION, HISTORY, TEMP HISTORICAL FEEDER PERFORMANCE, DEVICE RATINGS PEAK USAGE, DEVICE OPERATING HISTORY, OUTAGE HISTORY, OUTAGE, CAUSES, FORECASTED UTILIZATIONS NET GRID MANAGER FALLS TELEMETRIC NON EMS: LINE RECLOSER NON EMS: LINE RECLOSER PEAK USAGE OPERATIONS, OPERATIONS, STATION STATION READINGS READINGS LIGHTNING STRIKE LOCATION, TIMESTAMP, INTENSITY ELSSI PROJECT DEVICE OPERATIONS, TRANSMISSION FAULT LOCATIONS CAPE DEVICE OPERATING HISTORY, LINE RECLOSING OPERATIONS CLEARANCE INFORMATION PEAK USAGE National Grid 4-6 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

177 Attachment COMM 2-8(b) Page 41 of ELSSI Ad-hoc Browser This component will be a web application that provides a rich set of search and query functionality for the end users to access the ELSSI set of data repositories. This component shall provide the user the ability to retrieve the non-operational files in either their native format or in their industry standard format to be used with IED vendor specific tools or with 3rd party analysis tools Federated Database Information Integrator This component shall provide the logical independence and loose coupling to the various existing application data stores and databases. From a National Grid application point of view, it will provide a declarative (SQL) access mechanism that gives National Grid applications access to a federated set of systems. It shall allow the ELSSI ad-hoc browser application to perform traditional RDBMS correlations and aggregations on its configured set of data sources. It provides performance by optimizing its queries and by caching much of its set of sources data in a local RDBMS ELSSI Data Ingest Server This component shall provide the data ingest, data classification and data loading services required to get the non-operational data points and data files loaded into the ELSSI data repositories. The technologies used to build these components shall be selected and built assuming they will be allowed to run in a GRID environment. a) ELSSI Data Ingest This component shall provide the connectivity between the ELSSI enterprise components and the substation data concentrators. This component must support both the FTP protocol for the transfer of non-operational files and the DNP3 protocol over TCP/IP to support the collection of nonoperational data points required for maintenance. b) ELSSI Data Classify This component shall provide the classification of the data being ingested. It shall publish the data required by the condition based maintenance and condition based inspection analytical engines. It shall generate the meta-data required to load the non-operational files into the content management system. It shall generate the input format required by the data loader of nonoperational data point values. c) ELSSI Data Loader This component shall perform data quality checks and bulk load the non-operational data point values into the ELSSI data warehouse. National Grid 4-7 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

178 Attachment COMM 2-8(b) Page 42 of 160 d) ELSSI Content Loader This component shall perform the data loading of non-operational data files into the content management system ELSSI Data Mart Server To be consistent with National Grid s Hub-in-Spoke 4 approach to deploying data warehouses and data marts, KEMA have included a data mart server component as well as the data warehouse server. Initially it is envisioned that the users will have more need to look at the discrete event values and that once the data warehouse has been populated with a certain critical mass of data, then the need for the deployment of a reporting data mart will arise. Over time, both asset management and engineering will probably want to take advantage of the power that a well designed data mart with dimensions that may include: asset utilization, capacities, historical performance, historical reliability, forecasted demand, forecasted performance, forecasted reliability, asset costs, forecasted asset maintenance costs, historical revenue by set of assets and forecasted revenues by sets of assets Substation Data Concentrator This component will cache the information from the IEDs, RTUs, meters, digital fault recorders (DFRs), and sequence of event recorders (SERs). This component will be responsible to establish an FTP connection with the ELSSI data ingest component and publish, based on an event or periodically, the data it has cached from its various input devices. This component in the architecture should also be the component within the substations that needs to follow the corporate National Grid security and NERC cyber security guidelines for the management of user ids and passwords Substation Security Appliance This component shall be used as the demarcation node from a security point of view. It will act as the firewall for the substation local area network. 4 Hub-in-Spoke, also known as Hub-and-Spoke. This architecture has a number of spokes jutting outward from a central hub. A component is selected to be a hub, and the paths that lead from the hub to other components are considered spokes. National Grid 4-8 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

179 Attachment COMM 2-8(b) Page 43 of 160 Exhibit 4-2: System Component Overview Software Component View The next diagram, the Software Stack Diagram in Exhibit 4-3, break downs the significant architectural components into a layered set of functionality. This is the functional software stack that the component will need to support. For each component, the stack starts with the operation system, then any layered third party commercial products that are being envisioned to build the functionality around and then the envisioned custom software component. All custom components have been highlighted with a fill pattern and a gray fill color. Some of the stacks have no custom software components on them. These system components are comprised solely of third party products that need to be configured to meet the ELSSI requirements they are fulfilling. The third party vendors listed in this diagram are being used to generate a reasonable cost estimate. If the ELSSI project gets approved, KEMA assumes that National Grid will look at the identified vendors that National Grid already owns and evaluate them against ELSSI functional requirements. All new components are expected to go through National Grid s RFP process. Commercial products exist for most of the functionality as envisioned for ELSSI. The portions that are being estimated as software development are described here in the following subsections. National Grid 4-9 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

180 Attachment COMM 2-8(b) Page 44 of The ELSSI Thin Client This component will host the ELSSI developed plug-ins and execute the query and analysis services that are being hosted on the ELSSI Access Application Server and developed with a set of third party vendors: Business Objects, Crystal Reports, PI s Web and Websphere Federated Database Information Integrator. The main function of this software is to provide an ad-hoc browser for the user to query the ELSSI content manager, the existing enterprise EMS PI historian, the ELSSI data warehouse and any of the other existing operational systems such as AIMMS, IDS and ERS via the third party federated database component. The federated database component will need to be piloted to see if it can be used to access the content management system and the existing (currently New England only) enterprise EMS PI historian (see Section 5.2, page 5-3) and how well it can be accessed from Business Objects. One of the key requirements is to provide easy access to the distributed set of repositories and to allow the users to specify the format of the result non-operational data files. They will be allowed to return the files in either their IED vendor native format to be used with IED vendor desktop analysis tools or in IED industry standard format (COMTRADE, PQDIF) to be used with third party analysis packages such as Wavewin The ELSSI Content Application Sever This component will host the custom gateway services provided by a third party vendor such as Trinity on top of the native Documentum product. These services will support ELSSI tools like the Data Loader that needs to automatically store data within the Documentum content server The ELSSI Data Ingest Server This component will host four ELSSI custom software components: a) ELSSI Data Ingest is envisioned to be built around a third party product like the LiveData RTI Server to support the data collection of information from the substation data concentrators. The data ingest component will need to support both FTP for non-operational files being generated by the IEDs and cached at the substation data concentrators and DNPi for the non-operational point data values. b) ELSSI Data Classify is envisioned to be built around a set of third party products like IBM s Datastage and MQ. It will publish data for both the ELSSI Data Loader module and for the continuous condition based maintenance and continuous condition based inspection analytical engine. All meta-data required to be generated for the ELSSI Data Loader to used when storing the files in the content manager will be generated by this component. This component will also be used to classify existing data sources and pass them to the ELSSI Data Loader so they can be loaded into the ELSSI data warehouse to populate the ELSSI data warehouse with historical asset performance information, historical non-operational event files, historical sequence of event files, historical power quality images and historical power quality indicators. This component should National Grid 4-10 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

181 Attachment COMM 2-8(b) Page 45 of 160 also provide the data quality metrics using IBM s DataStage that ELSSI s users may need to judge the confidence they can have to make decisions based on the information managed within the ELSSI data repositories. c) ELSSI Data Loader is envisioned to be built around a set of third party products such as IBM s Data Stage to load point data into the ELSSI data warehouse. d) ELSSI Content Loader is envisioned to be built using third party products such as Trinity s custom gateway services for Documentum that supports web services to be used load files into Documentum. Exhibit 4-3: Software Stack Diagram Substation Data Concentrator Novatech Orion Embedded CE Derivative Substation Security Appliance Novatech SSA Linux ELSSI Thin Client Plugins IE Browser Windows XP ELSSI Access Application Server Query / Analysis Services Business Objects.NET IIS Windows 2003 Crystal Reporting Services ELSSI Content Application Server Trinity Gateway Services Documentum Webtop Apache / Tomcat AIX ELSSI Content Loader ELSSI Federated Database Integration Server WebSphere Information Integrator Server DB2 UDB Server ELSSI Data Ingest Server AIX Data Load IBM / Ascential DataStage Data Classify WebSphere MQ Windows 2003 LiveData RTI Server Data Ingest WaveWin ELSSI Content Server Documentum Content Server AIX ELSSI Data Warehouse Server Oracle Server AIX System Architecture View The system architecture view attempts to combine the high-level system functional components and maps their software components to them. This view shows the substation major components that will interact with the ELSSI components. Each of the ELSSI components have a list of the major third party components envisioned to be included and the ELSSI components that will need to be developed. The system architecture view includes a sample of the proposed applications and how they will be integrated with the ELSSI components. The diagram s legend actually highlights the three different types of new components: ELSSI front ends, ELSSI services and non-elssi identified productivity enhancing National Grid 4-11 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

182 Attachment COMM 2-8(b) Page 46 of 160 applications such as a Case Study Management application for planning that would benefit greatly by being integrated with the ELSSI data repositories. Exhibit 4-4: Strawman System Architecture System Integration View The system integration view shows for each component and its set of integration mechanisms that will allow the software component to integrate with other components. KEMA takes a pragmatic approach with respect to the reuse of integration mechanisms that currently exist and propose utilizing the existing National Grid EAI infrastructure tools to develop the new interfaces that are required. Existing desktop client/server tools such as AIMMS and GIS will utilize the federated database information integrator WII client to communicate with the ELSSI data repositories. Asynchronous integration requirements will utilize the egate toolset and use MQ as the messaging medium. Older applications can still use their ODBC or JDBC type of interfaces to access ELSSI data repositories. National Grid 4-12 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

183 Attachment COMM 2-8(b) Page 47 of Major Component Integrations Substation Data Concentrator -> ELSSI Data Ingest ELSSI Data Ingest -> ELSSI Data Classify ELSSI Data Classify -> CBM/CBI analytical engines ELSSI Data Classify -> ELSSI Data Loader ELSSI Data Classify -> ELSSI Content Loader ELSSI Data Loader -> ELSSI Data Warehouse ELSSI Content Loader -> ELSSI Content Server ELSSI ad-hoc browser -> ELSSI Data Repositories The ELSSI Data Ingest component will utilize the functionality of the a 3 rd party tool such as LiveData s RTI Service to communicate with the substation data concentrator via both DNPi and FTP. These two components will be hosted within a J2EE application server environment and should use the egate and JMS services to integrate with each other. The ELSSI Data Classify component will use egate and MQ services to publish messages for the analytical engines such as the condition based maintenance and the condition based inspection programs. These components will be hosted within an application server environment and should use the egate based web services to integrate with each other. The ELSSI Data Loader component will use DataStage to communicate with the ELSSI data warehouse. The ELSSI Content Loader component will use the services provided by the ELSSI Content Loader built on top a 3 rd party such as Trinity to load the files into the ELSSI Content Server. The ELSSI ad-hoc browser application will use the ELSSI Federated Database Information Integrator component to integrate and communicate with the ELSSI data warehouse, the ELSSI content management system, the enterprise EMS PI historian and other source systems such as AIMMS, IDS and ERS. National Grid 4-13 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

184 Attachment COMM 2-8(b) Page 48 of 160 ArcGIS Desktop -> ELSSI Data Repositories Wavewin -> ELSSI Data Repositories The existing desktop tools will have a choice to use ODBC/JDBC or the WII client to access the data accessible via the ELSSI Federated Database Information Integrator component. The existing desktop analytical tools such as Wavewin will have the ELSSI browser format the files in the proper format required and Wavewin will use ODBC to access point data values from the ELSSI data warehouse. Exhibit 4-5: System Integration View New Application Development within GIS Environment ELSSI integration development using egate and MQ suite ELSSI application development using Business Objects, Crystal Reports,.NET Framework CBM/CBI analytics ELSSI Data Classify Services Desktop Tools (WaveWin) ELSSI Ad-Hoc Browser App ArcGIS Analysis Tools WII Services ODBC/ JDBC ELSSI DM Loader DataStage ELSSI Access Services Websphere Information Integrator (WII) WII Services ODBC/ JDBC ELSSI Data Warehouse (Oracle) WII Services DataStage ELSSI Data Mart (Oracle) WII Services DataStage WII Services AIMMS WII Services ODBC/ JDBC Case Study (Smallworld) LiveData RTI Server ELSSI Data Ingest ELSSI Data Classify Trinity Documentum Services ELSSI Content Loader DataStage ELSSI Data Loader Trinity WII Documentum Services Services Content Server (Documentum) ELSSI Access ELSSI Data Classify Services Services Enterprise Historian Server (PI) COM DPS/RAM (CYME) Deployment View The deployment view is decomposed into two diagrams, one for the data ingest process and one for the data usage. The deployment view is using the National Grid deployment diagram nomenclature to depict the system components, the software components and the communications between the components. National Grid 4-14 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

185 Attachment COMM 2-8(b) Page 49 of 160 Exhibit 4-6: Deployment Diagram- Data Ingest National Grid 4-15 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

186 Attachment COMM 2-8(b) Page 50 of 160 Exhibit 4-7: Deployment Diagram- Data Usage Data View The set of data repositories in the next diagram are all of the existing databases that have been identified so far that need to be evaluated to see if they should have some of their content loaded directly into the ELSSI data warehouse and be refreshed on a periodic basis or if they should be integrated via the federated database information integrator component with the ELSSI data warehouse and provide the users access to current operational and non-operational information. Some of the historical operational time series data that is managed in the enterprise EMS historian will need to get loaded into the ELSSI data warehouse. A good example of this would be putting in the just the peak load values for all distribution feeders and transmission lines. As the users learn what is available, other data from the enterprise EMS historian repository will need to get loaded into the ELSSI data warehouse in order for users to perform relational aggregate analysis. An example of using both the federated database information integrator component and loading additional operational EMS time series data into the ELSSI data warehouse would be to do analysis on the performance of a type of asset by manufacturer. This type of analysis could be easily supported once the operational performance data was loaded into the ELSSI data warehouse from the enterprise EMS historian repository by using the relational aggregation power of the federated database information integrator component to join the National Grid 4-16 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

187 Attachment COMM 2-8(b) Page 51 of 160 nameplate data (that includes the manufacture information) managed within the AIMMS database with the newly loaded operational performance data within the ELSSI data warehouse. Providing a virtual data warehouse has been a consistent requirement throughout both Phase I and Phase II of the ELSSI project so far. The typical trade-offs will need to be made for each of the data sources in the diagram below to decide if they get loaded periodically into the data warehouse or if they should be integrated via the federated database information integrator. This is a trade off between access requirements, duplication of data, data synchronization issues and operational impacts. If users need to use the most current information: require zero-latency to derive the most benefit from their analysis, then the sources should be accessed directly via the federated database information integrator component. If the source business systems are not always available, or have utilization issues, or have response time issues, or have to support guaranteed service level agreements or have time critical deadlines for reporting purposes, than these source databases should have their information periodically loaded into the ELSSI data warehouse. Exhibit 4-8: System Repositories National Grid 4-17 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

188 Attachment COMM 2-8(b) Page 52 of Use Case Scenarios and Sequence Diagrams Introduction and Definitions The sequence diagrams in this section describe a structured series of representations of potential future state behavior as a series of sequential steps over time. They have been used to depict workflow, message passing and how elements in general cooperate over time to achieve various results. These scenarios are not meant to be comprehensive but represent architecturally interesting examples of processes from an integration perspective that show each ELSSI component and how they might interact within the context of proposed business scenarios that would be enhanced with access to ELSSI information. A sequence diagram represents the interaction between different objects in the system. The important aspect of a sequence diagram is that it is time-ordered. This means that the exact sequence of the interactions between the objects is represented step by step. Different objects in the sequence diagram interact with each other by passing messages. In order to help the reader to understand these diagrams a brief overview is provided here. Each sequence diagram has two dimensions: the vertical dimension represents time, and the horizontal dimension represents different systems or components. Normally time proceeds down the page, as is the case with all of the sequence diagrams shown in this report. There is no significance to the horizontal ordering of the instances. Call (message) arrows are usually arranged to proceed in one direction across the page; however, this is not always possible and the ordering itself does not convey information. A component is shown as a vertical dashed line. This line represents the existence of the component at a particular time. An object symbol is drawn at the head of the line with a label showing the name of the component or system. An interaction between two components is represented by an arrow specifying a communication, with a sender role and a receiver role identified by the vertical lines on the diagram. For some interactions the arrow may start and finish on the same line. Each arrow is also labeled with the name of the operation to be invoked or a description of the type of data being communicated. An arrow may also be labeled with a condition and/or iteration expression. There are different styles of arrows shown on the diagrams and these represent the following: Full Arrowhead The full arrowhead is used to denote operation calls from one component to another. Partial Arrowhead The partial arrowhead is used to denote asynchronous communication; that is, no nesting of control. The sender dispatches the message and immediately continues with the next step in the execution. National Grid 4-18 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

189 Attachment COMM 2-8(b) Page 53 of 160 Curved (Recursive) Arrow The curved arrow that starts and stops on the same line indicates the initiation of a sub process by the system or component indicated at the top of the line. Dashed Arrow The dashed arrow is used to denote a reply from operation call. Exhibit 4-9: Sequence Diagram Legend System Operations The following diagram represents what happens in the event of an outage where an IED provides alarm information and distance to fault back to the control room operators. In this example there is no direct interaction with ELSSI. This example represents what might happen with the ability to automatically collect distance to fault information from IEDs. If the control room operators wanted more information about alarms, more components (ELSSI browser, ELSSI database) could be added to the diagram to show the querying of the ELSSI database for more detailed information. National Grid 4-19 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

190 Attachment COMM 2-8(b) Page 54 of 160 Exhibit 4-10: System Operations Sequence Diagram Event Investigation The following diagram represents an engineer using an analysis tool to retrieve and analyze data relating to an event on the electric system. In the example the analysis tool is Wavewin. Wavewin is also used as the data ingest engine for non-operational data but in the context of this example it is being used to exploit its abilities for analyzing waveform data retrieved from IEDs. The proposed ELSSI solution involves storing waveforms in a content management system such as Documentum (already in use at National Grid) instead of the file system currently employed by Wavewin. In this proposed architecture Wavewin would receive input from the engineer and pass search parameters to the ELSSI browser to determine the location of the required data. The ELSSI browser in turn would request the appropriate data from Documentum and (if required) the ELSSI data warehouse. The data would be returned to Wavewin for analysis. National Grid 4-20 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

191 Attachment COMM 2-8(b) Page 55 of 160 Exhibit 4-11: Event Investigation Sequence Diagram In the example presented here there is a second request from the engineer who, having analyzed the data from the first request, decides that sequence of event information is required to complete the analysis. Again in this case the request for data is handled by the ELSSI browser, which in turn retrieves the SOE file from Documentum and returns it to the engineer for analysis Distribution Planning The following diagram represents a scenario where a distribution planning engineer, through the use of a GIS interface, interacts with the ELSSI federated database. This virtual database provides a single point of reference to look for data but retrieves the required data from other systems and passes the information back to the requesting system, in this case the GIS being used by the distribution planning engineer. In the example show the study being performed requires the access of data from the ELSSI database itself as well as from other systems via the ELSSI federated database component such as load flow information, National Grid 4-21 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

192 Attachment COMM 2-8(b) Page 56 of 160 and transformer load information. The sequence diagram reuses the existing integration between the GIS and CYME for the network model information. This allows the engineer to save and run multiple studies and to use the power of the GIS to perform thematic comparisons between the studies. Note that the data being retrieved may come initially from IEDs located in substations or out on distribution feeders. Exhibit 4-12: Distribution Planning Sequence Diagram Reliability Analysis The following diagram represents what happens when a set of devices, lines and their historical performance is analyzed from a reliability perspective and correlated with weather and maintenance activities (both historical and planned). Study of this example will show that it is very similar to what currently occurs at National Grid. The difference is that ELSSI provides a traffic cop role to enable access to data and also provides access to non-operational data if required (not shown on this diagram) and operational data not currently readily available such as distance to fault. This example also assumes that a GIS is being used as the engineer s interface to the system and for overlaying results that are returned National Grid 4-22 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

193 Attachment COMM 2-8(b) Page 57 of 160 using the capabilities of the GIS. As with other sequence diagrams, the systems shown are used for illustrative purposes and are representative of systems currently in use at National Grid. Exhibit 4-13: Reliability Analysis Sequence Diagram Condition Based Maintenance and Inspection The following diagram represents how condition based maintenance (and inspection) might benefit from IED data. In keeping with the goal of this section to illustrate the integration requirements of the various ELSSI components, this example is more like the System Operations diagram in that the information flow shown is not started by a request from an engineer but starts with data being collected in the field by IEDs and sent to the ELSSI data warehouse. In the data warehouse the data is analyzed and classified based on criteria set by the Predictive Maintenance department and data is sent to AIMMS where it is analyzed by maintenance and inspection modules. Any criteria that AIMMS determines requires attention will result National Grid 4-23 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

194 Attachment COMM 2-8(b) Page 58 of 160 in calls to ischeduler to generate work orders for inspection. The results of the inspections are entered into AIMMS once complete. Exhibit 4-14: CBM/CBI Sequence Diagram Maintenance Following on from the previous inspection example, the following example looks at an example where maintenance is determined to be required on an asset. In this case the AIIMS inspection and maintenance modules raise alarms based on data received from ELSSI and alerts an engineer. The engineer (though ArcGIS in this example) looks at the relevant devices and a request is passed to ELSSI for more data. Various information is returned to the engineer depending on the type of equipment etc. such as infra-red scans, voltage and current history, and non-operational data which are returned to the engineer for analysis. National Grid 4-24 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

195 Attachment COMM 2-8(b) Page 59 of 160 Exhibit 4-15: Maintenance Sequence Diagram Dynamic Ratings The following diagram depicts the dynamic rating of equipment, which is not currently performed at National Grid. This may be applied to a scenario where a poorly performing asset is de-rated or where several assets need to be re-rated due to peak loading conditions. In theory this could be initiated by an event raised by EMS or could be manually triggered such as an event called in from the field e.g. where a line is close to contacting a tree limb potentially causing the line to trip out and other lines to be assessed for higher loading. In this example the EMS requests that the ratings calculator perform the dynamic rating. The ratings calculator updates the equipment model with asset changes, which then cause the equipment model to retrieve current weather, load and other parameters, required. In this case this includes clearances for lines that is provided from TCM. A load flow analysis is then performed on the new system configuration. National Grid 4-25 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

196 Attachment COMM 2-8(b) Page 60 of 160 Exhibit 4-16: Dynamic Ratings Sequence Diagram Meter Data Services The following diagram represents a preliminary use case that shows the use of meter related data within National Grid. This is by no means comprehensive but demonstrates a degree of commonality between IED and interval meter data collection and sharing. To make a true use case this diagram needs to be broken into more discrete areas with a separate diagram for each, order to make it more readable. Although somewhat messy this diagram serves as a good illustration for the case in point. In the diagram, IEDs, intervals and residential meters are represented by actors (represented by stick men) with pale yellow heads. The ovals show actions/processes being taken and the actors with green heads represent user communities that have a need for data originating from meters. Some of these groups are also users of IED data. The actors shown with orange heads are areas where IED data is being used but not data from interval meters. This diagram does not reflect a single repository for interval meter data, which could be used to provide improved validation and analysis as well as more rigorous audit capabilities. National Grid 4-26 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

197 Attachment COMM 2-8(b) Page 61 of 160 Exhibit 4-17: MDS Use Case Diagram 4.3 Data Flows and Data Management Data Flow Definitions Context Diagram The context diagram (see Exhibit 4-1, page 2-3 for a larger version) shows the various National Grid systems that might be required to interact with ELSSI in the future. This was developed to ensure that all potential systems were identified. The flows of information depicted are not meant to represent any National Grid 4-27 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

198 Attachment COMM 2-8(b) Page 62 of 160 specific level of automation or otherwise. The flow of information from one system to another could be automated directly through a point to point interface, it could be automatically implemented through a messaging system based on a service oriented architecture or it could be a manual interface based on a person copying data from one system and entering it into another. The important thing here was to identify and document the systems together with examples of potential data flows. Exhibit 4-18: ELSSI System Context Diagram new ELSSI process PM Programs Apparatus (AIMMS) ERS DPS (Cyme) GIS (Smallworld) RAM (Cyme) TOA Scheduler Incident Investigation (WaveWin) existing NGRID business system Reliability/ Linear PM Programs (ESRI) Load Forecasting device condition, history, temp device condition, history, temp profiling meter usage device operating history, interval meter readings circuit config peak usage, temp, load circuit readings peak usage case study results device condition, history, temp, fault locations 14 day forecasted schedules, current outages event oscillograph files SOE files lightning strike location, timestamp, intensity PQ timeseries readings, peak loads FALLS MV90 Transformer Load Mgmt interval meter readings ELSSI Project none EMS: line recloser operations, station readings Telemetric Sub Trans GIS (MapInfo) 900MASS Devices ASRS OMS (PowerON) GEMS device lockouts, momentaries, conditions fault locations fault location, fault magnitude distribution switch logs gate station readings regulator readings, meter readings device operating history, line reclosing operations device ratings peak usage, device operating history, outage history, outage causes, forecasted utilizations peak usage historical feeder performance, Line condition, history, temp, configurations Device condition, history, temp clearance information Line condition, history, temp, configurations Device condition, history, temp Interval meter readings device outage codes, outage device ids, clearing devices profiling meter usage equipment ratings relay values, history device settings peak usage transmission device operations, transmission fault locations Loss of Life Ratings IDS (Oracle) EMS/SCADA EMS Historian (PI) FIST (Access) FeedPro (Oracle) Line condition, history, temp Line operating states, fault locations, magnitude TCM (Smallworld) Network Grid Manager (PSSE) Trans Planning (PSSE) New Meter Repository CAPE Device Coordination (ASPEN) The following table provides more detail on the information flows depicted in the context diagram. For each information flow, the table contains the existing National Grid system (shown on the outside boundary of the context diagram), the direction of the information flow (i.e. to ELSSI or from ELSSI), a brief description of the flow that indicates typical data items to be passed, and a description of the flow. National Grid 4-28 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

199 Attachment COMM 2-8(b) Page 63 of 160 Exhibit 4-19: High Level ELSSI Business Data Flows Existing National Grid System Direction Flow Name Description AIMMS From ELSSI Device condition, history, temp AIMMS user will benefit from ELSSI data repositories by being able to look up current device conditions, historical device conditions, and current operational factors such as temperature that may affect the performance of specific devices. Device Coordination (ASPEN, CAPE) From ELSSI Relay values, history These coordination programs could benefit from having easy access to both current and historical relay values. To ELSSI Device settings Based on the modeling runs, these programs provide the new settings for the field devices. DPS (CYME) From ELSSI Circuit config, peak usage, temp, load Planning engineers will benefit by being able to get peak circuit configuration information such as switch and fuse open/close status, peak customer usage, peak loads, air and device temperatures during the peak load from ELSSI to be used by their planning tools. EMS (PI) Historian To ELSSI Device operating history, line reclosing operations All operational information in time series form should stay in current environment and just be accessed from that environment. All operational data that needs to have aggregate analysis performed on it should be loaded into the ELSSI data warehouse from this source. EMS/SCADA From ELSSI Fault location, fault magnitude These are the non-operational data values that have been identified so far that should be included in the operational integration between the data concentrator and National Grid 4-29 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

200 Attachment COMM 2-8(b) Page 64 of 160 Existing National Grid System Direction Flow Name Description EMS/SCADA. The EMS/SCADA operators may identify a few more non-operational data values that would bring high value to an ops center. ERS To ELSSI Device operating history, interval meter readings FALLS To ELSSI Lightning strike location, timestamp, intensity Users of ELSSI will benefit by being able to combine this historical information for New York devices with other information available via the ELSSI infrastructure. Other users of the ELSSI infrastructure will benefit by having access to the lightning related information provided by FALLS. FeedPro (Oracle) From ELSSI Peak usage The analytical engine of FeedPro could make use of the peak usage and duration information ELSSI has to re-calculate its inventory of devices ratings. FeedPro will be a good source to populate historical information into ELSSI. Other users of the ELSSI infrastructure will benefit from having easy access to device ratings and historical feeder performance information. To ELSSI Historical feeder performance, device ratings FIST (ACCESS) From ELSSI Peak usage, device operating history, outage history, outage causes, forecasted utilizations Users of FIST reports should be able to easily use the new reporting mechanisms supported by the ELSSI infrastructure to get these type reports or to perform other interesting ad-hoc queries against a wealth of operational and non-operational information. GEMS From ELSSI Gate station readings, regulator readings, The ELSSI infrastructure could be used to provide non-operational gas device readings National Grid 4-30 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

201 Attachment COMM 2-8(b) Page 65 of 160 Existing National Grid System Direction Flow Name Description meter readings in the same manner as it does power device non-operational readings. The other alternative is to use data concentrators and use the current operational integration to add a few non-operational data points to the gas EMS set of values being retrieved from the field. GIS (Smallworld) From ELSSI Circuit readings, peak usage, case study results Assuming National Grid decides to provide their planning engineers a case study management tool within the GIS environment, these types of data flows will benefit planning engineers and support the case study environment that allows them to run multiple iterations and graphically analyze and compare different results within the same case study area. IDS From ELSSI Transmission device operations, fault locations Users of IDS could benefit by having transmission related outage information being integrated in a fashion similar to what IDS is getting from the OMS for distribution outages. To ELSSI Device outage codes, outage device ids, clearing devices Users of the ELSSI infrastructure could use it to get the abundant outage information managed within IDS and correlate it with other related information when performing planning and reliability engineering. Incident Investigation (Wavewin) From ELSSI Event oscillograph files, SOE files Users of Wavewin should have a much easier time searching for related and relevant files to be analyzed with their analysis tool. The ELSSI browser will give the users the option to use native vendor or industry standard formats of these files to be analyzed. Long National Grid 4-31 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

202 Attachment COMM 2-8(b) Page 66 of 160 Existing National Grid System Direction Flow Name Description term process should take advantage of the processing and filter power of Wavewin and have it already have gone through all possible files and tell the engineer which files need further evaluation. Load Forecasting From ELSSI Profile meter usage This application should get this type of information from ELSSI in the future. Loss of Life Ratings From ELSSI Peak usage, durations The loss of life application could make use of peak usage and durations of stress loads when recalculating its inventory s set of equipment ratings. To ELLSI Equipment ratings MV90 To ELSSI PQ time series readings, peak loads Other users of the ELSSI infrastructure will benefit by having access to power quality readings available from MV90 devices. Network Manager (PSSE) Grid From ELSSI Line: condition, history, temp, configurations; device: condition, history, temp Network planning engineers will benefit by having easy access to the network lines and network devices operational and nonoperational data that can be used to calibrate and tune their engineering model. New Repository Meter From ELSSI Profiling meter usage This future system will benefit from the ELSSI infrastructure by using it to get the usage readings from the profiling meters that have been installed. To ELSSI Interval meter readings Users of the ELSSI infrastructure will benefit by having easy access to all interval meter readings. OMS (PowerON) (ASRS) From ELSSI Device lockouts, momentaries, The ELSSI infrastructure could provide the mechanism to provide a one-way integration National Grid 4-32 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

203 Attachment COMM 2-8(b) Page 67 of 160 Existing National Grid System Direction Flow Name Description conditions, fault locations between the EMS/SCADA and the OMS. The diagram shows both PowerON and ASRS since the timing for when ASRS gets decommissioned and ELSSI might be activated is not known. To ELSSI Distribution switch logs When the OMS is used for emergency and planned switching, it s switch logs will be of value to other ELSSI users. Since IDS is the system of record for outage events, KEMA proposes to get that type of outage information from IDS and not from the OMS. RAM (CYME) From ELSSI Device condition, history, temp, fault locations Reliability and planning engineers should be able to get better fidelity models and more what-if iterations based on using all the detailed outage event, device condition information, operating temperatures and fault locations that will be accessible via the ELSSI infrastructure. Reliability/Linear PM Programs (ESRI) From ELSSI Device: condition, history, temp This application should take advantage of the ELSSI infrastructure to thematically analyze both non-operational and operational data values. TCM (Smallworld) From ELSSI Line: condition, history, temp; Line operating states; Fault: locations, magnitude Users of the transmission corridor management system will benefit from being able to quickly and easily thematically display line conditions, historical line performance, historical line configurations and locations of transmission faults and their magnitudes. National Grid 4-33 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

204 Attachment COMM 2-8(b) Page 68 of 160 Existing National Grid System Direction Flow Name Description To ELSSI Clearance information Users of the ELSSI infrastructure as part of a dynamic rating business flow could make use of information provided by TCM such as clearance information. Telemetric To ELSSI Non EMS station readings, line recloser operations Other users of the ELSSI infrastructure will benefit by having access to all of the nonoperational data that the devices monitored by Telemetric could be providing to National Grid. TOA (Scheduler) To ELSSI 14 day forecasted schedules, current outages Other users of ELSSI will benefit from being able to easily access this type of scheduling information via the ELSSI ad-hoc browser or by having it easily integrated with their existing business systems. Trans Planning (PSSE) From ELSSI Line: condition, history, temp, configurations; device: condition, history, temp Transmission planning engineers will benefit by having easy access to the transmission lines and transmission devices operational and non-operational data that can be used to calibrate and tune their engineering model. Transformer Load Mgmt From ELSSI Interval meter readings This application should get this type of information from ELSSI in the future Data Ingest Dataflow Diagram The data ingest dataflow shows the systems involved in collecting IED data. The high level data flows were split into ingest and usage to make the diagrams more easily readable and also to logically separate the requirements for gathering and processing data. These two dataflow diagrams also break out the ELSSI component represented in the context diagram and present more detailed architectural components of ELSSI as proposed by KEMA. This diagram depicts the collection of operational and non-operational data from IEDs, and data from meters and storing it in an ELSSI database and a content server. The content server is to catalog and store non-operational data that is to be kept in its native file format for use by specialized engineering tools. National Grid 4-34 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

205 Attachment COMM 2-8(b) Page 69 of 160 Exhibit 4-20: Data Ingest DFD IEDs DFRs SERs meters Telemetric EMS/SCADA alarm conditions fault locations, fault magnitude GEMS gate station readings regulator station, meter readings WaveWin Substation Data Concentrator digital, analog point values event oscillograph, SOE, PQDIF files ERS data files scalar attribute values ELSSI Data Ingest line recloser readings station readings FeedPro Power Quality digital, analog point values event oscillograph, SOE files PQDIF, IR images historical operations, interval meter readings historical feeder performance, device ratings all files, file classification meta data ELSSI Classify Data all scalar data values operational data points device outage codes, outage device ids, clearing devices ELSSI Data Loader EMS Historian (PI) IDS MV90 ELSSI Content Loader ELSSI Data Warehouse PQ Readings meter data repository ELSSI Content Repository ELSSI Data Mart Service ELSSI Data Mart Meter Data Mart Data Usage Dataflow Diagram The data usage dataflow shows the systems involved in using IED data. As with the data ingest diagram the ELSSI component represented in the context diagram is broken out into more detailed components. National Grid 4-35 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

206 Attachment COMM 2-8(b) Page 70 of 160 Exhibit 4-21: Data Usage DFD IRR (ESRI) PM Program Apparatus (AIMMS) Engineer Incident Investigation (WaveWin) Ad-hoc data request tabular attributes, listings of related files Content Query Processor Content Manager Content Repository ocillograph, SOE files outage events, temp, durations, cause codes, planned PM activities, historical PM activities, forecasted reliability ELSSI Access Service historical device operations, historical loads, durations OMS (PowerON) named case study results fault location distribution switch logs historical feeder peaks, temp, datetime Power Flow Case Study Manager (GIS) ELSSI Data Warehouse calculated device peak calculated date, hourly peak analysis results network model, analysis inputs Transformer Load Mgmt interval meter readings Meter Data Mart historical outage events, datetime, cause codes, durations, clearing devices, partial steps, device operations profiling meter usage Load Forecasting named case study results GIS Asset Repository Reliability Case Study Manager (GIS) network model, analysis inputs analysis results RAM (Cyme) distribution switch logs, linear outage event locations device conditions, historical maintenance line conditions historical maintenance Linear PM Programs (ESRI) OMS (PowerON) PM Programs Apparatus (AIMMS) GIS Asset Repository DPS (Cyme) IED Data IED templates have been developed for the four major classes of IEDs; relays, meters, DFRs and protection devices. The IED templates will provide NATIONAL GRID with the foundation to identify the necessary data and ports for all IED makes and models at NATIONAL GRID today and those that will be deployed in the future. These templates have been delivered in spreadsheet format under separate cover. Please refer to Appendix B Volume and Timing KEMA has genericized all of the possible IEDs, DFRs, SERs and meters that could be connected to the substation data concentrators in the following two tables. These will be used to drive all of the capacity planning numbers that are summarized in the table at the end of this section. To provide a little background and motivation for both the database sizing estimates and the peak backbone communication requirements, KEMA has used the following set of worksheets to generate peak communication requirements that are discussed in detail in Section 5 and both the annual growth rate and steady state sizing estimates for the set of ELSSI data repositories discussed later in this section. All of the columns below that have Operational in their name correlate to the storage requirements of the enterprise version of the EMS (PI) historian. As the ELSSI project progresses and specific IEDs, DFRs, SERs and meter National Grid 4-36 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

207 Attachment COMM 2-8(b) Page 71 of 160 types and deployment numbers are selected, the generic tables below should be updated to reflect the device types that are going to be implemented. The following table models both the data point and the data file sizing characteristics. Exhibit 4-22: IED Data Element Sizing Assumptions Sizing of individual Data elements Assumes 8 bytes of data and 8 bytes of quality, time, and other Data Element Size Bytes attributes Analog 16 Digital 2 Event Files Event File 65,536 # of Events 512 Oscillograph File 2,027,520 Bytes/Event 128 Including event message Kbytes/Event File It is assumed that the entire SER buffer of 512 events gets uploaded Oscillograph Data Bytes/Sample 8 Samples/Cycle 16 Number of Variables 24 Number of Cycles 15 Number of Faults 40 Overhead Factor 1.1 Kbytes /file It is assumed that the entire buffer of 10 Sec. Data need to be uploaded The next table abstracts the characteristics of all possible IEDs into three generic sizing classes: Small, Medium and Large. Exhibit 4-23: Generic IED Characteristics Typical IED Characteristics # of Analog Operational # of Digital Bytes / Scan # of Analog # of Digital Non-Operational # of SOE # of DFR Bytes / Upload Small IED ,616 Medium IED ,824 Large IED ,093,616 The next table defines a generic transmission substation configuration to be: 40 Small IEDs, 10 Medium IEDs and 10 Large IEDs installed. It defines a generic distribution substation configuration to be: 10 Small IEDs, 10 Medium IEDs and 5 Large IEDs installed. The table assumes 30%-40% of the possible data points will be collected from the 60 transmission substation IEDs and 10% of the possible data values will be collected from the 25 IEDs installed in a typical distribution substation. National Grid 4-37 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

208 Attachment COMM 2-8(b) Page 72 of 160 Typical Transmission Substation Exhibit 4-24: Generic Transmission and Substation IED Usage Patterns # of IEDs A point D Point Operational Non-Operational Total Bytes / Burst Accepted A D % of Sec / Accepted Point Sec / Scan Loading/ SER DFR Upload Latency Latency (Sec) Count Scan point Point Interest Sec Scan (sec) Bytes / Upload Burst Bytes / Sec A D A D Small IED , % ,624,640 2,625 Medium IED , % , Large IED , % ,936,160 27,915 Total SA ,840 1, ,219,040 31,198 Typical Distribution Substation ` # of IEDs A point D Point Total Point Count Sec / Scan Operational Accepted Latency (Sec) Bytes / Scan Burst Loading/ Sec A point D SER DFR Point % of Interest Non-Operational Sec / Upload Scan Accepted Latency (sec) Bytes / Upload Burst Bytes / Sec A D A D Small IED % , Medium IED , % , Large IED , % ,468,080 1,745 Total SA , ,782,480 1,964 ` The % of IEDs That get impacted on a major event for burst upload Data is uploaded in the data base within this latency timeframe National Grid 4-38 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

209 Attachment COMM 2-8(b) Page 73 of 160 Exhibit 4-25: Range of System Level Loadings % of Substations participating # of Substation s System Level Communication Loading Analysis Analog Count Digital Count Total Count Operational Data MBits / Scan Burst Loading (MBits/Sec) Non-Operational Data Mbits / Upload Burst Loading Mbits / Sec Transmission 7 7,840 5,040 12, , Distribution 33 6,600 11,880 18, , Total 40 14,440 16,920 31, , % of Substations participating # of Substation s Analog Count Digital Count Total Count Operational Data MBits / Scan Burst Loading (MBits/Sec) Non-Operational Data Mbits / Upload Burst Loading Mbits / Sec Transmission 21 23,520 15,120 38, , Distribution ,000 39,600 61, , Total ,520 54, , , % of Substations participating # of Substation s Analog Count Digital Count Total Count Operational Data MBits / Scan Burst Loading (MBits/Sec) Non-Operational Data Mbits / Upload Burst Loading Mbits / Sec Transmission 42 47,040 30,240 77, , Distribution ,000 79, , , Total , , , , % of Substations participating # of Substation s Analog Count Digital Count Total Count Operational Data MBits / Scan Burst Loading (MBits/Sec) Non-Operational Data Mbits / Upload Burst Loading Mbits / Sec Transmission 74 82,880 53, , , Distribution , , , , Total , , , , National Grid 4-39 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

210 Attachment COMM 2-8(b) Page 74 of 160 The annual growth rate in the table below assumes a 3% installation rate per year. The steady state ten year archive numbers assume a 35% penetration rate for transmission substations and a 20% penetration rate for distribution substations. Exhibit 4-26: Storage Summary 10 Years of Storage (Tbytes) Operational Non-operational Total Transmission 74 substations , , Distribution 220 substations Total , , Growth Volume 3% Peneration/Year Operational Non-operational Total Transmission 7/yr Distribution 33/yr Total Note: This is archival size for 10 years of data storage Note: This is incremental Archival size for 1 year Meta Data Part of Task 2 was to develop a conceptual definition of a meta model and information flow for ELSSIrelated data. Since this report and the work on which it is based was to develop reference systems architecture for enterprise-level integration of substation data at National Grid, a number of possible future state business processes were reviewed with National Grid in the form of UML sequence diagrams (see Section 4.2.8) and other diagrammatic forms, which are presented in other sections dispersed throughout this report. These various tasks represent the start of a meta model of business areas, systems and processes that may be potentially touched by IED data. This section refers the reader to the key meta and high-level model information that is contained in this report Context Diagram The context diagram (see Exhibit 4-2, page 4-9 and also Exhibit 4-18, page 4-28) shows the various National Grid systems that might be required to interact with ELSSI in the future. This was developed to ensure that all potential systems were identified. The flows of information depicted are not meant to represent any specific level of automation or otherwise. The flow of information from one system to another could be automated directly through a point to point interface, it could be automatically implemented through a messaging system based on a service oriented architecture or it could be a manual interface based on a person copying data from one system and entering it into another. The important thing here was to identify and document the systems together with examples of potential data flows Data Ingest Dataflow Diagram The data ingest dataflow (see Exhibit 4-20, page 4-35) shows the systems involved in collecting IED data. The high level data flows were split into ingest and usage to make the diagrams more easily readable and also to logically separate the requirements for gathering and processing data. These two dataflow diagrams also break out the ELSSI component represented in the context diagram and present more National Grid 4-40 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

211 Attachment COMM 2-8(b) Page 75 of 160 detailed architectural components of ELSSI as proposed by KEMA. This diagram depicts the collection of operational and non-operational data from IEDs, and data from meters and storing it in an ELSSI database and a content server. The content server is to catalog and store non-operational data that is to be kept in its native file format for use by specialized engineering tools Data Usage Dataflow Diagram The data usage dataflow (see Exhibit 4-21, page 4-36) shows the systems involved in using IED data. As with the data ingest diagram the ELSSI component represented in the context diagram is broken out into more detailed components Fact Model The initial fact model (see Section ) shows example terms that KEMA believes should have clear, unambiguous and consistent definitions within National Grid. The initial fact model illustrates the interaction of these terms. The definition of standard definitions for terms within National Grid will be key Meta-data. By taking care to standardize definitions of the terms and the data to be shared it will improve communications within the business and also between the business side and IT, thus providing a bridge between business analysis and system design UML Sequence Diagrams The UML sequence diagrams (see Section 4.2.8) describe a structured series of representations of potential future state behavior as a series of sequential steps over time. They have been used to depict workflow, message passing and how elements in general cooperate over time to achieve various results. These scenarios were not meant to be comprehensive but were developed as a replacement for the use cases originally planned and were built around the same business areas. The objective was to take a few business threads and develop UML use cases to show how the proposed ELSSI solution supports different business scenarios. It was felt that this could be better represented by using sequence diagrams that not only showcased each of the ELSSI components but did so in a way that allowed National Grid to see how these components would interact with other components. All of the diagrams reference other existing National Grid systems except for the dynamic ratings example with includes a ratings calculator component UML Use Case Diagram A use case diagram (see Section ) was produced for the Meter Data Services area. Since MDS also collect data from substation meters and other revenue (and non-revenue) meters and since meters are capable of collecting other information such as power quality, it is appropriate to develop a use case to show how these pieces fit together especially when the possibility of a single meter repository is National Grid 4-41 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

212 Attachment COMM 2-8(b) Page 76 of 160 considered for interval energy data. The use case describes how users will interact with the various components to access interval meter data or other data collected from meters Fact Model Introduction and Definitions A "fact model" is a set of statements (and a diagrammatic depiction thereof) that describe a business operation using structured business vocabulary to express rules in a consistent fashion. A fact model focuses on assertions (facts) involving core concepts (terms) of the business. Concepts are represented by terms and facts connect the terms and provide a business rule base for how the terms interact. Terms like outage, customer and transmission should have a precise, unambiguous definition in the business. For example, outage might be defined as: "An interruption of a circuit, either by a disturbance or planned or unplanned maintenance operation, which causes the circuit to become unavailable for normal power, flow", assuming of course that interruption, circuit, disturbance, planned, unplanned etc. are all unambiguously defined as well. Facts are given by simple, declarative sentences that connect the terms with a verb or verb phrase, such as, "Asset causes outage". A fact model should (eventually) serve as an initial blueprint for a data model, but its primary purpose is to capture knowledge about the business in a structured form. The business concept for which each word (term) or (fact) phrase should never be taken for granted, especially for terms that have self evident meanings (and which may vary in different geographical districts or business areas), for instance transmission Focus of Fact Model Since a fact model structures basic knowledge about business practices from a business perspective it is a crucial starting point for structuring knowledge about the business based on a standard vocabulary. This ensures that you can communicate effectively and exploit this standardized vocabulary to express other types of requirements, especially rules and to communicate about those effectively too. ELSSI has the potential to act as a single point of reference for many types of data could ultimately involve providing greater access of more data to more people. If the terms are not carefully defined, the data could potentially be inadvertently misused by those who do not understand it. The rules are simple: if you make data available within the business, you must make it clear (unambiguous) what the data represents. Developing a comprehensive fact model is a logical next step that National Grid should consider taking. The initial fact model presented here uses terms that KEMA believes should have clear, unambiguous and consistent definitions within National Grid. The initial fact model illustrates the interaction of these terms. Rules are another level of detail and can be developed from facts and essentially add the sense of the words must or must not to the terms and facts, as in, "Assets with a critical number less than 400 must not National Grid 4-42 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

213 Attachment COMM 2-8(b) Page 77 of 160 be worked on unless there is an overriding reason such as work on connected assets." For instance, if a critical (e.g. congested) transmission line is to be taken out of service for work to be performed, it may make more sense to use the opportunity to work on other assets to avoid the need for another future planned (or worse, unplanned) outage to do the work Differences to Data Models How does a fact model differ from a logical data model? The primary audience for fact models and data models are different. A fact model is part of the business model an enterprise or project should develop, whereas a logical data model is part of the system model it should develop. A fact model provides a business-based starting point for future development of a logical data model or database design. The primary audience for the fact model consists of business-side workers and managers (and business analysts), whereas the primary audience for the logical data model consists of system and database designers. It is very important to keep these distinctions in mind, especially since fact models use boxes and box-to-box connections that appear similar to logical data models. There are differences in perspective and purpose between fact models and logical data models. In contrast to a fact model, a logical data model generally places emphasis on the following areas. In a fact model, a box represents a term and the business concept for which it stands. In a logical data model, a box generally represents a collection of attributes or fields that are structured to retain the appropriate data for storage and manipulation by applications. Introducing features appropriate for database design in the given technical environment: normalization, denormalization, cardinality, optionality, associative entity types (bill of materials structure) to support many-to-many relationship types, mandatory fields and relationships, etc. Fact models do account for temporal changes. They simply identify what should be known about the basic business process at any given point in time. A logical data model, in contrast, must concern itself with the points-over-time aspect of data so the business (and its rules) can deal with the past (and the future). None of the previous bulleted items are appropriate for a fact model nor for the level of architecture being developed by this ELSSI Phase II project Benefits of a Fact Model The business benefits of a fact model can be substantial by enabling knowledge transfer and knowledge automation. While these are concepts that are not new and thus sometimes easily dismissed, they are National Grid 4-43 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

214 Attachment COMM 2-8(b) Page 78 of 160 concepts that represent necessary steps that many utilities need to take. With early retirement plans being used to bring in younger (and less experienced) staff and with age profiles that face a serious skills exodus in the next ten years, a way to capture valuable knowledge in a way that facilitates its use is essential. The business needs an organized approach that enables business professionals to drive the development of requirements. In traditional development approaches, much can be lost in the translation of upfront requirements to the actual running system. Writing a set of clear business rules improves communications between the business side and IT, and provides a bridge between business analysis and system design. The business rule approach helps to close the requirements gap between the business side and the IT side. This fact model illustrates areas where National Grid IT and business staff should consider focusing efforts to determine common definitions, driven by the business side, based on ELSSI work to date to help facilitate better requirements in the future. Note that sub-transmission is not related to any facts. This is due to Terms that Need Defining Example Terms to be Defined Distribution Sub Transmission Outage Inspection Transmission Region Trip Asset Operational Load Trip Substation Non-Operational Data Threshold Linear File Information Work IED Point Knowledge Maintenance Operations Planning Benefit Customer Planned Unplanned Circuit Line Feeder Disturbance Interruption Emergency Critical National Grid 4-44 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

215 Attachment COMM 2-8(b) Page 79 of Initial Fact Model Exhibit 4-27: Initial Fact Model 4.4 Data Requirements Matrix Update In previous data requirements exercises KEMA has performed three iterations of data requirements gathering. The first round gathers as much information as possible about current and desired data use, and how the currently used data is collected. This is what was performed at National Grid during Phase I and which was preceded by a questionnaire to help streamline the process The second round usually involves a broadening of requirements based on different departments seeing the types of data used or desired by other groups that were reported in the first round of meetings. Often this generates a broadening of requirements in some areas due to the I d like that as well factor. The final round of meetings is generally to review specifics, edit any inconsistencies, changes etc. Due to the short duration of this project KEMA worked with National Grid to update the data requirements from Phase I with minimal disruption to National Grid staff. The Phase I requirements were compiled from a review of interview notes, questionnaire responses, and information provided by National Grid 4-45 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

216 Attachment COMM 2-8(b) Page 80 of 160 National Grid about IEDs currently deployed in New England and New York. The follow up in Phase II was performed via and some telephone conversations. Since requirements will change over time as devices change and as National Grid changes the way it uses data from these devices, and also due to the I d like that as well factor, this Phase II exercise was intended to refine the current data requirements as well as desired data that were described in Phase I. An was sent to each business area (not each interview) from Phase I that contained the IED data identified for that area which was been highlighted in yellow in an attached Excel spreadsheet. Recipients were requested to look at the data requirements and respond with any changes. Specifically to help with prioritizing data requirements recipients were requested to ensure that they check the data use/collection frequency, size of data and the priority/importance of the data to their job. They were also asked to comment on the highlighted rows but also to take the opportunity to review types of data in other rows (from other interviews) in case there were data there that were also useful to them. A significant number of responses were received and which included alterations to collection frequencies, priorities of data items, clarifications of usage, and additional requirements based on other business area requirements from Phase I, though the data size column is mostly incomplete. These updates are reflected in the revised version of the Data Requirements Matrix. Examples of updates are The requirements for data points like Breaker open/close status should also include an additional requirement that the data must be time-tagged at the source to 1ms accuracy. This was provided in feedback to the but is not included in the data requirements matrix currently since the focus is to identify data sources and user groups in order to integrate existing or potential IED data with groups of users who would benefit from using it. The time stamp requirement is an upstream issue to be resolved separately but for future iterations of the data requirements matrix it could be added as an additional column so as to provide the ability to analyze variations in data requirements by time stamping sensitivity, assuming that different points have different time stamp requirements. It was also pointed out in some review comments that the same group was represented in different sets of rows and that the requirements appeared different. This is because the structure of the data requirements matrix reflects the interview process performed in Phase I and indicates that different requirements were discussed between meetings in New England and New York. The updated data requirements matrix could be sorted by user group and not by interview session so that all requirements received from similar functions are grouped together. A possible future update could include a column to represent a region or combination of regions and allow duplicate rows to be removed but still allow for differences to exist between regions to support local requirements if desired i.e. it could indicate NE, NY, or BOTH. A new user group that was added was Transmission Line Engineering (TLE) as a result of National Grid feedback. By using several sources of information (such as Wavewin and TCM) to pin point the potential National Grid 4-46 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

217 Attachment COMM 2-8(b) Page 81 of 160 location, TLE can focus patrol areas and potentially cut down on operating expenses. By using protection/relay information in conjunction with GIS trace functionality and possibly lightning strike data (as applicable) TLE are able to create target information to minimize and focus circuit patrols. Distribution Engineering and Planning highlighted many data items of interest raised from other user groups/interviews that would also be applicable for them. These changes are reflected in the updated Data Requirements Matrix. Power Control currently receives data (KV, MW, MVAR) in EMS from some stations that come from power donuts which communicate with the EMS RTU at the substation. One concern raised was the potential loss of telemetry data that is currently received from these devices but so long as the RTU remains the SCADA entrance device this will not be an issue. Even if the substation architecture were changed so that the RTU was acting as an IED for hardwired I/O and was providing data to a substation data concentrator, the data from these power donuts should still be accessible. National Grid is currently installing NxtPhase TESLA 2000s. A recent bid resulted in two vendors disturbance recorders being installed over the next six years. They are the NxtPhase TESLA 3000 and the Qualitrol IDM disturbance recorders. However, National Grid will be upgrading most of the existing Mehta Digital Fault Recorders (DFRs) installed in New England and New York. It is also expected that approximately 18 of the 23 installed Mehta DFRs will be updated with Mehta provided upgrades. The remainder will be replaced with NxtPhase or Qualitrol units. The advantage of the NxtPhase and Qualitrol units is that they also have slow speed (Swing) recording capabilities. National Grid is working with the New York ISO and Transmission Planning to develop triggering criteria for these events. 4.5 Substation Architecture Phase I SA Vision Following is a list of ELSSI SA objectives that were identified during Phase I of the project. The proposed ELSSI SA architecture must: Enable National Grid to fully exploit the wealth of information contained in existing and future substation IEDs Enable National Grid personnel to access operational and non-operational data from substation IEDs without having to travel to the substation Promote data sharing and minimize the existence of data silos and duplication of data Enable National Grid to accomplish high payback application functions by providing access to IED data required to support these functions. National Grid 4-47 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

218 Attachment COMM 2-8(b) Page 82 of Substation Classes Two classes of substations were considered by KEMA for National Grid: transmission substations and distribution substations. For the purposes of this study, transmission substations are those stations that include switching and protection facilities for 220kV and 345kV equipment that can have considerable impact on the integrity of the bulk power grid. Distribution substations are primarily intended to transform voltage to distribution primary level and serve as the source of supply for distribution feeders. For the most part, the operation or failure of the power apparatus at distribution substations will have little or no impact on the integrity of the bulk power system. The design characteristics of the ELSSI SA facilities may differ somewhat for transmission substations and distribution substations due to differences in the physical, functional, and performance requirements at these two classes of substations. Some of the items where differences may exist include: Physical size transmission substations may be physically much larger than distribution substations, and may include multiple control buildings. It may be quite difficult to quickly assess the state of substation at a glance in a transmission substation. So, a mechanism for viewing the status of all substation equipment from a single location may be needed more in a transmission substation. DFR/SOE requirements DFR/SER facilities are usually considered a necessity at transmission substations due to the need to coordinate protection and control facilities at multiple, widely dispersed locations. DFR/SOE facilities are less useful at distribution substations which are radial in nature and do not require a great deal of coordination between substations. Sequence control requirements sequence control requirements, such as the handling of bus failover and capacitor switching logic, can be considerably more complicated at distribution substations than at transmission stations. Therefore, dedicated programmable controller facilities may be required at distribution substations, and may not be required for handling less complicated sequence control functions at transmission. Cyber security requirements Cyber security requirements may be different for transmission substations and distribution substations because the NERC CIP requirements only apply to transmission facilities. However, some electric utilities have elected to apply similar cyber security measures at distribution substations as a prudent course of action against cyber attacks at distribution substations. Availability of high-speed communication facilities Major transmission substation are more likely to be equipped with high bandwidth communication facilities than National Grid 4-48 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

219 Attachment COMM 2-8(b) Page 83 of 160 distribution substations. Communication facilities at some distribution substations may be limited to dialup telephone, which will limit the possible ELSSI functionality. In the discussion of specific ELSSI/SA design issues that follows, the impact of the substation class differences is considered and architectural changes needed to address the differences are identified. One overriding recommendation applies to the ELSSI/SA architecture applied at each substation. Despite having some differences, the ELSSI SA architecture should be consistent across all substations (e.g., same family of substation processors) for ease of operation and maintenance Specific Design Issues This section covers the following specific design issues pertaining to the ELSSI/SA architecture: Handling of Non-operational data Interface to EMS SCADA (handling of operational data) Handling of Non-Operational Data National Grid is one of the industry leaders in implementing facilities for automatically retrieving one form of non-operational data - outage event data (DFR and SOE reports) - from dedicated digital fault recorders in transmission substations via its Wavewin system. This is an excellent starting point for supporting the ELSSI Vision. KEMA recommends several modifications and additions to the existing Wavewin system that build upon this existing design to more completely support the vision. The basic Wavewin concept is well suited to handling applications other than DFR/SER data files, such as the such as the acquisition of data used for condition based inspections and maintenance of substation assets. Non-operational data acquired via the ELSSI SA facilities should also include equipment condition data from existing Schweitzer (SEL) relays (circuit breaker monitor, substation battery monitor, etc.), Beckwith LTC controls (drag hand data, tap position statistics/histogram), Hydran DGA monitors, and other existing substation IEDs. These are high payback items that will help justify the expense of adding this new capability. Techniques for acquiring non-operational data from substation IEDs are different than the traditional SCADA techniques used to acquire operational data items from the IEDs. While industry standard communication protocols, such as DNP3 and Modbus, can be used to acquire the operational data items, manufacturer-specific ( proprietary ) languages are needed in most cases for the non-operational data. National Grid 4-49 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

220 Attachment COMM 2-8(b) Page 84 of 160 Examples of manufacturer-specific languages include acselerator for SEL relays, Tap Talk for Beckwith tap changer and regulator controls, and Hydran Host for GE s dissolved gas monitor. There are two main approaches for acquiring non-operational data and delivering this data to a data warehouse on the corporate network: 1) Substation Data Concentrator Pass through 2) Substation Data Concentrator as Enterprise Gateway These two approaches are described in the following sections, followed by a recommendation for National Grid Pass Through Approach This approach utilizes the pass through capabilities of the substation data concentrator in which the data concentrator appears to be a simple port switch. This allows the user to make what appears to be a direct RS232 connection to the IED in question. Once the connection is established to the IED in question, the appropriate manufacturer specific software is executed on the network server to acquire the required non-operational data directly from the IED. The pass through approach to obtaining non-operational data operates as follows: 1) The network data warehouse server establishes a pass through connection to IED in question via the substation data concentrator 2) The server interacts with the IED and downloads the required data as though server was directly connected to the IED in the substation 3) Downloaded data is then stored in the data warehouse where it can be accessed by authorized users Exhibit 4-28 depicts this approach. National Grid 4-50 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

221 Attachment COMM 2-8(b) Page 85 of 160 Exhibit 4-28: Pass Through Approach Enterprise Gateway Approach With this approach, the direct interface to the individual substation IEDs is handled by the substation data concentrator. Each substation data concentrator includes copies of the manufacturer specific software for communicating with the IEDs connected to the SDC in question. Non-operational data is acquired by the SDC, which stores ( buffers ) the data and notifies the corporate network facilities that new data is available for retrieval. The enterprise data warehouse facility then retrieves the data from the SDC using a network protocol such as FTP. The basic approach for acquiring non-operational data from substation IEDs and delivering this information to the enterprise is as follows: 1) IED manufacturer proprietary software loaded onto substation data concentrator (e.g., Wavewin functionality) 2) Data concentrator communicates directly with the IEDs to acquire non-operational data files using IED manufacturer protocol 3) Data concentrator buffers data files as a local data warehouse 4) Data concentrator notifies ELSSI enterprise facilities that data is available for upload National Grid 4-51 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

222 Attachment COMM 2-8(b) Page 86 of 160 5) Stored data file pushed or pulled into Enterprise Level Data Warehouse Exhibit 4-29 depicts the basic approach: Exhibit 4-29: Enterprise Level Data Warehouse WAN Enterprise Level Data Warehouse Firewall/Router FTP TCP/IP Substation Data Concentrator (Orion) IED manufacturerspecific protocol IED IED IED Recommendation KEMA recommends the enterprise gateway solution (Approach 2) for the following reasons Access control is considerably simpler with the enterprise gateway approach. The network server only needs to communicate with one device per substation, versus having to maintain access control rules for many individual IEDs which would be much more difficult to manage. The SDC has much more storage capability than the individual IEDs for buffering the non-operational data files prior to transmitting this data to the network server. Therefore, the risk of non-operational data being overwritten before it is stored away is much lower with the SDC. The Orion data concentrators being deployed by National Grid at Ward Hill (New England) and Clay (New York) substations appear to have sufficient capacity to perform the functions that are currently being performed by the Wavewin PC at each substation. The Orion s can be equipped with a fair amount of local storage capability (up to 24 Mbytes), which should be sufficient for buffering data files acquired National Grid 4-52 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

223 Attachment COMM 2-8(b) Page 87 of 160 from substation IEDs. The Orion data concentrators include support for network protocols such as FTP and HTTP which can be used for transferring data to the corporate network. By incorporating the Wavewin functionality in the Orion data concentrator, the Wavewin PC at each substation could be eliminated, thereby reducing the overall cost and complexity. To replicate the existing Wavewin functionality, the DFR/SER protocols developed by National Grid and SoftStuf, Inc for Wavewin would have to be added to the Orion, along with the proprietary protocols used by other IEDs (such as equipment condition monitors) that may be added to the system. KEMA believes that Novatech is capable of developing the necessary interfaces due to Novatech s excellent industry reputation for interfacing to a wide range of IEDs from many suppliers. In addition, KEMA is aware that Novatech may partner with SoftStuf to incorporate the Wavewin functionality in the Orion data concentrator. Should this partnering arrangement occur, moving the Wavewin functionality from the PC to the data concentrator and eliminating the PC from the substation configuration is recommended Interface to EMS SCADA The substation automation facilities must be capable of supporting the SCADA requirements for the National Grid EMS facilities. These requirements include: The acquisition of real-time operational data from substation IEDs and hardwired measurements The execution of remote control commands initiated by the EMS, such as opening/closing of circuit breakers and raising/lowering tap settings on LTCs and voltage regulators There are two main approaches for accomplishing the EMS SCADA interface functions within the ELSSI/SA architecture: Interface to Substation RTU or Equivalent Data Acquisition Device Interface to Substation Data Concentrator Approach 1: Interface to Substation RTU With this approach, the EMS SCADA interface would continue to operate as it does today at substations that are equipped with SCADA facilities. At substations that are already equipped with an RTU, the EMS SCADA communication facility would remain directly connected to the existing RTU. At substations that do not have existing SCADA facilities, the SCADA EMS communication facilities would connect directly to new data acquisition facilities (RTU or programmable logic controller (PLC)) that are separate National Grid 4-53 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

224 Attachment COMM 2-8(b) Page 88 of 160 from, but connected to, the substation data concentrator. The RTU would continue to handle all remote control operations initiated by the National Grid Control Center. Exhibit 4-30 depicts the ELSSI SA configuration for Approach 1. Exhibit 4-30: ELSSI SA Configuration for Approach 1 Enterprise Level Data Mart WAN Serial DNP3 Operational Data And Controls Modem To SCADA EMS Orion Substation Data Concentrator DNP3, Modbus, or ASCII Firewall/Router Serial DNP3 IED IED Remote Terminal Unit (or Equivalent) Hardwired I/O Approach 2: Interface to Substation Data Concentrator With this approach, the substation data concentrator would handle the EMS SCADA interface and the existing RTU becomes an IED talking upstream to the data concentrator. Operational data required by the EMS would be acquired directly from the substation data concentrator, which in turn would receive operational data from substation IEDs and hardwired input/output (I/O) facilities. EMS control commands would be passed by the SDC to the hardwired I/O facilities, which in turn would open/close appropriate contact outputs to accomplish the desired control function. Exhibit 4-31 depicts Approach 2. National Grid 4-54 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

225 Attachment COMM 2-8(b) Page 89 of 160 Exhibit 4-31: ELSSI SA Configuration for Approach 2 Enterprise Level Data Mart WAN Serial DNP3 Operational Data And Controls To SCADA EMS Firewall/Router Modem Orion Substation Data Concentrator DNP3, Modbus, or ASCII Serial DNP3 IED IED Remote Terminal Unit (or Equivalent) Hardwired I/O Recommendation KEMA recommends Approach 2, Use Substation Data Concentrator as the EMS SCADA interface. This approach is the most common configuration used by electric utilities that have implemented operational and non-operational data interfaces similar to that being considered by National Grid. It is considerably less complex (and therefore less costly to implement and maintain) than Approach 1, which requires additional database and mapping information to associate data stored in the data concentrator, RTU (or equivalent device) and EMS. KEMA understands National Grid s concern that the recommended approach may create some initial confusion among the various technical groups that operate and maintain the SCADA and protection equipment in the substations. However, KEMA believes that an acceptable maintenance and support arrangement can be worked out that is mutually agreeable to all groups, as it has been many other electric utility companies. National Grid 4-55 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

226 Attachment COMM 2-8(b) Page 90 of 160 As part of this investigation, KEMA considered a backup approach at a pilot substation that would permit the existing RTU approach to replace the recommended data concentrator interface should the data concentrator fail. The possible backup solution is depicted below, to prove that the RTU could be replaced with the data concentrator, so both are ready to be in operation at the pilot substation. However, only the data concentrator is the entry point to the EMS SCADA. Should the data concentrator fail, the RTU can quickly be cut over, since it is already wired in a panel on standby. The use of data concentrators for EMS SCADA purposes is well proven within the industry, so there is no real need for this backup approach after the pilot substation. Exhibit 4-32: Possible Interim Approach Enterprise Level Data Mart WAN Serial DNP3 Operational Data And Controls Modem To SCADA EMS Orion Substation Data Concentrator DNP3, Modbus, or ASCII Firewall/Router Serial DNP3 In one pilot substation, as a backup plan, the RTU can be manually cutover to replace the data concentrator as the entry point to EMS SCADA should the data concentrator fail IED IED Remote Terminal Unit (or Equivalent) Hardwired I/O There is ready acceptance in NY for the data concentrator to be the entry point from the substation to the SCADA/EMS - no backup step is needed. It would be much easier for National Grid to avoid this backup step and, whenever a data concentrator is used in the substation, it is the entry point to the SCADA/EMS National Grid 4-56 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

227 Attachment COMM 2-8(b) Page 91 of 160 and the existing RTU becomes an IED talking upstream to the data concentrator, without any backup capability provisions Cyber Security See Section 3.2 and Appendix C for cyber security issues Substation Operator Interface For the purposes of this report, the substation operator interface includes all electromechanical, electronic, and computer-based devices that enable persons working in National Grid s substations to: View information about the status, loading, and performance of the substation power apparatus Execute Control actions for the substation equipment. Control actions include opening/closing circuit breakers and other switchgear, raising and lowering tap settings, turning automatic reclosing on and off, resetting lockouts, and other such actions. The existing substation operator interface used at most National Grid substations today consists primarily of electromechanical devices ( pistol grip style control switches and indicating lamps) with panel mounted meters for viewing load and voltage information. The operator interface at a few substations in New York and New England has been updated to allow viewing of substation data on computer monitors. Each substation that has been updated in this manner includes either one or two panel-mounted computer monitors with touchscreens to enable users to select the displayed data. At distribution substations that have been equipped with computer monitors, substation personnel are able to initiate control actions for substation devices through the computer screens. At transmission substations that have been equipped with computer screens, no control capability is included. That is, the computers at transmission substations are strictly view-only devices. All equipment control actions at transmission substations continue to be performed through the traditional electromechanical devices. The substation operator interface is a key architectural component/feature of the ELSSI/SA architecture. Recommendations for the handling of data viewing and device control are contained in the following separate sections. In each section, the possible approaches are discussed, followed by KEMA s recommendations Viewing Substation Data Two basic approaches were considered by KEMA for viewing substation data: National Grid 4-57 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

228 Attachment COMM 2-8(b) Page 92 of Approach 1: Use traditional electromechanical devices and panel meters 2. Approach 2: Use a PC-based human-machine interface (HMI) With Approach 1, National Grid would continue to use electromechanical devices (indicating lamps), panel meters (where available) and protective relay IEDs for viewing substation data. The primary advantages of this approach are that the electromechanical devices are very reliable and that substation personnel are familiar with this approach. Another advantage is that the electromechanical interface already exists at National Grid substations, so there is no additional expense to add new equipment. With Approach 2, a PC-based HMI would be added at each substation for viewing substation data. The PC s would include one or more panel-mounted PC monitors (screens), plus a keyboard and mouse (or touchscreen) for user interaction with the PC. The PC-based approach would enable to view substation data in a variety of tabular and graphical formats, including substation one-line (mimic) displays and simulated alarm annunciator displays. The PC would receive its data from the ELLSI/SA substation data concentrator, as depicted in Exhibit 4-33 below. The primary advantage of the PC-based HMI approach over the traditional approach is that it enables substation personnel to make a rapid ( at a glance ) assessment of the condition of all substation equipment from a single location during an emergency. This is especially important at transmission substations that are spread out over a large physical area and may in fact involve more than one substation control building. Other advantages of this approach include: Flexibility to make changes when needed through software only modifications Eliminates the need for some separate display units, like alarm annunciators. Alarm annunciator capability can be built into the displays and separate alarm annunciator units eliminated from future substation designs. The primary disadvantage of the PC-based approach is field personnel lack of familiarity with this approach. This is particularly troublesome at this time because so few substations have a PC-based HMI, so field personnel have to re-learn the interface with each visit to a substation that has a PC. Another disadvantage of using PC s is that that the physical equipment is more complex and hence less reliable than the traditional electromechanical approach. If the PC fails when needed, field personnel will either need to get the computer running again (hopefully through something simple like rebooting ) or fall back to using the protective relays and meters as a backup source of information. National Grid 4-58 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

229 Attachment COMM 2-8(b) Page 93 of 160 Exhibit 4-33: SA Architecture for PC-Based Local HMI Orion Substation Data Concentrator Local HMI Ethernet Visual Basic (VB) based displays containing real-time substation data from substation data concentrator DNP3, Modbus, or ASCII Serial DNP3 Remote Terminal Unit IED IED Hardwired I/O KEMA recommends Approach 2, the PC-based approach, for viewing substation data at both transmission and distribution substations. We believe that the ability to perform at a glance assessment of substation conditions is a significant advantage of this approach, particularly at larger transmission substations, where rapid assessment of substation conditions during an emergency is essential. Another advantage of this approach is that it will furnish a consistent look and feel between substations, even if IEDs are different. This approach will also enable National Grid to eliminate some traditional display devices, such alarm annunciators and panel meters, from its substations for a resultant cost savings. For example, with the PC, metered values can be acquired from relay IEDs and viewed as tabular or graphic displays from virtually any format, or viewed as mimic displays. Even if NATIONAL GRID elects not to add a PC-based user interface, panel meters aren t required because front panel displays on relays can be used instead. End user familiarization is a significant issue that must be considered with this recommendation. We believe this will become much less of a problem as the deployment of PC-based substation HMI facilities grows. During the implementation period, KEMA recommends that these facilities be rolled out on a district basis to maximize effectiveness capability to minimize the time period when field personnel will have to use significantly different systems depending on substation in question. National Grid 4-59 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

230 Attachment COMM 2-8(b) Page 94 of Local Control Facilities There are several key requirements pertaining to the control of substation equipment that must be satisfied (cannot be compromised): The substation operator interface must permit substation personnel to control the substation equipment from a safe distance (away from the equipment being controlled). This is a critical safety requirement! The substation operator interface must be highly reliable. Inability to perform a control action when needed due to failure of the control facilities is unacceptable. The substation operator interface must support National Grid tagging procedures. As with the operator interface for viewing substation data, KEMA has identified two main approaches for handling substation control requirements: 1. Approach 1: Use Traditional Electromechanical Controls 2. Approach 2: Use a PC-Based HMI for Executing Control Actions With Approach 1, all control actions would be performed in much the same way they are today via electromechanical control switches and/or pushbuttons. If a PC-based HMI is used (as is recommended by KEMA in the previous section), these switches/pushbuttons would be completely separate from the PC used to view substation data. The electromechanical controls may be part of the substation IED, rather than separate pistol grip style control switches, Exhibit 4-34 shows pushbuttons that are incorporated in the SEL 351S relay that could be used to support the equipment control requirements (similar control facilities are available from other IED vendors). Exhibit 4-34: Trip/Close Pushbuttons on SEL 351S Relay Trip/Close Pushbuttons National Grid 4-60 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

231 Attachment COMM 2-8(b) Page 95 of 160 Approach 2 would use the PC HMI for performing all control actions. With this approach software tagging would be used. The primary advantage of this approach is the ability to perform any control action for substation equipment from a single location. The major disadvantage of this approach is that the PC used to perform the control actions is considerably less reliable than the electromechanical approach due to the relative complexity of the PC versus a simple electromechanical switch. If the PC fails when a critical control action is needed, it is not acceptable to delay the control action until PC rebooting or, in the worst case, repairs and/or replacement take place. To avoid this single point of failure problem, a redundant backup PC would be required. For new construction only, using PCs for control and eliminating electromechanical control switches may result in savings in construction costs, but such cost savings would not apply at existing substations where electromechanical control switches already exist. KEMA recommends that Approach 1, hardwired electromechanical controls, be used for substation controls for both transmission and distribution substations. Presently, control is done through the PC (where available) for distribution substations but not for transmission substations. Hardwired controls are field proven and reliable and do not require a backup or redundant control facility which could actually reduce reliability due to the added complexity. Since hardwired electromechanical controls already exist at all substations, the cost of Approach 1 should be less than Approach 2, the PC based control solution. However, KEMA s recommendation to retain electromechanical controls is based primarily on maintaining safe and efficient control facilities and not on the potential for cost savings. A common approach across National Grid regarding control is important Summary of Local HMI Recommendations KEMA recommends using a PC-based local HMI for viewing data at all substations. The PC-based local HMI should provide the capability to view operational data (breaker status, bus voltage, equipment loading, etc.) as well as non-operational data for equipment condition monitoring purposes (contact wear indication, GE Hydran outputs, etc.). Local display of oscillographic current/voltage waveforms for fault analysis via the local HMI is not required. KEMA recommends using hardwired controls that are separate from the PC for all substation control actions Hardwired Inputs With the widespread deployment of substation IEDs and the growing trend toward obtaining real-time SCADA data from the IEDs, the quantities of SCADA input and output points that are directly-wired (in some cases via transducers) is greatly diminished. Even as IEDs are deployed in National Grid substations, the need for hardwired analog and digital inputs will continue to exist for power system apparatus that does not have an associated IED. Therefore, the ELSSI SA architecture must include a mechanism for acquiring data from and controlling power apparatus that is not equipped with an IED, National Grid 4-61 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

232 Attachment COMM 2-8(b) Page 96 of 160 such as motor operated disconnects substation capacitor banks, and station auxiliaries (smoke detectors, intrusion alarms, etc.). At substations where an RTU already exists, KEMA recommends that the existing RTU continue to be used for handling hardwired inputs. For the ELSSI/SA architecture, the RTU will interface with the substation data concentrator and will appear to be another IED feeding data to and receiving control commands from the data concentrator. For substations that are not currently equipped with an RTU and ultimately for all substations when existing RTUs require replacement substations, three approaches were considered by KEMA for handling hardwired I/O at National Grid substations: 1. Approach 1: Use Conventional Remote Terminal Unit (RTUs) 2. Approach 2: Use programmable logic controllers (PLCs) with add-on support for SOE time tagging 3. Approach 3: Use I/O modules supplied by substation data concentrator manufacturer that include support for SOE time tagging Ultimately, Approach 3 which uses I/O modules that plug into the substation data concentrator is the preferred approach because: This approach requires less space (does not require a separate cabinet) One level of database mapping is eliminated. There would only be one database in the substation the data concentrator database. National Grid will not have to map data variables from a separate device (RTU or PLC) to the data concentrator. Hence the cost to configure and maintain the system would be lower. This approach is not practical at this time because the Orion data concentrator (and data concentrators from other suppliers) do not support all of the necessary input requirements, such as handling of analog inputs and SOE status inputs. Development of these capabilities is currently underway, and should be available in the next 2 3 years. If no RTU exists, KEMA recommends that National Grid use PLC s for handling hardwired inputs. Today, many suppliers of LAN-based (distributed) SA systems have opted for more economical Programmable Logic Controllers (PLCs) for handling direct-wired inputs. Programmable logic controllers have proven to be an effective mechanism for handling this requirement. PLCs have analog and status input cards similar to those available on RTUs for direct I/O inputs. Past PLC limitations, such as the inability to support SOE time tagging and difficulties in synchronizing the PLCs to a time reference unit, National Grid 4-62 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

233 Attachment COMM 2-8(b) Page 97 of 160 have been resolved by third party products. For example, Monaghan Engineering offers SER modules and an External Time Reference (ETR) card for Modicon PLCs. Similarly, Control Technology International (CTI) offers high accuracy time stamp/sequence of events recorder facilities for Allen-Bradley PLCs. From KEMA s experience, this approach is supported by substation data concentrator vendors, including Qualitrol/Hathaway and Plan B Solutions (former Tasnet), as well as the Novatech Orion data concentrator that is currently used by National Grid Hardwired Outputs The ELSSI SA architecture must provide a reliable mechanism for handling control outputs initiated remotely (from the control center) via the SCADA EMS or, where applicable, initiated locally (from within the substation) via the ELSSI SA human machine interface (HMI). The required control outputs include open/close or on/off type control actions (e.g., opening and closing switches, enabling/disabling automatic reclosing) and raise/lower type control actions (e.g., change tap position on LTC or voltage regulator). There are two main approaches for handling control outputs from the ELSSI SA facilities: (1) Use RTU/PLC outputs and (2) Use IED outputs to execute control actions. Descriptions of these two approaches are contained in the following sections, along with a summary of the advantages and disadvantages of each approach and KEMA s recommendation for the ELSSI SA architecture. 1) Approach 1: Use RTU/PLC Outputs: This approach, which would be considered the traditional SCADA approach, utilizes control output contacts from the Remote Terminal Unit (RTU) or Programmable Logic Controller (PLC) to execute the required control actions. RTU/PLC output contacts are connected to the DC control circuitry of the power apparatus in question using appropriate interposing relays where necessary. This approach is reliable, well proven in the electric utility industry, and familiar to the technicians responsible for maintaining this equipment. If an RTU or PLC already exists in the substation for handling SCADA outputs, the RTU or PLC could be re-used in the ELSSI SA architecture with no additional cost since modifications to the interfaces with DC control circuitry would not be required. This approach also maintains separation between SCADA facilities and protective relay equipment, which is consistent with National Grid s existing maintenance organization and philosophy. 2) Approach 2: Use IED Outputs: This approach uses output contacts on substation IEDs for executing the required control actions. Output contacts on the protective relay IEDs would be used to control switches. LTC or voltage regulator IEDs (such as the Beckwith M2001 controllers) would be used to change LTC or voltage regulator tap settings. The required control actions would be sent from the substation data concentrator (SDC) to the National Grid 4-63 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

234 Attachment COMM 2-8(b) Page 98 of 160 IED in question over the substation LAN or serial connection using the communication protocol supported by the IED (DNP3, Modbus, etc.). Use of the substation IEDs for handling SCADA control outputs is part of National Grid's LPS03 SA design standard. The primary advantage of this approach is that it eliminates the need for separate RTU/PLC output cards and associated interposing relays. For new construction, eliminating the output cards and interposing relays is a direct cost savings. For substations where SCADA controls via and RTU already exist, eliminating the control output cards means one less device to maintain. Some electric utilities have objected to relying totally on the IEDs both as the primary source of information and as a means of executing control actions. This is especially critical if traditional electromechanical (E/M) control switches and indicating lamps are eliminated in favor of PC-based user interface in the substation. In this case, loss of an IED would result loss of remote control and loss of local control from within the substation building. However, this issue should not be a major issue if NATIONAL GRID adopts KEMA s recommendation to retain existing E/M devices for local control purposes. Use of protective relay IEDs for executing SCADA EMS control actions may require to adjust its current maintenance policy and strategy, which is based on separation of protection and SCADA control capabilities. KEMA recommends Approach 2 (Use IED Outputs) for the ultimate ELSSI SA solution. This approach will provide cost savings for new construction because it eliminates the need for some RTU/PLC control output cards and interposing relays. For substations that are already equipped with SCADA RTUs with control output cards, KEMA recommends that the existing RTU control output facilities be retained because the construction cost savings would not apply. However, when RTU replacement is required, KEMA recommends making the transition to Approach 1 at that time. For substations with redundant protective relay IEDs, the capability to execute SCADA or SA control actions through either set of IEDs should be provided so that SCADA control capabilities are not lost during protection system maintenance or relay failures. KEMA recognizes that using protective relay IEDs for SCADA control purposes will require some alterations to current maintenance and support policies, which assume that these two functions are separate. However, we believe that an acceptable maintenance arrangement can be established that satisfies all parties involved. National Grid 4-64 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

235 Attachment COMM 2-8(b) Page 99 of 160 Note that the above recommendations to use IEDs for SCADA control purposes are based on the assumption that NATIONAL GRID will retain E/M control switches on the substation control panels. These switches provide local backup capability if the controlling IED is out of service for any reason. If NATIONAL GRID elects to remove the E/M devices at future substations and rely completely on a PCbased HMI, there is concern about total dependence of the relay IED for both monitoring and control. In that case, KEMA would recommend the addition of a separate control device (such as an RTU) that does not depend on the relay IEDs Handling of Sequence Control Logic National Grid has been using programmable logic controllers (PLCs) instead of hardwired control logic for handling some of its more complex control schemes. PLC control schemes that have been implemented by National Grid at distribution substations include switching and tagging logic, substation capacitor bank control, and bus failover control for substations with breaker and a half configuration. PLC s have also been used instead of hardwired control logic at some transmission substations. However, further investigation is needed to identify the control applications that have been implemented in PLCs at transmission substations. The use of PLCs for implementing control schemes is well proven in numerous industries. PLCs are used extensively in the process control industry for implementing mission-critical control schemes. The use of PLCs is also well established for monitoring and sequence control applications in electric utility substations. PLCs are a clear choice over hardwired control logic for implementing more complicated control schemes, like bus failover and capacitor switching, due to better flexibility (easier to change when needed) and lower cost. Some relay technicians at National Grid have questioned why hardwired logic isn't used in place of PLC ladder logic in transmission substations, because they feel much more comfortable working with hardwired logic than PLC logic. Further information is needed to explore these concerns in detail. One possible reason for the lack of comfort is lack of familiarity and experience with PLCs because PLC s are not used at all substations. The comfort level can be expected to grow as the use of PLCs becomes more widespread. Another possible reason is that PLCs have been used for very simple control functions that could easily have been implemented in hardwired logic, and there is concern that troubleshooting the PLC-based logic when problems occur will be more difficult than the corresponding hardwired logic. Lack of proper training, special tools, and documentation is another possible reason for lack of comfort. KEMA believes that PLCs should be used for handling sequence control logic, and strongly recommends this approach at least for more complicated control schemes. However, it is very important that the National Grid 4-65 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

236 Attachment COMM 2-8(b) Page 100 of 160 concerns of the field personnel be investigated further and addressed as needed before committing to widespread deployment of PLCs for all control applications. From an ELSSI/SA architectural viewpoint, there are several ways to implement the ladder logic control schemes: Approach 1: Continue to use programmable logic controllers (PLCs) Approach 2: Use ladder logic language programming provided in the substation RTU or data concentrator Approach 1, using PLCs, is recommended by KEMA. PLCs are easily capable of handling National Grid s control requirements using international standard ladder logic (IEC 61131). Ladder logic programming and debugging tools in PLCs are excellent and are far superior to corresponding facilities in RTUs and data concentrators, which generally rely on non-standard languages with relatively primitive tools. The use of automation in electric utility substations is growing rapidly at the present time, with a growing number of utilities replacing existing hardwired electromechanical control schemes with schemes that run on intelligent processors, such as PLCs. KEMA believes that the result of this trend will be improvements in the handling of sequence control applications by data concentrator vendors. Such changes are likely within the next 1 2 years due to the high demand for data concentrators at this time. Since vendor research and development activities are largely driven by customer demands, it would be in National Grid s best interest to apply commercial pressure on Novatech and other suppliers being considered to improve PLC capabilities as soon as possible. We believe that further improvements in RTU capabilities for processing sequence control are not as likely due to the migration of electric utilities away from RTU technology Communication Facilities and Protocols A limitation of the existing Wavewin system is its use of dialup telephone as the primary media for transmitting information from the substations to the Wavewin server on the corporate network. The use of dialup telephone to retrieve data from substation IEDs has been sufficient to demonstrate the benefits of Wavewin, and made the best use of limited communication facilities at each substation. However, dialup telephone represents a weak link of the current design due to the speed limitations and reliability of these connections. It is not practical to extend Wavewin use to a large number of substations without significant improvements in the communication facilities at each substation. A key to improving the scalability of this system is to implement a high speed, network connection that will, in essence, extend the corporate network to each substation. These communication facilities should provide sufficient National Grid 4-66 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

237 Attachment COMM 2-8(b) Page 101 of 160 bandwidth to support the ELSSI data transfer requirements and should allow the use of FTP, HTTP, and other network protocols for handling data transfers to the network server. Another limitation of the dialup communication facility is that data transfers from the substation take place once a day (at midnight), so that DFR/SER information is not available to protection engineers immediately following an event requiring review. DFR/SER reports can be requested on demand, however, data retrieval will still take a considerable amount of time due to the speed and reliability of the dialup telephone line. Once network facilities are added, the data files can be pushed to the corporate enterprise network as fault events occur so that data can be delivered to appropriate subject matter experts as soon as possible following the fault event. It is understood by KEMA that the priority for implementing high bandwidth network communication connections will be lower at Distribution substations and that communications may be limited to dialup at Distribution substations for the foreseeable future. National Grid 4-67 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

238 Attachment COMM 2-8(b) Page 102 of Implementation Considerations One issue that surfaced during the work in Phase 2, and which was addressed at a theoretical level in Phase 1, was that of rollout rate; i.e. how quickly should National Grid be deploying IEDs in the field?. KEMA offered several possible strategies for rates of deployment in Phase 1 but ultimately the decision must be driven by National Grid s corporate goals, priorities and available resources. 5.1 Opportunities and Benefits The benefits to National Grid in this report have been estimated on a per station basis assuming an allocation of the enterprise costs for IT and communications across the 345kV and 230kV stations pro rata. However, these costs are largely incurred up front and the NPV analysis reflects this. As the analysis did not consider individual stations, except Sandy Pond, and was thus undifferentiated in detail, it is likely that a more detailed analysis that established priorities among the classes of stations would produce at least an idea of what the optimum roll out strategy would be for a given level of available funding and resource. Against this must be set the consideration that intangible benefits obtained from streamlined work processes will be greatest if the penetration of ELSSI rollout is above some critical mass To Roll-out or Not to Roll-out, That is the Question Seeking the solution that best fits with National Grids goals is the right path to take and the rate at which National Grid decides to rollout IEDs needs to be aligned with the company s goals and long-term objectives. That could mean deploying more IEDs or retaining electro-mechanical devices so long as the reasons are clear. Currently National Grid installs IEDs in new substations and also on a one for one functional replacement basis. One issue is whether to keep implementing IEDs at the rate dictated by these types of work or whether to accelerate the deployment. The number of IEDs deployed will thus continue to increase and rollout rates were discussed in KEMA s Phase I Final Report. Since the decision has already been made to deploy IEDs the issue over benefits and cost becomes less about deciding whether or not to proceed based on the ratio of benefits to costs. The decision has already been made. The issue now becomes how to make best use of the data and standardized designs and processes to reduce costs and increase benefits. It is difficult for KEMA (or anyone else) to make a recommendation without being privy to internal goals etc. at National Grid. Ultimately National Grid needs to make a decision and KEMA s recommendation is to gather more information to help make the decision. To this end KEMA recommends an ELSSI pilot (Recommendation #6) that builds on the work of TISC as well as ELSSI Phase 1 and Phase 2. This pilot would enable National Grid to prove the concepts, establish the economic and reliability benefits through experience, and to thus plan a wider roll-out and more intensive IT and business process implementation with the benefit of experience. There will certainly be organizational change issues associated with the National Grid 5-1 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

239 Attachment COMM 2-8(b) Page 103 of 160 rollout of IEDs but then again National Grid is facing relatively high staff turnover in several years time due to the age of its staff. This also raises the issue of knowledge management and how to retain and utilize the in depth knowledge that has been built up by existing staff over many years. IED training for relay technicians must also not be overlooked. There is a risk of a scarcity of trained personnel as more LPS03 stations are implemented if training does not keep pace with implementation yet there is another issue here, in that there are so few integrated substations deployed that staff have difficulty retaining familiarization with the systems in those stations Discussions from Phase I A slow rate of deployment could result in the use many different IED types but with relatively few of each type being used and with multiple different substation interfaces (HMIs). As discussed in the report from Phase I, any rollout that lasts longer than the life of an IED (how ever long that might be) will incur compound rollout costs, not unlike paying compound interest on a credit card. Another discussion from Phase 1 was to look at how to back into a rollout rate rather than plan forwards by assuming a date when widespread IED deployment would be required based on a relative lack of support for electromechanical devices and more widespread acceptance, training and familiarization of IEDs. Another factor that was discussed in Phase I with respect to rollout rates was that different IED types may have different life spans, and refits may well be based on device type rather than be performed by station (for stations already fitted with IEDs). Also, individual or type based IED refits that do not involve other changes at the substation (such as wiring or control house changes) will be performed more quickly so once a significant number of substations (e.g. 30%) are fitted with IEDs a change in policy for refits is probable. This in turn will affect future replacement rates and may tend to shorten the refit cycle and it is expected that future programs are likely to be a combination of wholesale refits and selected device type refits. At some point the need to replace electromechanical devices will be so overwhelming that it becomes a massive undertaking, unless the rate of IED deployment is increased. Even without significantly increased IED deployment rates it may be a long time before this situation occurs due to the longevity of many electromechanical devices Implementation Considerations The following list of issues summarizes some of the issues that will affect the rate of IED deployment: Expected life of electromechanical devices Expected life of IEDs Cost of IEDs National Grid 5-2 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

240 Attachment COMM 2-8(b) Page 104 of 160 Decreased spending due to better data Acceptance of IEDs and Data Concentrators by field staff Communications cost to collect the data Maintenance, reliability and automation benefits Cost of integrating and sharing data Priority of CAPEX jobs Corporate goals This is by no means an exhaustive list but reflects several significant factors that will influence how and when IEDs are deployed at National Grid. Some of these have somewhat subjective components and others still require a degree of estimation or speculation, both of which lend themselves to argument and debate. Even with long IED life, acceptance by staff, low cost communications and benefits that accrue from their use, IEDs will not get deployed if there are other projects with more urgent priorities and if the objectives and benefits of deployment do not match up with corporate goals. To help understand the affects of each of these issues (except for the lifetime and CAPEX priority issues) KEMA recommends an ELSSI pilot (see Section 6.3) as one of the six major recommendations from Phase II 5.2 On-going and Planned Projects As KEMA progressed through this work, a number of projects and initiatives have been encountered that were important to consider as this work was completed. Some of these projects were uncovered during Phase I and were outlined in the Phase II proposal, while others were uncovered as the ELSSI architectures were completed. Each of these projects was considered in the analysis during the course of completing this work and was considered either directly or indirectly in the implementation roadmap. Some of the projects are information technology-based while others are more traditional engineering projects. Initiative Expansion of PI Expansion of SoftStuf s Wavewin Consideration PI is already installed in New England and being considered for expansion into New York. The ELSSI data mart would interface to PI and would benefit by interfacing to one operational data environment. National Grid has recognized much success with its limited installation of Wavewin. ELSSI would benefit from a more expansive implementation of Wavewin or like technology, particularly because it can read various forms of IED data by a variety of IED manufacturers. National Grid 5-3 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

241 Attachment COMM 2-8(b) Page 105 of 160 Initiative Integration of a new Meter Data Repository Data Warehouse Corporate Data Warehouse New Gas SCADA New Electric EMS TISC initiatives Reliability initiatives Consideration New York is seriously considering replacing ERS with a more modern data warehouse environment. The meter and performance data that is included in ERS was considered to be part of the ELSSI data warehouse for consideration of additional benefits. KEMA discussed the ELSSI scope with National Grid s IT to ensure that there will be no overlap between ELSSI and this finance-oriented data warehouse. ELSSI considered the potential for incorporating gas operational data from the new gas SCADA that is under serious consideration. ELSSI considered the potential for future EMS integration capabilities when National Grid decides to replace one or both of its EMS systems. This ELSSI project, along with Phase I closely reviewed and based KEMA s substation architecture, data mart architecture, communications requirements and cyber security requirements based upon the current LPS03 design for both New England and New York and the HMI. National Grid s evolving Reliability team has a need for a considerable amount of equipment performance data to augment its current and future initiatives designed to continually increase network reliability. 5.3 Technology and Platform Considerations Legacy Systems There are existing systems that collect the non-operational data from field devices: the Wavewin product for non-operational IED, DFR and SER files; a corporate file server for the power quality PQDIF and infrared images; and a stand alone PC for native SEL formatted files. There are existing systems that collect and archive the operational data points from the field devices in both New England: the enterprise EMS historian using the PI product, and in New York: the ERS system built on DB2 and PowerBuilder. There are existing substations that have data concentrators in them but are not connected to the corporate WAN and do not have dial-up connectivity. There are existing substations that have been designed to have data concentrators in them but since there are no communications to them, they just have their IEDs installed but no data concentrator yet. The current data collection and analysis infrastructure at National Grid is in the diagram below. It supports using current communication paths via modems to connect to existing substations that have National Grid 5-4 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

242 Attachment COMM 2-8(b) Page 106 of 160 IEDs installed and are connected to either a communication processor, a data concentrator, or directly to the local modem in the substation. KEMA recommends keeping the existing architecture components to support the substations that will have IEDs, DFRs, SERs, meters and even data concentrators in them and have not been connected to the corporate WAN but do have dial-up connectivity to avoid spending money where a solution already exists. Exhibit 5-1: Data Retrieval Network Diagram with Network Bridge Capabilities and Limitations The current architecture as depicted in the above diagram supports a subset of the IED vendors that have been installed at National Grid. The current architecture requires significant effort and testing to incorporate new device formats. The recommended ELSSI architecture isolates this functional requirement and places it on the data concentrator. This should shorten the time and effort required to support both old and new device types in the architecture. The current architecture is constrained by both the existing communications bandwidth and quality of service. KEMA is recommending extending the corporate WAN to these substations. The current architecture has a master station and a data bridge. The proposed architecture would keep these for dialup connections, but replace them with a data ingest server that has the following components: Data ingest National Grid 5-5 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

243 Attachment COMM 2-8(b) Page 107 of 160 Data classify Data loader Content loader These components were introduced to eliminate the current limitations around archival performance and reporting on the data quality of the information being managed. KEMA recommends taking a component based approach, so the data classify, data loader and content loader should be prototyped or piloted to work within the existing architecture. Until more data concentrators get installed and improved communications are supported to the substations, the data ingest component should not be prototyped or deployed. From looking at the existing architecture s set of limitations for data retrieval and data archiving and discussing them with the current software vendor, KEMA believes that the current software vendor is targeting the engineering analysis market and not the enterprise level data retrieval/data ingest market. These limitations are the main drivers for introducing the ELSSI data ingest server, the ELSSI data warehouse server, the ELSSI content management server and the ELSSI ad-hoc browser. Breaking up the architecture into discrete functional components allows National Grid to make a build vs. buy decision for each functional component and provides the opportunity to bring in enterprise best in class technologies and apply them to this problem domain. The ELSSI architecture also addresses the fact that multiple non-operational data retrieval environments exist at National Grid, the one above centered around Wavewin, the one centered around the SEL data analysis tools and the one used by the power quality engineers to store the study results. To support the growth and evolution of these non-operational data analysis tools, the ELSSI architecture has been designed to support the storage of these non-operational files in multiple formats (a native Documentum feature). The ELSSI ad-hoc browser tool will allow users to search and retrieve the non-operational files they need in the format they need for their analysis tool of choice. The current architectures at National Grid for non-operational data do not archive non-operational point data values but just the transient files. The recommended ELSSI architecture has a data warehouse component to support the storage of non-operational point information. The data ingest server has been allocated to support both FTP and DNPi to support the collection of point data using DNP over TCP/IP. KEMA does recommend using the pilot to prototype the types of analysis that could be done against multiple sources of information. To support this, we ve levied the requirement for the data classify, data loader and content loader to work with each of the existing non-operational file systems as well as the existing operational and maintenance systems. To prototype the analysis portion, putting all of this information into the data warehouse is sufficient. To prototype the virtual data warehouse functionality of the ELSSI architecture, KEMA would recommend not putting all of the interesting systems into the data National Grid 5-6 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

244 Attachment COMM 2-8(b) Page 108 of 160 warehouse, but rather configuring and using the federated database component to access the data natively from the source system. Both ERS and IDS would be natural systems to prototype the use of the federated database since these systems will be around for a long time and are mainly data retrieval and reporting systems. The current architectures at National Grid result in separate systems for operational and non-operational data analysis both in New England and New York. As New York progresses on it s new EMS initiative, National Grid should have a single system that contains all of the historical EMS operational time series point data. There will still be at least three separate systems for operational data and non-operational data. A key feature of the ELSSI architecture is to support a component based architecture and allows National Grid to bring in as much automation as fast as it deems necessary. So the ELSSI federated database and the ELSSI ad-hoc browser components should be piloted to see how well they provide the access to the operational EMS historians: PI and ERS and to the non-operational ELSSI data warehouse Migration Strategies To support the migration from the current architecture, the ELSSI architecture should first be populated with all of the non-operational historical information that is currently available at National Grid. This allows the ELSSI data classify, ELSSI content loader and ELSSI content management system components to be piloted. A populated content management system will support the end-user piloting of the ELSSI ad-hoc browser as well. Since KEMA is not proposing to decommission any of the current data retrieval architecture components, the ELSSI architecture supports a phased integration strategy with the current data retrieval architecture. KEMA anticipates needing to use the dial-up functionality and the device specific formatting functionality of the existing architecture components for the foreseeable future. If we go back to all of the potential business systems that could make use of the ELSSI infrastructure as discussed in Section , there will need to be detailed implementation schedule for new interfaces and a migration schedule for existing interfaces. Since the architecture is going to support both client/server type of integrations as well as web services based integrations, the integration architecture provides National Grid with complete flexibility of whether to migrate an existing client/server application to the ELSSI federated database information integrator component or to look for newer engineering applications that are based either.net or J2EE architectures and use the web services provided to integrate them with the ELSSI architecture Communications Requirements This section details the implementation considerations required of the communications infrastructure to support the migration to the ELSSI target architecture. Three areas of communications requirements are discussed: (1) intra-substation, (2) substation to Corporate WAN, and (3) the Corporate WAN itself. National Grid 5-7 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

245 Attachment COMM 2-8(b) Page 109 of Existing Situation The National Grid Communications network is made of three components: (1) the operational network, (2) the non-operational network, and (3) remote access network. Presently these three components utilize a variety of communications services and devices to support various types of equipment at substations in New England and New York. While fiber exists to some substations, almost all substations are still on low speed (9600 or 1200 bps) digital or analog communications. To support existing SCADA requirements, a variety of circuits are used: Multipoint analog circuits (private or leased) with either asynchronous or synchronous modems Multipoint, or point-to-point private or leased DDS circuits Point-to-point virtual circuits made of private or leased Frame Relay PVCs with integrated access devices. For non-operational and remote access traffic, many devices at substations (i.e. IEDs, DFRs, communications processors which provide a single access point to multiple IEDs) are accessed via autoanswer modems connected to the PSTN and/or cellular networks. In some stations a separate Modem and telephone network link is dedicated to one particular device, resulting in multiple modems and telephone links in a single substation. In others, devices are shared via a Code operated switch. These dial-up links can become a back door to the SCADA network, providing the ability to perform SCADA operations without direct access to the normal SCADA systems. National Grid also has in operation a corporate network consisting of a Lucent DMX OC-48 next-gen SONET backbone fiber optic network, used mainly for corporate data Ethernet traffic and a portion of traffic to the EMS. The SONET backbone node is approximately 50% utilized. At some substations, a SONET node also is co-located, allowing a cost-effective utilization of the SONET capacity Requirements to be Supported by Communications Network IED Penetration If National Grid follows recommended best practices, IEDs, particularly relays, reclosers and condition monitors, will be implemented in new and existing stations through funded programs at steady or fast rates. IED manufacturers and models are standardized. Standardization provides an operating and maintenance environment that is familiar to engineers, operators and technicians. Similarly, interfaces to information systems are standardized and easier to maintain, operating conditions can be standardized and schemes can be implemented easier. Finally, operational and non-operational data formats are standardized to simplify storage, maintenance and analysis requirements. National Grid 5-8 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

246 Attachment COMM 2-8(b) Page 110 of 160 Protective relay IEDs are capable of providing oscillographic data and maintenance-related information such as I2t at locations where DFRs are not available Communications Best Practices A best practice is continuous monitoring of devices in the field and bringing data back to a central repository where it can be viewed and analyzed on demand. On demand is based upon limited time delays in transferring data from IEDs to both EMS (operational) and the data warehouse (operational and non-operational). Best practice for substation communications is network based communications with a minimum data throughput rate of 56kbps. Routable protocols, such as DNPi (DNP3 over TCP/IP) are commonly used to handle substation data acquisition requirements. With substations there is one (or more) physical LAN, which is connected to the corporate WAN using a router (demarcation point). Protection from unauthorized access to the Corporate WAN and the station LAN is provided by configuring firewalls within NERC CIP Standards criteria Intra Substation Requirements The most significant industry trend noted by utilities is migration to IP based networks. This movement affects every aspect of communications networks, including digital microwave and WAN/LAN. The proliferation of services enabled by abundant available bandwidth further drives IP network deployment. For example, applications made available through broadband and substation automation quickly are becoming regarded as operating necessities. This trend facilitates the notion of network convergence. The goal for network convergence is to find a common platform allowing multiple networks to communicate and interact. At the level of an electrical substation, network convergence and automation can be accomplished through an Ethernet LAN providing linkages to high-bandwidth optical communications equipment, energy management equipment, and security systems that include video surveillance, access controllers and remote monitoring. Advantages of utilizing Ethernet architecture in substations is that it offers standardization, pushing the trend toward network convergence as discussed above. IEDs are typically IP based interfaces, which allow for interoperability with multiple vendors of IEDs and other equipment necessary in the substation. The intra-substation communications should also include the use of data concentrators (such as the Orion 5R) to more effectively groom the traffic for transport on the WAN. A couple of key standards for substations include: C , which provides communications equipment standards, and IEEE , considerations for substation equipment. To support local networking National Grid 5-9 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

247 Attachment COMM 2-8(b) Page 111 of 160 needs at a substation, vendors are providing hardened equipment, such as the Ruggedcom Ethernet switch. In addition, for security access, Ruggedcom is about to implement the 812.1x security standard x provides a layer 2 authentication mechanism, providing a link-layer authentication to network devices. Exhibit 4-31 on page 4-55 shows the target intra-substation communications architecture. Orion 5R data concentrators connect to Ethernet switches. Connected to the Ethernet switches via routers would be a WAN link for the non-operational data, and point-to-point links for operational data (over private or leased facilities), as discussed in Section , Substation to Corporate Requirements IED Characteristics Exhibit 5-2 summarizes for estimation purposes, typical IED characteristics for small, medium, and large IEDs, in support of operational and non-operational data. Exhibit 5-3 shows the derivation of the assumptions used to calculate data for operational and non-operational data. Analog point data samples are assumed to contain 16 bytes of data, and digital point data samples, 2 bytes. Operational bytes per scan for each type of device are developed by multiplying the data sample size times the number of analog or digital points in an assumed proportion of different classes of IEDs: small, medium, or large. Non-operational data sizing includes point data, plus assumed sizes of files for SOE and DFR, derived in Exhibit 5-2 as Event file and Oscillograph, respectively. These are intended to be representative broadgauge estimates, with different IEDs from different vendors having differing characteristics. Some IEDs support encryption, some don t, some provide flexible passwords while others have a fixed password. Exhibit 5-2 and Exhibit 5-3 were developed as an Excel spreadsheet, which can be modified based on specific vendor characteristics. IEDs generally support the industry standard protocol DNP3. For this estimate the DNP structure is assumed, however not defined down to specific object/variation levels. The columns in Exhibit 5-2 shows the number of analog and digital scan points estimated for different sizes of IEDs, and typical quantity of Bytes that are generated for all the scan points. Exhibit 5-2: Typical IED Characteristics # of Analog Points Operational # of Digital Points Bytes / Scan Analog Digital # of Analog Points # of Digital Points Non-Operational # of SOE # of DFR Bytes / Upload Small IED ,616 Medium IED ,824 Large IED ,093,616 National Grid 5-10 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

248 Attachment COMM 2-8(b) Page 112 of 160 Exhibit 5-3: Background of Sizing of Data Elements Sizing of individual Data elements Assumes 8 bytes of data and 8 bytes of quality, time, and other Data Element Size Bytes attributes - could be less Analog 16 Digital 2 Event Files Event File 65,536 # of Events 512 Oscillograph File 2,027,520 Bytes/Event 128 Including event message Kbytes/Event File Oscillograph Data Bytes/Sample 8 Samples/Cycle 16 Number of Variables 24 Number of Cycles 15 Number of Faults 40 Overhead Factor 1.1 Kbytes /file It is assumed that the entire SER buffer of 512 events gets uploaded It is assumed that the entire buffer of 10 Sec. Data need to be uploaded Substation LAN Considerations Administration of Network Addresses As with administration of any LAN, management of IP addressing is important. The first step is to determine organization responsibility for managing the addresses. In some utilities, operations is in charge of managing IP addressing for substations and other operations areas, while IT or telecommunications organizations manages addresses for data that runs strictly on the telecommunications backbone, or IT issues addresses where responsibility is shared. However, KEMA recommends that a central location be established for administration of IP addresses to ensure there will be no duplication or confusion as the system grows. The number of IEDs for typical Transmission and Distribution substations are estimated to be 60 IEDs in a transmission substation, and 25 IED devices in a typical distribution substation (these quantities are shown in Exhibit 5-4 and Exhibit 5-5). This suggests at least that number of device addresses per substation. The use of routers or data concentrators provide subnet addressing that will factor into the manageable number of addresses. National Grid 5-11 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

249 Attachment COMM 2-8(b) Page 113 of 160 Virtual LAN (VLAN) The Layer 2 switch envisioned for the ELSSI substation is capable of grouping subsets of its ports into virtual broadcast domains isolated from each other. These domains are commonly known as virtual LANs (VLANs). To establish a secure communications on shared facilities the use of VLANs are recommended. Some utilities use the VLAN to keep Corporate IT and substation data management information separate. This is in addition to using all the proven security tools available on IP devices: from the very basic configuration of system passwords, the use of IP permit filters, and login banners, all the way to more advanced tools such as RADIUS, TACACS+, Kerberos, SSH, SNMPv3, IDS. The concept of VLAN is similar to other concepts in the networking world where traffic is identified by the use of a tag or label. Identification is crucial for a Layer 2 device to be able to isolate ports and properly forward the traffic received. If any packet in a device is tightly coupled to an appropriate VLAN tag, it is always possible to reliably discriminate traffic into separate and independent domains. This is the basic premise of VLAN-based switching architectures. This is the limitation on the number of tags (4,904) for defining separate Virtual Local Area Networks (VLAN) under IEEE 802.1q. While vendors offer stacked VLAN implementations these are proprietary, giving up the benefits of interoperability between vendors that standards based implementations offer Consideration of WiFi for Intra-substation Communications The IEEE specification covers the standards for wireless local area network (WLAN) applications. The specifications specify an over-the-air (unlicensed) interface between a wireless client and an access point, as well as among wireless clients. The standard for wireless networks, is the equivalent to the standard for Ethernet for wired LANs. The standard addresses the physical (PHY) and media Access Control (MAC) layers and are intended to resolve compatibility issues between manufacturers of wireless LAN equipment. To economically justify a WLAN, there are 2 primary considerations: (1) increased productivity gains that can be quantified through increased mobility, and (2) savings in network infrastructure that can be realized with a WLAN over a wired connection. Productivity Gains The typical WLAN users would be those who gain productivity requirements through their mobility; i.e. when they are away from their workstation they could still continue to access the network for needed National Grid 5-12 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

250 Attachment COMM 2-8(b) Page 114 of 160 information to make more effective meetings, or work sessions with other team members. This could also include employees who move to other work locations on a regular basis and don t always have convenient access to a wired LAN connection. In many cases these users would have laptops that allow the mobility of computing. Savings in Network Facility and Infrastructure Costs The WLAN could best have application in setting up new users in temporary work environments or users where the wired infrastructure is at its capacity, or it is not feasible to install new wired facilities, such as burying cable across a substation yard, and would require a significant investment to add another user or set of users. In these cases, however, the WLAN connection would be viewed as a temporary solution, until a wired infrastructure could be implemented for the longer term. Thus the WLAN supplements a wired network, rather than replaces it. National Grid has installed WLAN at a substation to control regulators on feeders. This was done to avoid plowing in fiber to regulators to avoid costly trenching and digging underneath the substation within the fence. For the long term, the wired connection will still provide the best flexibility and throughput to support the enterprise wide applications currently in use in the National Grid environment. Several utilities utilize the WLAN to facilitate a convenient connection to the corporate network. Potential Applications KEMA has worked with utilities who have considered WLAN applications such as meter reading and wireless access in substations. However, use of WLANs at substations were considered as a desired rather than required. There could be effective productivity gains in that area, as field personnel could drive by the nearest substation to receive work order updates, project information, etc. rather than drive to the nearest office to access a PC. Productivity gains in this regard are from reduced driving time and vehicle wear and tear. WLANs are currently in use in power plant environments for isolated control area networks (CANs); these types of applications are well suited for WLANs. Other potential applications for WLAN that can be considered include: Communications path to operate reclosers in the field Backup communications path back to SCADA via multiple hops While a few utilities are known to use WiFi for these applications, it is not in widespread use; 900 Mhz MAS radios are also suited for this type of application, and have been more widely used. National Grid 5-13 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

251 Attachment COMM 2-8(b) Page 115 of 160 WLAN Considerations A big concern to address is security. Because the WLAN connection is available to anyone who is in range of an Access Point, the risk of a rogue user is greater and stricter measures must be implemented to ensure National Grid database systems and applications are protected. The designated security administrator of the WLAN will be required to shoulder this added responsibility. However, considering that such a user could plug into an operating Cat-5 jack, security measures similar to those used on the wired network can be applied. To consider wireless in a substation, National Grid should develop a comprehensive security policy to determine who has access rights, as well as developing a plan to ensure protection of sensitive data that would traverse the wireless media. Another consideration is to ensure WiFi equipment will be compatible with EMI emitted in the substations, especially the HV substations. A review of vendor capabilities to provide hardened equipment should be conducted Substation to Corporate Requirements Exhibit 5-4 and Exhibit 5-5 show the estimated data requirements from typical transmission and distribution substations, for both operational and non-operational data. These figures were developed with the intention of driving functional requirements that might differentiate between communications infrastructure requirements, and are not intended to be actual detailed estimates taking into account specific operational parameters. These figures indicate a representative number of IED scan points for small, medium, and large IEDs. For operational data from a typical transmission substation, the assumption is that IED scans would occur simultaneously every 2 seconds for digital points, and 6 seconds for analog points. Further, when the scans occur, the data must be delivered real-time to the Control center prior to the start of the next scan. Based on these parameters, Exhibit 5-4 shows the burst loading per second on a WAN network connection from the transmission substation to support the operational data is 1,787Bytes, or 14.3 Kilobits. The equivalent data rate per second is derived by assuming the 2 second digital data, and 6 second analog data is sent uniformly over the respective intervals between scans. Therefore, average throughput per second required would be equal to: ((analog data burst)/6 seconds + (digital data burst)/2 seconds). In reality, the scans from IEDs may be more staggered than simultaneous, thus the peak burst traffic would most likely be somewhat less than the 14.3 Kilobits per second. However, for planning purposes, the worst case burst scenario has been assumed. Added to the burst data are network communications protocol overheads including ACK and headers, which can vary up to an additional 20% depending on the type of communications circuit used (see Section ). National Grid 5-14 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

252 Attachment COMM 2-8(b) Page 116 of 160 Exhibit 5-4: Typical Transmission Substation Data Offered to WAN # of IEDs A point D Point Total Point Count Sec / Scan Operational Accepted Latency (Sec) Bytes / Scan Average Loading/ Sec A D A D Small IED , % ,624,640 2,625 Medium IED , % , Large IED , % ,936,160 27,915 Total SA ,840 1, ,219,040 31,198 Equivalent traffic in kilobits Equivalent traffic in megabits A point D Point SER DFR % of Interest Non-Operational Sec / Upload Scan Accepted Latency (sec) Bytes / Upload Burst Bytes / Sec Exhibit 5-5: Typical Distribution Substation Data Offered to WAN # of IEDs A point D Point Total Point Count Sec / Scan Operational Accepted Latency (Sec) Bytes / Scan Average Loading/ Sec A D A D Small IED % , Medium IED , % , Large IED , % ,468,080 1,745 Total SA , ,782,480 1,964 Equivalent traffic in kilobits Equivalent traffic in megabits A point D Point SER DFR % of Interest Non-Operational Sec / Upload Scan Accepted Latency (sec) Bytes / Upload Burst Bytes / Sec For non-operational data from a typical transmission substation, the total burst bytes per upload is 24.2 MB, or equivalent to 250 Kbps. This includes analog and digital data points, plus event files and DFR data generated from each IED in the substation (in the quantities shown in Exhibit 5-4 and Exhibit 5-5). This assumes that when a major event occurs, a certain percentage of IEDs will be impacted (the % of interest column in Exhibit 5-4 and Exhibit 5-5.) Scans for data occur every 300 seconds, and an accepted latency to send data in response to a data pull from users outside the substation is 600 seconds. For communications link sizing calculation, the lowest case of 300 seconds is used, which assumes that all scan data must be sent before the next scan cycle. Similarly, for typical distribution substations, a burst loading of approximately 7.2 Kb for operational data, and 16 Kbps for non-operational data is calculated, based on the given parameters and assumptions Communications Link Sizing from Substation to WAN The National Grid Document Functional Requirements For Integrated Access Devices Supporting Communications with Station Equipment, Draft Revision 01, May 11, 2005 indicates that a shared communications bandwidth of 56 Kbps is envisioned by National Grid to support communications with both SCADA and non-scada equipment at a station. SCADA communications traffic (async DNP) will National Grid 5-15 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

253 Attachment COMM 2-8(b) Page 117 of 160 consume 9.6 kbps of the shared bandwidth, and the remaining bandwidth (about 46 kbps) would be available for other IP based non-scada traffic to and from various non-scada devices at a station. The recommended ELSSI architecture identifies a need for a WAN connection for the non-operational data. Because operational data has different requirements from non-operational data, such as a more stringent latency requirement, and a higher need to be secure and protected, the operational data should be separate from the non-operational data as much as possible. This can be accomplished over a single physical connection through segmenting the data into separate logical networks from the substation and through the Corporate backbone network if it is routed over shared private facilities. This allows the most flexibility in protecting the operational data to the greatest extent possible from unauthorized intrusions, either unintentionally or maliciously. Transmission Substations Exhibit 5-6 summarizes bandwidth requirements for different ranges of IED penetration, based on the calculation model in Exhibit 5-4. From Exhibit 5-6, it can be seen that for operational data, a 56 kbps circuit (the nearest standard bandwidth telecommuncations circuit) would be adequate up to an IED penetration of 100%, with additional capacity available for growth, overhead, and unforeseen peak loads. Exhibit 5-6: Data Bandwidth Requirements for IED Penetration Ranges-Transmission Substations IED Penetration Peak Bandwidth for 2 second DI and 6 Second AI delivery of operational data 0-20% 2.9 kbps 50 Kbps 20-45% 6.5 kbps 100 Kbps 45-80% 11.4 kbps 200 Kbps % 14.3 kbps 250 Kbps Peak Bandwidth for 300 second upload scan of nonoperational data For non-operational data, Exhibit 5-6 shows that growth to multiple 56 Kbps equivalent circuits would be required (up to 5 x 56k circuits, or 4 x DS0s), as IED penetration increases. To support both the operational and non-operational data requirements, it may be beneficial to initially install a T1 link in the near term and segment bandwidth on the T1 for the operational and nonoperational data, partitioning the data by utilizing Virtual LANs (VLANs.). Generally, a T1 becomes cost effective when the requirements approach multiple 56 Kbps circuits. A single DS0 could be reserved for the operational data, and 4 x DS0s (256 Kbps) reserved for the non-operational data. Excess capacity within the T1 could also be used for other services such as video surveillance or other corporate requirements. National Grid 5-16 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

254 Attachment COMM 2-8(b) Page 118 of 160 Distribution Substations From Exhibit 5-7, long-term 100% IED penetration for the distribution substation suggests that the equivalent of a single 56 Kbps circuit for operational and non-operational traffic would be sufficient. However, if the data shared a single 56 kbps circuit, utilization for growth or unforeseen peak traffic would be limited. Separate 56 Kbps or DS0 circuits for the distribution substation operation and nonoperational data on a common T1 line may be more beneficial for the long term after IED penetration reaches 45% or greater. Exhibit 5-7 summarizes bandwidth requirements for different ranges of IED penetration based on the calculation model in Exhibit 5-5. From Exhibit 5-7 it can be seen that a shared 56 kbps line could be adequate for all levels of IED penetration for operational and non-operational data, if cost is a constraint in the near term and only leased facilities are available. However, by the time IED penetration exceeds 45%, spare bandwidth capacity approaches 50%, which limits the ability to plan for unforeseen or additional growth requirements. At this point, migration to a T1 line should be considered. Exhibit 5-7: Data Bandwidth Requirements for IED Penetration Ranges-Distribution Substations IED Penetration Peak Bandwidth for 2 Peak Bandwidth for second DI and 6 Second 600 second delivery AI delivery of of non-operational operational data data 0-20% 1.4 kbps 3 kbps 20-45% 3.7 kbps 6 kbps 45-80% 5.7 kbps 13 kbps % 7.2 kbps 16 kbps For a physical Implementation over private facilities, a single T1 could be provisioned to support both data types, segmenting the data types by implementing VLANs, the same as indicated for the transmission substations. If other services requiring bandwidth are deployed, such as video surveillance, this would further justify a need for a T1 link. However if private facilities are available and can be provisioned to the substation (a more preferred approach as discussed in ), cross connect at the DS0 level may require Digital Access Cross-connect System (DACS) equipment if it is already not in place. In the case of a shared 56 Kbps circuit in the near term, a data prioritization scheme would need to be implemented to ensure operational data had priority delivery over the non-operational data Network Based vs. Point-to-Point Circuits For transport, data can either be established as point-to-point circuits as done today by National Grid, or network based. The characteristics of each method are summarized in Exhibit 5-8 below. National Grid 5-17 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

255 Attachment COMM 2-8(b) Page 119 of 160 Exhibit 5-8: Comparison of Network Based vs. Point-to-Point Circuits Network/Routable Protocol Point-to-Point Allows Flexible Reconfiguration Potential Network Latency Efficient use of Network Capacity Plug & Play Capability NERC CIP Considerations Typically Serial Interface & Dedicated Circuit Minimal Overhead Circuits Groomed for Data Delivery Simpler Network/Less O&M Requirements Pt.-to-Pt. IP Circuits Still Considered Routable Protocol if there is access to IEDs May Not Be As Cost-effective As Routed Data Network In the spirit of Convergence of networks and technologies, the trend is away from point-to-point circuits (either dedicated or dial-up) for non-operational data and toward ubiquitous IP networks. It should be noted that point-to-point IP is routable and falls under NERC CIP, if there is access to IED s through this path. However, there are still advantages to utilizing point-to-point network for operational data, in terms of minimal latency, and implementing non-routable protocols to avoid NERC security measures required at the substation level for the EMS SCADA data path. Many utilities are implementing point to point over private facilities, utilizing the SONET backbone and establishing TDM over SONET with an overlayed DACS network. SONET is essentially a point-to-point network, however it can be configured to transport routable traffic over it (i.e. Gigabit Ethernet over SONET). For the ELSSI architecture utilizing a private network, the use of an IP network is recommended for nonoperational data, and direct point-to-point circuits for operational data. This is consistent with the practice of many utilities. Data Latency Point-to-point circuits typical have very low latency (on the order of a few milliseconds) because once the circuits are set up, there is no significant processing required to maintain the connection. There are minimal communications protocols required to be added as transport overhead (generally less than 10% added to data payload). In essence, the circuit is nailed in a static configuration until it is physically taken down. The tradeoff for the low latency is capacity that must be dedicated to the connection, which in essence increases bandwidth requirements and network costs to provide it. Operational data delivery requirements from the source to the EMS may range from typically 2 seconds for certain data (such as Automatic Generation Control data), and 5 to 10 seconds for all other data. The point-to-point circuits will ensure this delivery requirement can be met. National Grid 5-18 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

256 Attachment COMM 2-8(b) Page 120 of 160 Network based circuits, on the other hand, utilize routers through the network, which actually perform a store and forward routing function. This requires a greater amount of processing and communications protocol overhead (on the order of 25% added to data payload) for a data packet to be passed from an entry port to an exit port on the router. For a networked based operation to perform efficiently, the expected traffic through the network must be carefully forecasted for peak loads, to ensure router processors and circuit bandwidths can be matched to accommodate the load. Otherwise, packets will be dropped, resulting in retransmission requests (negative ACKs), which add to the aggregate network traffic and link congestion. In the well designed and planned network, a rule of thumb for latency through a router is on the order of ms. However, this figure can degrade quickly if congestion through a router develops and packets are placed in queue until it is their time to be processed and routed to the appropriate port on the router. Under these conditions router latencies can increase by a factor of 10 or more, resulting in significant end-to-end delay through the network: each router latency in the network that the data traverses is additive to the overall performance. The network-based IP connection can provide cost-effective communications connectivity and is the recommended approach for substation connection for non-operational data through the corporate WAN. With a carefully planned and designed network to support the communications traffic, end-to-end latency through the network for non-operational data should be within acceptable parameters. Long latencies of the respective networks for operational and non-operational data are not anticipated to be an issue in the ELSSI environment. Use of time stamps to send along with analog quantities should not be necessary. Most equipment in the typical utility environment today is not set up for the generation of time stamps, and there may be significant and unnecessary costs to provide it Communications Facilities from Substation to WAN Most of the National Grid substations currently are connected to the SCADA monitoring and control centers by leased DDS or frame relay circuits. These lines are planned for upgrade to 56 Kb frame relay circuits leased from Verizon. As indicated previously, the intent is to support both SCADA and non- SCADA data. With the ELSSI architecture, additional bandwidth will be required to support nonoperational data delivery. Also a consideration is the continued use of leased facilities to support operational data, especially from transmission substations. A best practice philosophy is to move to private facilities where it makes economic sense, however in the case of SCADA and critical infrastructure elements, the tendency of many utilities is to keep those on private facilities to ensure they are within the utility s control. They are less concerned with the use of leased facilities for corporate traffic. The KEMA Team understands that service reliability from Verizon is spotty, being rated good in Massachusetts, but not reliable in upstate New York. (However the big difference in the two systems is the EMS at the DNP level; they are connected in different ways, by serial in the Northeast, and IP in New York.) National Grid 5-19 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

257 Attachment COMM 2-8(b) Page 121 of 160 With the leased services, Verizon also provides a backup service where upon request from National Grid, the frame relay traffic can be rerouted from the primary EMS site to the secondary EMS site. However, to get this implemented could require several hours to implement after National Grid contacts Verizon to make the change. Options for Communications Upgrade from Substation to WAN The National Grid system consists of 210 transmission substations and 1100 Distribution substations. Of those, 20 sites are on the SONET backbone network in Buffalo, and the bulk power stations in New York are fairly close to fiber. However, the majority of the substations are several miles from a SONET node. A goal for National Grid should be to connect to the SONET backbone network, and utilize private facilities whenever it can be proven feasible. Advantages to Private Network build out include: Enables next generation native Ethernet from initial implementation Network under Company control, so add & changes are easily handled Performance monitoring of network is inherent. Trouble calls will be reduced and MTTR s will be reduced because you don t have to wait for vendor No high voltage protection (HVP) or facility issues Similarly, disadvantages to leased services include: Budgetary costs assume no facility issues; facility availability is typically verified once a circuit order is placed. Additional facilities or upgrade to HVP will impact circuit costs Adding services is dependent on vendor Personnel need to accompany vendor on site for trouble calls The following connectivity schemes are recommended in the order they should be analyzed for feasibility on a case-by-case basis for each substation: 1. Fiber to the Substation Installation of fiber ensures a secure facility that would have maximum reliability. This would include installation of a dark fiber from the substation to the nearest entry point to the SONET backbone. The nearest entry point may be a SONET node, or tributary that eventually connects to the SONET node. Based on the capacities indicated previously, a low capacity fiber transmission system capable of supporting up to 4 to 5 T1 lines appear to be sufficient. Generally the fiber may be ADSS type, National Grid 5-20 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

258 Attachment COMM 2-8(b) Page 122 of 160 installed overhead on poles and towers. In discussions with National Grid personnel, it appears that past similar projects have cost from $20K to $30K per mile. Terminal equipment on each end of the fiber run is approximately $10K per end. Additional interface cards, router, or cross-connect equipment may be required at the SONET node location. 2. Wireless/radio to the Substation This option establishes a broadband radio connection to the substation. There are 2 types: point-to-point and WiMax. a. Point-to-point This generally can be accomplished with low capacity radio terminals having 3-4 T1 capacity that operates in the unlicensed bands, such as the 5.8 Ghz band. The path must be designed to ensure a propagation reliability of better than %. Use of the unlicensed band is subject to potential interference. Alternatively, a licensed radio can be used (such as a narrowband microwave), but the costs may increase. Equipment hardware reliability can be increased by utilization of hot-standby radios. In this arrangement, the costs per end is approximately $30K, or $60K for a system. This cost does not include cost of towers; if a tower does not exist at a substation, one may be needed to obtain the necessary clearance above obstructions. The further the path distance of the radio hop, generally the higher the towers needed. When these costs are compare to (1) fiber to the substation, it can be seen that a breakeven mileage can be identified where radio will prove in over fiber. Generally, in this case the breakeven could be 5 miles or less. b. WiMax A new technology that is becoming in use in WiMax, which adheres to the IEEE specification. It can have a maximum range of 30 miles under the right conditions, and may be a more cost-effective solution when multiple substations within the signal radius can be captured by a single radio, thus enjoying a degree of economy of scale. Since this is a relatively new technology the KEMA Team is unaware of any systems that are in service in utilities, but many vendors such as Alvarion, are actively pursuing the market and its implementation may be just a matter of time. 3. Leased Facilities Leasing T1s or Frame relay facilities from a telecommunications provider is always available as an option for comparison, but with handing off traffic to the provider, a certain degree of control is lost. Disadvantages to leased lines are previously mentioned, and implications should also be considered from National Grid 5-21 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

259 Attachment COMM 2-8(b) Page 123 of 160 a security standpoint, as well as NERC guidelines. Leased facilities are established to the nearest SONET backbone access point from the substation. A typical access configuration from the substation to the corporate network is depicted in Exhibit 5-9. Exhibit 5-9: Typical Access Configuration from the Substation to the Corporate Network Non-Ops data Ops data Substation LAN SONET Fiber Extension DACS Device Non-Ops data Ops data Radio/wireless Non-Ops data Ops data SONET Node Lucent DMX Next-gen OC-48 SONET Ring Substation LAN Non-Ops data Ops data Leased Lines Substation LAN Other Considerations for Substation to WAN Communications Diversity & Criticality If separate networks are established for operational and non-operational data, the use of a priority scheme to separate the traffic is unnecessary, unless there is a need to prioritize traffic within each data type. If prioritization is required, then the ability is required to prioritize IP traffic based on source address, source port, destination address, and destination port. Remote Access and Changing of Device Configurations With the recommended ELSSI architecture, remote access will come into a substation via the nonoperational data path. Access will occur through various firewalls and security mechanisms that are implemented at the substation, using VPN. VPNs require a level of authentication of the end user (device) requesting connection, and protects the data in transport from eavesdropping or undetected modification. Proxy connections have also been used, but the general trend is to move away from the proxy connection. National Grid 5-22 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

260 Attachment COMM 2-8(b) Page 124 of 160 A proxy inspects the data (specifically commands) to verify that they are valid and passes only the valid commands. The proxy is complex to implement and is protocol version dependent Interim Substation to WAN Architecture A an intermediate step, there will be a period of time where legacy serial equipment will exist with newer IP and Ethernet based IEDs. To accommodate the serial equipment, a communications processor will provide the legacy interfaces as shown in Exhibit 5-10 (possibly interfacing with external network interface devices). As the RTUs are replaced with IEDs and migrated to the ELSSI Architecture, traffic through the communications processor would be gradually off-loaded. Exhibit 5-10: Interim Substation Architecture Channelized T1 56Kbps DDS TELCO DDS WAN Fiber, radio, or leased T1 facilities Communications Processor at RTU Site "1" RS-232 ASYNC Ethernet NGRID SONET WAN BACKBONE DNP RTU Serial Devices Ethernet Devices Corporate Communications Requirements This section discusses the impact of the substation traffic requirements on the corporate WAN, as traffic is gradually migrated to the corporate SONET backbone Non-Operational Traffic Corporate communications will include non-operational data and uses routable protocols through the corporate WAN. This includes a virtual data mart that would include mechanisms to use data in other systems such as PI, Wavewin, and any new repository for meter data. Data flows between systems are broadly categorized as either collecting or disseminating data. The virtual data mart is ideally suited for enterprise applications to support ELSSI, where data must be brought in from multiple sources. Remote access via the web can be established through the corporate firewall, and via a corporate connection to the substation. National Grid 5-23 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

261 Attachment COMM 2-8(b) Page 125 of Operational Traffic In the long term, operational data should be migrated as feasible to the National Grid SONET private backbone network. A portion of the SONET bandwidth should be segmented for operational data. Currently, National Grid primarily uses leased lines from Verizon for its operational traffic, and is currently upgrading the operational connections to 56 kbps frame relay access. By migrating to bandwidth on the SONET network, flexibility to expand past 56kbps will be possible as the model substation approaches increased IED penetration. The use of point-to-point circuits for the operational data is recommended. The use of point-to-point circuits will utilize non-routable protocols, and new security measures will not be required at the substation level for the EMS SCADA data path. Many utilities have their operational traffic remain virtually point to point over private facilities, utilizing the SONET and establishing TDM over SONET with an overlayed DACS network. This allows crossconnect down to the DS0 level, and establishes an operational network of channelized DS0s which can be protected through the SONET protection switching schemes. Utilization of these circuits on the SONET also makes use of SONET protection switching through careful provisioning of the SONET connections to the primary and backup EMS. It is recommended that National Grid review the cost-effectiveness of this approach in SONET architecture and the substation connectivity that is available. Exhibit 5-13 shows a typical configuration utilizing TDM over SONET with substation connectivity. DACS nodes with 56/64 kbps interface cards (or channel banks) could be located at strategic SONET nodes where 56/64 K circuits would be gathered and groomed for transmission over VT1.5 paths through the SONET network Corporate WAN Loading The actual aggregate traffic load on the Corporate SONET Backbone increases incrementally as substations penetrated with IEDs are connected. An assumption of an annual 3% penetration growth rate has been assumed by the KEMA. A model of this growth is summarized in Exhibit Exhibit 5-11: Basis of 3% Yearly IED Penetration 3% of Substations participating # of Substation s Analog Count Digital Count Total Count Operational Data MBits / Scan Burst Loading (MBits/Sec) Non-Operational Data Mbits / Upload Burst Loading Mbits / Sec Transmission 7 7,840 5,040 12, , Distribution 33 6,600 11,880 18, , Total 40 14,440 16,920 31, , National Grid 5-24 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

262 Attachment COMM 2-8(b) Page 126 of 160 Based on this assumption, yearly data additions to the WAN backbone are estimated to be approximately 340 Kbps for operational data, and 2.3 Mbps for non-operation data. Exhibit 5-13 contains the results of a spreadsheet analysis that shows the impact of adding 10%, 20%, and 35% levels of National Grid substation participation on the WAN. For operational data, additional load on the WAN ranges from 1.1 Mbps with 10% of National Grid s substations participating, to 3.8 Mbps with 35% participation. Nonoperational data added ranges from approximately 7 Mbps with 10% participation, to 24.5 Mbps with 35% participation. This assumes proportional percentages each of transmission and distribution substations are at the given participation levels. To plan the WAN for accommodating the load, a determination should be made of substation penetration objectives over a five-year planning horizon. Projects to add capacity to the SONET WAN should be established to accommodate the additional load in that time period. A reasonable target deployment percentage for planning purposes, based on a minimum 3% yearly growth rate is 20%. Based on that assumption, the target aggregate bandwidth to be accommodated on a network is approximately 2.2 Mbps for operational data, and 14 Mbps for non-operational data. Non-operational data could be accommodated within an OC-1 group of the SONET OC-48 capacity, configured to support IP WAN traffic. Operational data requirements justify two T1 equivalents groomed and separately segmented on the SONET backbone for the point-to-point traffic. If a leased frame relay network is used end-to-end in lieu of a SONET backbone for operational data, National Grid should establish service level agreements with Verizon to ensure performance parameters for latency and Quality of Service (QoS) are established that will meet the necessary operational data delivery requirements. National Grid 5-25 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

263 Attachment COMM 2-8(b) Page 127 of 160 Exhibit 5-12: Typical TDM over SONET Configuration Data Collection & Monitoring of Substation 3 RTU U Off-Premise Extension of PBX DS0 DS0 DS0 DS0 Channel Bank (multiplex) T1 Circuit ( pipe ) Special Purpose Data Circuit Gas Substation DACS Node (DS0 cross-connect) X X X X X X Data Collection & Monitoring of Substation RTU 3 U DS0 DS0 Channel Bank (multiplex) Electric Substation Protective Relay DS0 DS0 Mobile Radio System Electric/Gas Control Centers DS0 DS0 DS0 DS0 DS0 Channel Bank (multiplex) DACS Node (DS0 crossconnect) X X X X X X Transport Network T1 Circuit ( pipe ) Mobile Dispatch Home System PBX DS0 DS0 3 RTU U Electric Substation Special Purpose Data Circuit T1 Circuit ( pipe ) Protective Relay Off-Premise Extension of PBX Data Collection & Monitoring of Substation DS0 DS0 DS0 DS0 DS0 Channel Bank (multiplex) X X X X DACS Node (DS0 crossconnect) T1 Circuit ( pipe ) The scenarios described above assume that all IED scans are done simultaneously throughout the network. In reality, the IED scans will be spaced out such that 2 second digital, and 6 second analog scans will be staggered throughout the network, which in effect reduces the actual instantaneous burst traffic. National Grid 5-26 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

264 Attachment COMM 2-8(b) Page 128 of 160 Exhibit 5-13: Effects of Penetration of IED Substations on the Corporate WAN 10% of Substations participating # of Substations Operational Data MBits / Scan Burst Loading (MBits/Sec) Non-Operational Data Mbits / Upload Burst Loading Mbits / Sec Transmission , Distribution , Total , % of Substations # of Burst Loading Burst Loading participating MBits / Scan Mbits / Upload Substations (MBits/Sec) Mbits / Sec Transmission , Distribution , Total , % of Substations # of Burst Loading Burst Loading participating MBits / Scan Mbits / Upload Substations (MBits/Sec) Mbits / Sec Transmission , Distribution , Total , Enterprise Data included in Non-operational The non-operational data than would transport across the corporate environment would include: Non-point data collected by WAVEWIN (or eterra-analyst) and MV90 Revenue metering data Corp data access for example, timekeeping, or documents available on the corp. network, that can be accessed through an internet connection Corporate network access in the substations Video surveillance is a potential requirement that is separate from the traffic volumes previously discussed. However, the video could be superimposed on the substation traffic with the non-operational National Grid 5-27 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

265 Attachment COMM 2-8(b) Page 129 of 160 data as bandwidth permits. Current video surveillance uses (Clay, Edic, New Scotland) private corporate leased full T1 into substation. However, camera technology has progressed, and more resolution at lower bandwidth with built in compression can be achieved with newer technology Push vs. Pull of Data The Corporate architecture should have the ability to push operational data from substations to the monitoring center based on time, demand, or event triggered. The mechanism chosen will depend on functional requirements and internal policies. This would mainly be to report fault events or other occurrences that would generate an alarm condition. To support the ability to push data for alarm monitoring purposes, network latencies must be kept to an acceptable minimum. Since the pushing of data is operational, this supports the need for a separate network, for security and to ensure network performance will not be affected by non-essential traffic. Utilities that are doing this push of data recognize tremendous benefits. Two distinct advantages of pushing data promote ease of data access and the availability of data. Architectures that push data cut across the many different vendors proprietary IED formats to make it much easier to access important data. At the same time, data extracts can be performed automatically at rates acceptable to National Grid so that data are made available in the data mart to users much quicker and when it is needed the most. For non-operational data, the enterprise data mart can pull data from the local data repository on a substation. However to alleviate potential congestion issues in the network with multiple users trying to get data at the same time, a coordination process may need to be implemented to ensure that a data flow can be reasonably spread out more efficiently. The key is to make it as controlled as possible by laying out requirements appropriately, identifying what users require data and how often, and justifying the need with justifiable business requirements. Network impact can be avoided by making sure data is already in the data mart when users need it. The need to interrogate IEDs should be avoided; anticipate data that will be needed by particular users based on occurrence of certain events. See Section 3.2 for more discussion on pushing and pulling of data Summary of Recommendations for Communications Implementation Considerations Substation Communications 1) Establish Ethernet LAN and data concentrators in Substations to support migration to IEDs and the ELSSI architecture. 2) Utilize Virtual LAN connections to keep operational and non-operational data logically separate within the physical structure. National Grid 5-28 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

266 Attachment COMM 2-8(b) Page 130 of 160 3) Wireless LANs should be considered as a means to supplement other facilities, rather than replace them. The use of the unlicensed RF bands are subject to interference, and the openness of the signal over the air is cause for security concerns Substation to Corporate WAN Requirements 1) Because operational data has different requirements from non-operational data, operational data should be kept separate from the non-operational data as much as possible. Operational data should be established on point-to-point circuits without a need for a routable protocol. Non-operational data can be established on an IP network, which uses routable protocols. These networks to serve the specific data types will also ensure network latencies are within acceptable limits for the specific data types. 2) New Requirements for video surveillance could be accommodated on the non-operational data path as the bandwidth permits, depending on the video transmission technology used, and number of video signals to be supported. This will also require the use of a data priority scheme (based on packet type) that can be applied to video data packets to ensure the latency of the video is maintained within acceptable limits. 3) To support ELSSI requirements at a transmission substation, a T1 should be provisioned, and the operational and non-operational data partitioned through the use of VLANs. Spare capacity would also be available to support other services, such as video surveillance. This will also require the use of a data priority scheme (based on packet type) that can be applied to video data packets to ensure the latency of the video is maintained within acceptable limits. 4) To support ELSSI requirements at a distribution substation, data requirements from operational and non-operational data path up to a 45% IED penetration could be accommodated by the sharing of a leased 56 Kbps circuit. At 45% penetration, migration to a T1 line is recommended. However, to ensure flexibility for expansion, the preferred approach would be to initially implement a T1 link to the distribution substation. Excess capacity could be utilized for other services, such as video surveillance. To support these requirements, a single T1 could be provisioned, setting up VLANs for each operational and non-operational data, similar to the transmission substations. 5) National Grid should make use of its SONET backbone network to the greatest extent possible, and transport substation traffic (both operational and non-operational) over the private WAN back to the control centers. Private facilities, in lieu of leased facilities, should be built from the substation to the nearest SONET node as deemed feasible through a cost-benefit analysis per substation. Advantages to Private Network build out include: Enables next generation native Ethernet from initial implementation National Grid 5-29 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

267 Attachment COMM 2-8(b) Page 131 of 160 Network under Company control, so add & changes are easily handled Performance monitoring of network is inherent. Trouble calls will be reduced and MTTR s will be reduced because you don t have to wait for vendor. No High Voltage Protection (HVP) or facility issues Similarly, disadvantages to leased services include: Budgetary costs assume no facility issues; facility availability is typically verified once a circuit order is placed. Additional facilities or upgrade to HVP will impact circuit costs. Adding services is dependent on vendor Personnel need to accompany vendor on site for trouble calls The following substation to Corporate SONET backbone connectivity schemes are recommended in the order they should be analyzed on a case-by-cases for each substation to determine feasibility: a) Fiber to the Substation. Installation of fiber to ensure a secure facility that would have maximum reliability. This would include installation of a dark fiber plus low capacity fiber transmission equipment from the substation to the nearest entry point to the SONET backbone. The nearest entry point may be a SONET node, or tributary that eventually connects to the SONET node. b) Wireless/radio to the Substation. This option establishes a broadband radio connection to the substation. There are 2 types: point-to-point and WiMax. i) Point-to-point: this generally can be accomplished with low capacity radio terminals having 3-4 T1 capacity that operate in the unlicensed bands, such as the 5.8 Ghz band. The path must be designed to ensure a propagation reliability of better than %. Use of the unlicensed band is subject to potential interference. Alternatively, a licensed radio can be used (such as a narrowband microwave), but the costs may increase. Equipment hardware reliability can be increased by utilization of hot-standby radios. If the construction of new towers are required, costs for this option could increase significantly. ii) WiMax: A new technology that is becoming in use in WiMax, which adheres to the IEEE specification. It can have a maximum range of 30 miles under the right conditions, and may be a more cost-effective solution when multiple substations within the signal radius can be captured by a single National Grid 5-30 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

268 Attachment COMM 2-8(b) Page 132 of 160 radio, thus enjoying a degree of economy of scale. This is a relatively new technology, but several vendors such as Alvarion are actively pursuing the market. c) Leased Facilities. Leasing T1s or Frame relay facilities from a telecommunications provider is always available as an option for comparison, but with handing off traffic to the provider, a certain degree of control is lost. Disadvantages to leased lines are previously mentioned, and implications should also be considered from a security standpoint, as well as NERC guidelines. Leased facilities are established to the nearest SONET backbone access point from the substation. 6) If separate networks are established for operational and non-operational data, the use of a priority scheme to separate the traffic is unnecessary, unless there is a need to prioritize traffic within each data type. If prioritization is required, then the ability is required to prioritize IP traffic based on source address, source port, destination address, and destination port. If video traffic is mixed with other traffic it may be prioritized based on data type Corporate Communications Network Requirements 1) Many utilities have their operational traffic remain virtually point to point over private facilities, utilizing the SONET and establishing TDM over SONET with an overlayed DACS network. This allows cross-connect down to the DS0 level, and establishes an operational network of channelized DS0s which can be protected through the SONET protection switching schemes. Utilization of these circuits on the SONET also makes use of SONET protection switching through careful provisioning of the SONET connections to the primary and backup EMS. It is recommended that National Grid review the cost-effectiveness of this approach utilizing existing SONET architecture. 2) To plan the WAN backbone for accommodating traffic load as IED penetration increases at substations, a determination should be made of substation penetration objectives over a five-year planning horizon. Projects to add capacity to the SONET WAN should be established to accommodate the additional load in that time period. A reasonable target deployment percentage for planning purposes is 20% substation penetration, based on a minimum 3% yearly penetration growth. Based on that assumption, the target aggregate bandwidth to be accommodated on a network is approximately 2.2 Mbps for operational data, and 14 Mbps for non-operational data. Non-operational data could be accommodated within an OC-1 group of the WAN OC-48 capacity, configured to support IP WAN traffic. Operational data requirements justify two T1 equivalents separately segmented on the SONET backbone for the point-to-point traffic. If a leased frame relay network is used end-to-end in lieu of a SONET backbone for operational data, National Grid should establish service level agreements with Verizon to ensure performance parameters for latency and Quality of Service (QoS) are established that will meet the necessary operational data delivery requirements. National Grid 5-31 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

269 Attachment COMM 2-8(b) Page 133 of 160 3) To ensure the transport of pulled non-operational data though the backbone network will not create excessive congestion and latency issues in the network, a coordination process should be implemented among organizations needing certain data, so data flow can be reasonably spread out more efficiently. The key is to make data access as controlled as possible by laying out requirements appropriately, identifying what users require data and how often, and justifying business requirement needs for the data Security Requirements There are a number of Cyber Security issues that must be considered when implementing ELSSI, the most contentious being the definition and implementation of an effective electronic security perimeter. The electronic security perimeter is defined by the implementation of the firewall device (whether it is a router with access lists, a traditional firewall, or another device which implements access control and logging). When defining the electronic security perimeter, all critical cyber assets (i.e., the substation automation equipment, associated IEDs and the HMI) must be contained within the perimeter. Because of the nature of a wireless network, wireless technologies do not easily or economically permit establishing a full perimeter. A fully encrypted network might meet the requirement, but it s purchase, maintenance and performance costs and impacts are unknown. Note that while corporate communications may be available in the substation environment, the electronic security perimeter and the discussion of firewalls and configurations in this section deal with access to the operational substation equipment (i.e., IEDs, substation automation equipment, HMI, and data concentrators). The rules in the firewall at the perimeter should all be based on the business needs and requirements for access. For example, it is unlikely that is required to pass in either direction across the substation electronic perimeter, so the firewall should not be configured to permit traffic. If remote access to an HMI function is required, it should be done across a VPN, rather than allowing web browsing, X windows, or some similar non-secure mechanism is used to cross the electronic perimeter. The rules need to be set up to allow normal and emergency operations access to the substation equipment, but nothing more. Firewall rules should be able to be specified by 1) source and destination IP address, 2) TCP or UDP port, and 3) direction. Unless specifically allowed by a rule, access should be denied. Logging of access, especially denied access, should be maintained and stored in some form of central security server to determine if there are network breach attempts. Logs of successful access, particularly for configuration access should also be maintained to provide some form of audit log. When designing firewall rules, simpler is better. If a limited number of data warehouse nodes and remote access servers are configured in the corporate environment, the firewall rule sets at the substation electronic security perimeter will be easier to manage and maintain. Similarly, the monitoring of authorized and unauthorized access will require fewer resources. National Grid 5-32 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

270 Attachment COMM 2-8(b) Page 134 of 160 Using VPN technology to authenticate and encrypt the traffic may increase security of the communications to and from the substation equipment. Most commercial firewalls provide VPN termination functions, eliminating the requirement for additional hardware. This functionality may also be available in some newer data concentrators. Simplifying the software requirements within the substation will also assist in the security management of the substation environment. Less software running means less software to maintain, and less possibility for misconfiguration. Since the substation equipment is relatively static, serious consideration should be given to statically assigning addresses inside each substation, and not using dynamically assigned address technology (DHCP) or network based name to number lookup (DNS) inside the electronic security perimeter. This will reduce the maintenance of software within the substation environment (there is less software running within the substation), and will make it more difficult to inadvertently connect to the substation network (a laptop will no be assigned an IP address simply by plugging it in and turning it on). Any piece of equipment that is expected to be running within the environment will be configured with the proper addresses. The use of data concentrators and/or access servers that require individual authentication will provide an auditable record of who has accessed the devices, and when the devices were accessed. It may be possible to also audit what actions were performed, depending on the sophistication of the data concentrator or access device. In addition to the technical aspects of cyber security, there are procedural and policy aspects that must be addressed. These include the following: A Security Policy, specifically written to address the requirements of the substation and control network environments must be approved. Training and vetting of personnel with access to the critical cyber assets must be performed. Accurate records of who has access to the equipment, and what kind of access they have must be available. A process to remove access when it is not longer needed, or must be removed expeditiously. A Change Control process, including provision for testing must be implemented. Vulnerability assessments must be conducted to determine if there are any potentials for successful attack, and to determine how the system will respond to an attack. A process must be in place for managing passwords and accounts, including those provided by equipment suppliers. An incident response and recovery process, following ES-ISAC protocols, and generally accepted business continuity principles must be implemented. National Grid 5-33 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

271 Attachment COMM 2-8(b) Page 135 of Recommendations and Issues 6.1 Major Recommendations This section describes the major recommendations from this ELSSI Phase II report and is essentially a list of next steps that KEMA believes National Grid needs to take Recommendation #1 Condition Based Inspection and Maintenance Determine areas where improved CBI/CBM could have greatest benefits and prioritize substations, equipment, and specific monitoring technologies where these benefits would have most impact for early implementation of IEDs. For substations that already have IEDs deployed, identify CBI/CBM potential benefits and begin collecting data. Review CBI/CBM methodology to identify how continuous monitoring data can facilitate process changes and improvements Recommendation #2 Extension of Cost-Benefit Analysis and Deployment Plan Extend the cost benefit model used in Phase 2 and perform different scenario analysis to evaluate sensitivity of the model with respect to individual costs, benefits and current capabilities. Evaluate a target list of specific stations so that the variability in equipment age, station configuration, and station impact on system costs (congestion especially) are understood. Also, as part of this analysis some lower tech alternatives (inventorying spare power transformers and circuit breakers) could be considered and analyzed Recommendation #3 Working with Existing and Planned Initiatives National Grid is planning a number of initiatives (discussed in the Phase 1 report) which will benefit the functional requirements for ELSSI data integration and allow National Grid to take full advantage of these initiatives through their integration with ELSSI. In particular, the EMS and Gas SCADA design can be tasked with ensuring support of ELSSI objectives and Enterprise Architecture. National Grid 6-1 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

272 Attachment COMM 2-8(b) Page 136 of Recommendation #4 Business Process Design for Asset Management Asset Management business processes should be reviewed and plans made to improve them where indicated in order to exploit ELSSI capabilities and to ensure that the analyzed benefits are obtained and maximized. Also, the broad class of automation equipment should itself be considered as a set of assets subject to asset management and portfolio optimization in an integrated and holistic way Recommendation #5 Distribution ELSSI Investigation IEDs have the ability to provide data from all aspects of the electricity network. Currently transmission substations have had a high focus but distribution substations and distribution feeders have many locations where IED data would be beneficial. Telemetric provides the ability to cheaply collect information from dispersed feeder locations but require cost effective communications. KEMA recommends that National Grid conduct an ELSSI visioning and planning exercise aimed at distribution stations and feeders to see what reliability and economic benefits are possible Recommendation #6 ELSSI Pilot (See Sections and 3.1.6) KEMA recommends an ELSSI pilot that builds on the work of TISC as well as ELSSI Phase I and Phase II. The scope of this pilot and the justification and benefits are outlined in Section Summary of Other Recommendations This section is a reference for the reader to specific recommendations made in this report and summarizes the recommendations with a reference to the section where they were made Recommendation #7 Use proven data storage components (See Section 3.1.4) KEMA recommends using proven data storage components: Oracle, Documentum, PI as well as enterprise class data ingest components such as LiveData to ensure the architecture will scale over time as the number of automated substations grow. National Grid 6-2 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

273 Attachment COMM 2-8(b) Page 137 of Recommendation #8 Fact Model (see Section 4.3.5) Developing a comprehensive fact model is a logical next step that National Grid should consider taking. The initial fact model presented here uses terms that KEMA believes should have clear, unambiguous and consistent definitions within National Grid. The initial fact model illustrates the interaction of these terms Recommendation #9 Common Approach to Control (see Section ) KEMA recommends that hardwired electromechanical controls be used for substation controls for both transmission and distribution substations. Presently, control is done through the HMI for distribution substations but not for transmission substations. Hardwired controls are field proven and reliable and do not require a backup or redundant control facility which could actually reduce reliability due to the added complexity. A common approach across National Grid regarding control is important. The use of the HMI for viewing and hardwired electromechanical controls may be an issue for some people within National Grid. Conversely the use of the HMI for viewing and control may be an issue for others. KEMA believes that a common approach is very important Recommendation #10 Sequence Control Logic (see Section 4.5.9) KEMA believes that PLCs should be used for handling sequence control logic, and strongly recommends this approach at least for more complicated control schemes. However, it is very important that the concerns of the field personnel be investigated further and addressed as needed before committing to widespread deployment of PLCs for all control applications Recommendation #11 Data Concentrators (see Section ) Establish Ethernet LAN and data concentrators in Substations to support migration to IEDs and the ELSSI architecture. National Grid 6-3 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

274 Attachment COMM 2-8(b) Page 138 of Recommendation #12 Virtual LANs (see Section ) Utilize Virtual LAN connections to keep operational and non-operational data logically separate within the physical structure Recommendation #13 Wireless Communications (see Section ) Wireless LANs should be considered as a means to supplement other facilities, rather than replace them. The use of the unlicensed RF bands are subject to interference, and the openness of the signal over the air is cause for security concerns Recommendation #14 Separation of Data (See Section ) Operational data should be kept separate from the non-operational data as much as possible to ensure its critical delivery requirements will not be impeded Recommendation #15 Transmission Bandwidth (See Section ) To support ELSSI requirements at a transmission substation, a T1 should be initially provisioned for the operational and non-operational data. Both data types would share the physical facility via VLANs implemented for each data type to ensure data separation Recommendation #16 Distribution Bandwidth (See Section ) To support ELSSI requirements at a transmission substation, a T1 should be initially provisioned for the operational and non-operational data. Both data types would share the physical facility via VLANs implemented for each data type to ensure data separation Recommendation #17 Use of SONET (See Section ) National Grid should make use of its SONET backbone network to the greatest extent possible, and transport substation traffic (both operational and non-operational) over the private WAN. National Grid 6-4 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

275 Attachment COMM 2-8(b) Page 139 of Recommendation #18 Use of a Priority Scheme (See Section ) If separate networks are established for operational and non-operational data, the use of a priority scheme to separate the traffic is unnecessary, unless there is a need to prioritize traffic within each data type. If prioritization is required, then the ability is required to prioritize IP traffic based on source address, source port, destination address, and destination port Recommendation #19 TDM over SONET (see Section ) It is recommended that National Grid review the cost-effectiveness of having their operational traffic remain virtually point to point over private facilities, utilizing the SONET and establishing TDM over SONET with an overlaid DACS network utilizing existing SONET architecture Recommendation #20 Planning Horizon (see Section ) To plan the WAN backbone for accommodating traffic load as IED penetration increases at substations, a determination should be made of substation penetration objectives over a five-year planning horizon. Projects to add capacity to the SONET WAN should be established to accommodate the additional load in that time period Recommendation #21 Transport of Pulled Data (see Section ) To ensure the transport of pulled non-operational data though the backbone network will not create excessive congestion and latency issues in the network, a coordination process should be implemented among organizations needing certain data, so data flow can be reasonably spread out more efficiently. 6.3 ELSSI Pilot Details To demonstrate the collection of IED data and to help prove the use of the Orion 5R as the SCADA entrance device KEMA recommends an ELSSI pilot. The benefits of this are that it provides: National Grid 6-5 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

276 Attachment COMM 2-8(b) Page 140 of 160 Earlier payback on investment than projected by the benefit cost analysis since it allows credit to be taken for the LPSnn stations that have already been implemented, thus effectively reducing the cost basis of the benefit cost model. A basis for developing new methodologies by providing a database to store IED data and allow for analysis of that data. Proof of the key concepts discussed in both Phase I and Phase II as well as the work performed by TISC in developing and implementing the LPS98 and LPS03 substations. A way to keep the momentum moving and redirects it to the analysis and use of IED data which is where the benefit will ultimately be driven from. Low investment requirements since the substations selected for the pilot either already have a data concentrator installed or are able to be integrated into the pilot with the addition of a data concentrator at the substation. This leaves the database as a cost factor but with limited initial integration with other systems reduces the cost factor of integration with other systems until the data collection, storage and access is proven. A low risk approach to achieving a notable percentage of integrated substations providing data for analysis. Significant IT integration is not required and IED refits to large numbers of substations are not required. A reduced reliance of the benefit cost analysis on congestion by effectively reducing the cost component and reduces the payback time to recover the investment. Direct value to T&D Tech Services by adding extra data types that can be used by this group for reliability analysis simply by using the database as an ODBC data source for ArcGIS or other tools. Coverage of both New England and New York and a single system for storage and analysis of IED data. The ability for National Grid to reap further benefits from the existing LPS98/LPS03 work performed by TISC. The following steps (and subsequent diagram) provide an outline of the proposed ELSSI pilot. Document the purpose of the pilot, the questions it is meant to answer, and the criteria for determining if it has been successful. National Grid 6-6 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

277 Attachment COMM 2-8(b) Page 141 of 160 Extend the proposed Orion pilot from a single New England Distribution substation and include stations 28, 36, and 48 in New York. Preferably eliminate the interim RTU/Data Concentrator design and use the Orion as the SCADA entrance device. Deploy IEDs at Sandy Pond substation in the 345 kv yard. Develop a basic ELSSI database to hold the data and permit limited analysis. Design should be based on conforming dimensions of current and planned IEDs. IED data templates could also be incorporated into database to enable it to be used as a configuration management tool based on data requirements and IED capabilities. Use Wavewin for data collection (including substations currently connected to Wavewin) and prototype the use of a content management system for data storage. Develop a meta data lookup that can be used by the prototype ELSSI browser to catalog and search for Wavewin data. Users would access the data and perform analysis through Wavewin. Develop integration cases to exploit the data in ELSSI. Based on the Data Requirements Matrix and benefit cost analysis, plan interfaces between ELSSI and a low number of other systems to enable simple lookups and transfers to be performed. Initial candidate options will include an ELSSI ODBC service to enable access by ArcGIS and other tools, an interface to AIMMS, an interface to PI, and a transfer of responsibility from Wavewin to Documentum for file storage. This would also involve developing an access method for Wavewin to extract and analyze data from Documentum. Implement Orion 5Rs at 15 additional stations in New York that are ready to have them implemented. Implement the Orions as the SCADA entry device. Use spread spectrum or other lower cost means to connect the stations. (This is to prove feasibility and value of data, not the communications path). Enhance ELSSI pilot database and automate at least one integration with another system. This could be to perform reporting by exception event messaging or to automate regular data transfers. Use lessons learned to enhance the database design based on usage patterns and user requests. Integrate the 22 LPS98 substations into the ELSSI database. The RTU could be left as the entrance device and the data concentrator can be used to collect the IED data. Integrate the 15 New York stations into ELSSI. National Grid 6-7 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

278 Attachment COMM 2-8(b) Page 142 of 160 Integrate the 6 (may be more by then) LPS03 substations into ELSSI. Develop ELSSI browser as single point of contact for existing data. Develop methodologies for use of data in potentially high payback areas. Develop interfaces and tools to support these goals. Specifically, the ELSSI phase II work has identified 345kV transformer monitoring and digital protection integration as the highest payback first step. KEMA recommends that National Grid select a specific set of continuous transformer monitoring IEDs for installation in conjunction with the ELSSI pilot. Specific transformer failure prediction / detection algorithms should then be implemented in an ELSSI application integrated with the pilot ELSSI database. Evaluate the pilot. Exhibit 6-1: ELSSI Pilot Overview National Grid 6-8 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

279 Attachment COMM 2-8(b) Page 143 of Other Issues Some key questions still remain unanswered. Even with a favorable benefit:cost ratio, it would still be a lengthy and expensive prospect to retrofit existing substations with IEDs at an increased rate. Spending this money should yield not only improved business practices, and information quality, but also a payback on the investment and more. There is no issue of whether IEDs will get deployed. They are currently being deployed in new substations and are also being used in one for one replacements for electromechanical devices. Despite any potential benefits this is a long-term prospect and there will always be fires to be put out in the short term. This is why National Grid needs to decide whether or not the benefits of IED deployment meet corporate goals (including best practices) and whether or not the company can afford not to implement IEDs at a faster rate. Aside from the aspect of the resultant organizational change, the biggest unanswered question is the rate at which IEDs ought to be deployed (assuming that National Grid continue to move forward). This issue was discussed in the Final Report from Phase I and is also referred to in Section of this report. KEMA s recommendation is to continue the rollout of IEDs with a pilot project (also discussed in Section 6.3) and in the Executive Summary that focuses on collecting and analyzing the data from the IEDs. National Grid 6-9 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

280 Attachment COMM 2-8(b) Page 144 of 160 Appendix A Data Requirements Matrix Excel spreadsheet provided to National Grid under separate cover. National Grid A- 1 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

281 Attachment COMM 2-8(b) Page 145 of 160 National Grid Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

282 Attachment COMM 2-8(b) Page 146 of 160 Appendix B IED Templates Excel spreadsheet provided to National Grid under separate cover. National Grid B-1 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

283 Attachment COMM 2-8(b) Page 147 of 160 National Grid Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

284 Attachment COMM 2-8(b) Page 148 of 160 Appendix C Relevant Industry Standards National Grid C-1 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

285 Attachment COMM 2-8(b) Page 149 of 160 Relevant Industry Standards The widespread use of multifunctional Intelligent Electronic Devices (IEDs) with advanced communications capabilities has resulted in a new trend in Substation Automation (SA) the implementation of high-speed peer-to-peer communications based control schemes. The IEDs exchange messages over a substation Local Area Network (LAN). The communication messages replace the hardwired interface between control devices in conventional substations in order to perform critical control functions, as well as other user specific distributed control or recording applications. The following standards are all relevant in respect to this area. IEEE 1613 A task force (C2TF1) of the IEEE Power Engineering Society Substations Committee prepared this standard. Engineers from electric utilities, consultants, and manufacturers of IEDs, substation remote terminal units (RTUs), and communications networking devices formed this twenty-two member task force. Eleven months after the first meeting of the task force, the balloting was completed. The IEEE-SA Board, at its March 2003 meeting, approved IEEE 1613 (IEEE Standard Environmental and Testing Requirements for Communications Networking Devices in Electric Power Substations) as a standard. The PDF and paper versions of IEEE 1613 were published in August A number of manufacturers of communications networking devices report that they have development projects underway or completed to produce devices that meet this new standard. The successful application of communications based control schemes is possible only if the communications equipment used for the substation LAN or for interface with the utility Wide Area Network (WAN) meets the same (and in some cases perhaps even higher), environmental, and performance requirements than the existing control devices being integrated into a distributed control system. This is due to the fact that the failure of a single element of the substation LAN, such as a switch or hub, will result in a loss of communications between the different IEDs and in the failure of any distributed control scheme that relies on the exchange of communications messages. Task force C2TF1 decided to build on the relevant standards developed by the Power System Relaying Committee (PSRC), and make no changes in the specified test waveforms or test methods. The focus of the task force was to define the communications to be underway during the transient tests, and the new criteria for acceptance. In substations, when high voltage switches are opened or large inductors are deenergized, large oscillatory or fast rising transients are generated. These transients can be coupled to control wiring and be present on relay and control panels mounted in the substation control house. PSRC had developed four IEEE standards for protective relays and relay systems installed in the harsh substation environment: National Grid C-2 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

286 Attachment COMM 2-8(b) Page 150 of 160 C37.90 defines service conditions, voltage ratings, temperature operating ranges, humidity, and the required dielectric and impulse withstand capability C defines the required capability to withstand these known transients without damage or false operation C defines the required immunity to the operation of hand held transceivers ( walkie-talkies ) in close proximity to solid state relays C defines the required immunity of relays and relay systems to transients due to low humidity, high voltage static charges The task force decided to utilize the PSRC standards and adapt it to substation applications not involving protective relaying. They also chose to make the standard apply at the device, not system, level. In C , C , and C , each has a requirement of no false operations of the relay system during these transients. This requirement had to be modified for communications networking devices. Thus, IEEE 1613 defines the communications to be underway during the device tests. The PSRC standards require that no false operations occur as a result of these transients. These standards allow communications to be disrupted during the transient tests provided recovery is automatic without human intervention. IEEE 1613 defines two performance classes. Performance Class 1 is essentially identical to these PSRC requirements. The Substations Committee Task Force concluded that an additional optional performance class was warranted, requiring that the devices ride through these transients without interruption of communications. As a result, an optional Performance Class 2 was added. Note that Performance Class 1 is the minimum requirement in IEEE Performance Class 2 is not mandatory; it is an option that may be specified by a user or manufacturer. To meet the requirements of Performance Class 2, it is expected that fiber optic media will be necessary for the communications cables. The internal power supplies of communications networking devices that meet Performance Class 2 will also need to be very well shielded such that these transients that do not appear on the power supply input which could affect communications. Performance Class 1 This is for devices used for general-purpose substation communications where temporary loss of communications and/or communications errors can be tolerated during the occurrence of the defined transients. National Grid C-3 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

287 Attachment COMM 2-8(b) Page 151 of 160 Performance Class 2 This is for communications devices used in substation communications when it is desired to have error-free, uninterrupted communications during the occurrence of the defined transients. For both Performance Class 1 and 2, the Conditions to be Met (Acceptance Criteria) in passing the tests during the SWC, EMI, and ESD interference are: Equipment (both Performance Class 1 and 2) shall be considered to have passed tests if - during, or as a result of, the tests - all of the conditions below are met for the performance class of the device. UCA/UCA2 1) No hardware damage occurs. 2) No loss or corruption of stored memory or data, including active or stored settings, occurs. 3) Device resets do not occur, and manual resetting is not required. 4) No changes in the states of the electrical, mechanical, or communication status outputs occur. This includes alarms, status outputs, or targets. 5) No erroneous, permanent change of state of the visual, audio, or message outputs results. Momentary changes of these outputs during the tests are permitted. 6) No error outside normal tolerances of the data communication signals (SCADA analogs) occurs. Since the 1980s, EPRI had recognized the potential benefits of a unified scheme of data communications for all operating purposes across the entire utility enterprise. They focused on the ease of combining a broad range of devices and systems; and the resultant sharing of management and control information among all departments of the utility organization. EPRI had commissioned the Utility Communications Architecture (UCA) project, which identified the requirements, the overall structure, and the specific communications technologies and layers to implement the scheme. By 1994, EPRI had recognized the importance of tying substation control equipment and power apparatus into the UCA scheme, but had not defined a particular approach. Accordingly, they launched Research Project 3599 to define, demonstrate, and promote an industry-wide UCA-compatible communications National Grid C-4 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

288 Attachment COMM 2-8(b) Page 152 of 160 approach for substations. The objective was, and is, to avoid a frustrating and expensive marketplace shakeout of extremely complex incompatible systems. UCA was concerned with identifying the functions that utilities need to perform, or operability. UCA2 was concerned with defining the methods and language that would allow devices from different vendors to understand each other, or interoperate. Many progressive utilities, and most of the relay and IED manufacturers, took an immediate interest in UCA work and joined in the effort to define and demonstrate a communications network stack. The forward-looking approach was to define the technical requirements for a system to control and monitor substations of all sizes. The specification includes the requirement for fast messaging - in milliseconds - among peer IEDs to achieve fault-related control over the data communications system. The objective is to use substation local area network (LAN) messaging to ultimately replace the mass of dedicated wiring among the IEDs and power apparatus. Another feature of the approach was to identify communications system layers, which might already exist in widespread use, which would meet the requirements for substation control. This would allow the project workers to buy widely used hardware and software components, and focus their development efforts on the layers which really have to deal specifically with the needs of an electric utility substation control system. IEC At the request of users in the late 1980s, European suppliers created the communications standard International Electrotechnical Commission (IEC) whose subsections provided for basic information transfer and control between one vendor's IED and the overall system of another vendor. However, the markets where these vendors sold their products tended to support more expensive, futuristic systems as part of a major project. These systems could not be sold in North America due to their complexity and the North American market being in the infancy of substation automation. In 1995, IEC commissioned a new project, 61850, to define the next generation of standardized highspeed substation control and protection communications. The main objective, like the Electric Power Research Institute s (EPRI's), was to have vendors and utilities work together in the definition of the communications infrastructure for substation monitoring and control. The generation of this standard would assure interoperability of the various vendors IEDs avoiding the extremely complex incompatible systems. The IEC project organization, tasked with defining the communication standard, resides under the Technical Committee 57 (TC 57). IEC TC57 is chartered with developing standards for electric power system management and associated information exchange in the areas of generation, transmission and National Grid C-5 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

289 Attachment COMM 2-8(b) Page 153 of 160 distribution real-time operations and planning. There are several different initiatives within TC57, each dealing with a selected part of the real-time operations and planning. Each has a specific objective and may have sufficient breadth of scope to provide the bulk of the relevant standards needed for product vendors to develop products based on those standards. In today s utility enterprise, where information exchange between the various generation, distributed resource, transmission, and distribution management systems, as well as customer systems and other IT systems is not only desirable but necessary in most cases, each system plays the role of either supplier or consumer of information, or more typically both. That means that both data semantics and syntax need to be preserved across system boundaries, where system boundaries in this context are interfaces where data is made publicly accessible to other systems or where requests for data residing in other systems are initiated. In other words, the what of the information exchange is actually much more important for system integration purposes than how the data is transported between systems. As the need for standards to address substation automation was identified, new working groups were formed (WG10-12) to develop standards for architectures and interfaces within substations and on distribution feeders. Because of similar and closely related objectives, these three working groups eventually merged into a single working group working group 10. Unlike some other TC57 protocols which have a flat or tag oriented data hierarchy, IEC models are hierarchical in nature. The general philosophy is to represent substation functions (e.g. metering or protection) as objects and to define standardized naming, attributes, data, and methods of such objects. A client then interacts with this object model directly in order to access it for purposes of reading attribute values, such as nameplate data or measured values, or to control the device, rather than indirectly through an RTU. The common services needed by all substation devices, especially field devices, are modeled as server objects, which are defined by the Abstract Communication Service Interface (ACSI). Field devices incorporate these services by specifying which objects within their models inherit the class objects defined in the ACSI. For example, if a model of a utility field device contains a measured value, which needs to be read by a substation host, the object inherits the attributes and methods associated with the class object Basic Data Class defined in The standards are based largely on object models for substation devices that originated with the EPRI-sponsored UCA2 project. The standards specify two buses: a station bus, which is used to communicate between Intelligent Electronic Devices (IEDs), and a process bus communication, which is used to send information from digital sensors to IEDs. There are two data exchange patterns supported: client/server and publish/subscribe. Several types of communication interfaces and requirements are specified in the standards. Interfaces 1,3,6, 8, and 9 comprise the station bus. Specifications for these interfaces define the requirements for peer-to-peer information exchange of physical substation devices (e.g., Bay controllers, protection relays, meters). Interfaces 4 and 5 comprise the process bus. Specifications for these interfaces National Grid C-6 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

290 Attachment COMM 2-8(b) Page 154 of 160 define the requirements for sensor-to-device information exchange. Interface 7 enables remote engineers to communicate directly with devices on the station bus for the purposes of monitoring, configuration, and diagnostics. Interface 10 supports SCADA data exchange with a remote SCADA master. Although it has similar requirements to Interface 7, it is not within the scope of WG10. The protocols to support this information exchange are the responsibility of other working groups. Standardized mappings of these abstract services to different application layer communications protocols profiles are defined in IEC x (station bus) / x (process bus), so that common utility functions will be performed consistently across all field devices independent of the underlying communication stacks. As described earlier, the architecture of the station bus and its communication services enables bay/unit level devices to interact in a peer-to-peer fashion. It also supports the remote monitoring of these devices either directly (e.g., peer-to-peer) or through some type of information aggregator (e.g., sub-scada master, RTU, or Bay controller) at the Station Level. IEC contains three protocols that are peer-to-peer multicast datagrams on a substation LAN and are not routable. The messages need to be transmitted within 4 milliseconds and so that encryption or other security measures which affect transmission rates are not acceptable. Therefore, authentication is the only security measure included, so IEC provides a mechanism that involves minimal compute requirements for these profiles to digitally sign the messages. The different communication profiles of IEC require security enhancements to ensure that they can be implemented and used in non-secure environments. 1) Client/Server, which will specify security primarily through TLS and MMS, but may include additional security measures such as VPNs 2) Generic Object Oriented Substation Event (GOOSE) analogue and digital multicast 3) Generic Substation Status Event (GSSE) digital only multicast, Manufacturing Message Specification (MMS) involved 4) Generic Substation Event (GSE) Management 5) Sampled Measured Values (SMV) In Logical Device models represent the behavior of real devices. This is accomplished by defining standard classes and objects (instances of classes) built up through inheritance and aggregation from a common set of ACSI class definitions. The ACSI is defined in and basically defines the server objects that are used for communicating with field devices but not the content of the information National Grid C-7 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

291 Attachment COMM 2-8(b) Page 155 of 160 transferred. Logical Devices, which are the subject of and 4, define the content of the information transferred. Users of ACSI-based devices can access the device features through well-defined network services operating on the objects. The ACSI data access model defines the rules for defining and organizing the object models specified by industry groups into objects that can be used in communications. The standards incorporate the services and models from the EPRI Common Access Service Methods (CASM) with some revisions based on more recent developments. The object-oriented terminology used in these standards is similar to the UML used in the CIM and includes: class, object, method, attribute, inherit, instantiate, and aggregate. However, uses the object modeling facilities of ASN.1, ISO/IEC rather than UML. The type language specified in ISO/IEC is used for describing the abstract structure of a protocol, that is, the data present in a message. It does so by providing a notation for data values and data types ACSI In ACSI, a client/server model is assumed, in which the client initiates transactions that are processed by the server. The ACSI server hides real data and devices, using objects to represent them instead. Objects that are directly accessible by a client through a network are contained in an object from the server class. The servers, and the object instances they contain, are mapped to the communication stacks, for communication with the real devices. Logical Devices Logical Devices are virtual representations of real substation and field devices. As in real devices, Logical Devices are a composite of multiple parts, which are represented by Logical Nodes. The collection of these Logical Nodes provide the functionality of the complete device. For example, a distribution relay device might include several standardized relay functions. In addition, an electronic distribution relay would likely have a capability to measure the voltages and currents in the conductors it is controlling. To represent this device in 61850, a Logical Device would be created that contained a nameplate, Device Identity, a measurement unit Logical Node, and one or more standardized relay function Logical Nodes. It should be noted that allows for arbitrary assembly of Logical Nodes into Logical Devices. The composition of a Logical Device is left to the manufacturer and can always be determined from an instance of a Logical Device via the communications services of ACSI. With the use of Logical Devices and the ACSI for communication with substation and field devices, the properties of each device can be discovered and used to populate a database in the control center. Any changes in the field (i.e., new installations, revisions to existing installations, removal of field National Grid C-8 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

292 Attachment COMM 2-8(b) Page 156 of 160 equipment, etc.) can be discovered automatically as the changes are made, rather than requiring a separate manual data entry at the control center. Communication Abstraction In such computerized substation systems it is still possible to connect to control centers with RTU-based telecontrol protocols, as e.g. with a protocol of the IEC series using a virtual RTU gateway for protocol conversion (objects and services). This includes with IEC also the use of IPnetworks. But as an option the local protocols specified by the IEC series for the LANenvironment can be used as well as telecontrol protocols in the WAN eliminating the need for such gateways. This solution is called a seamless communication profile of the IEC TC57 Reference Architecture allowing seamless communication from the primary process equipment and IEDs in substations up to the control center and any other application that needs to exchange information with IEDs, e.g., engineering station and condition monitoring station. It is important to note that seamless in this sense means an abstract communication layer (abstract nodes, objects and services). Seams will still exist on real communication layer if different protocols are used for substation communication and telecontrol. Even if the same protocols are used seams exist with substation proxies. Nevertheless the seamless abstract communication layer allows a more efficient data management of the overall control systems eliminating unnecessary protocol conversions and has the potential to reduce cost of implementation and over the live cycle of systems. A typical system implementing the seamless profile would either directly interconnect the IEDs over the substation network with a router to the control center or via a central substation host acting as a proxy. Typically such a substation host has a process database and can perform application services need by the control center such as routing, filtering, general interrogation. Before any communication between control center and substation can take place the process data models and system configurations of both sites must be synchronized by the exchange of configuration data common to both substation and control center. This configuration data are defined with the substation configuration language IEC (SCL). They are a subset of the substation configuration data and include besides device specific data the description of related power resources (topology and primary equipment), the mapping to real protocols, and communication network configurations. Device specific object names can be associated with power resources names and alias names to enable the association of device objects to instances of power resources governed by the Common Information Model (CIM) used in control centers. NERC Cyber Security (CIP) NERC standards CIP-002 through CIP-009 provide a cyber security framework identifying and assisting with the protection of Critical Cyber Assets to ensure reliable operation of the Bulk Electric System. National Grid C-9 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

293 Attachment COMM 2-8(b) Page 157 of 160 These standards recognize the differing roles of each entity in the operation of the Bulk Electric System, the criticality and vulnerability of the assets needed to manage Bulk Electric System reliability, and the risks to which they are exposed. Business and operational demands for managing and maintaining a reliable Bulk Electric System increasingly require Cyber Assets supporting critical reliability functions and processes to communicate with each other, across functions and organizations, to provide services and data. This results in increased risks to these Cyber Assets. The NERC standards apply appropriate from traditional TI security concepts to real time environments, including policy and procedure development, access control, security perimeters, auditing, and change management. See Section 3.5 for more discussion on the NERC CIP standards as they apply to the ELSSI project. The following paragraphs provide a brief overview of each of the CIP standards as they exist in Draft 3. CIP 002 (Critical Cyber Assets) This standard requires the identification and enumeration of the Critical Cyber Assets that support the reliable operation of the Bulk Electric System identified through the application of a risk-based assessment procedure. CIP 003 (Security Management Controls) This standard requires that Responsible Entities have minimum security management controls in place to protect Critical Cyber Assets, specifically including a Cyber Security Policy, Senior Management support, an Information Classification and Protection program, an Access Control program, and a Change Management program. CIP 004 (Personnel and Training) This standard requires that personnel having authorized access to Critical Cyber Assets, including contractors and service vendors, have a higher level of risk assessment (background check), training, and security awareness, than personnel not provided access. CIP 005 (Electronic Security) This standard requires the identification and protection of the Electronic Security Perimeter(s) inside which all Critical Cyber Assets reside, as well as all access points on the perimeter. CIP-006 (Physical Security of Critical Cyber Assets) This standard is intended to ensure the implementation of a physical security program for the protection of Critical Cyber Assets. National Grid C-10 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

294 Attachment COMM 2-8(b) Page 158 of 160 CIP 007 (Systems Security Management) This standard requires that Responsible Entities have system security controls in force to detect, deter, and prevent the failure or compromise of critical functions performed by Critical Cyber Assets caused by mistake, misuse, or malicious activity. This standard requires Responsible Entities to define methods, processes, and procedures for securing those systems determined to be Critical Cyber Assets, as well as the non-critical Cyber Assets within the Electronic Security Perimeter(s). CIP 008 (Incident Reporting and Response Planning) This standard ensures the identification, classification, response and reporting of Cyber Security Incidents following established thresholds and procedures. CIP 009 (Recovery Plans for Critical Cyber Assets) This standard ensures that recovery plan(s) are put in place for Critical Cyber Assets that these plans follow established business continuity and disaster recovery techniques and practices, and that the plans are exercised annually. IEEE C This IEEE Standard, Standard for Common Format for Transient Data Exchange (COMTRADE) for Power Systems, is part of the C37 family on Switchgear, Substations and Protective Relays. COMTRADE (Common Format for Transient Data) is an important IEEE standard first published in 1991 and updated in It defines a common format for data files and an exchange medium used for the interchange of various types of fault, test, or simulation data for electrical power systems. The IEEE standard also describes the sources of transient data such as digital protective relays, digital fault recorders and transient simulation programs and discusses the sampling rates, filters, and sample rate conversions for the transient data being exchanged. The C standard also establishes that the following files are necessary for the data exchange: header file (.HDR), configuration file (.CFG) and data file (.DAT). Digital fault records are an invaluable tool for power system analysis. Relay engineers and test technicians can analyze transient records for information not only on power system disturbances, but also during routine commissioning of transmission, distribution and generation protection systems, and the related protected equipment. The advanced capabilities of digital relay systems allow for the ability to capture, store, retrieve, and analyze a fault record with very little effort. A 3-phase test set allows the test technician to playback a COMTRADE file simulating the exact power system conditions that were originally recorded. This is an extremely useful tool for determining relay performance during an event or questionable operating situation, or to assist in the commissioning of new facilities. National Grid C-11 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

295 Attachment COMM 2-8(b) Page 159 of 160 COMTRADE files can provide accurate simulation of power system events. Transmission and distribution system faults that are recorded can easily be re-played to assure accurate relay performance, validating relay setting information, as well as relay scheme logic, relay coordination, communicationsaided schemes, and SCADA system actions. ISO/IEC This standard (Information technology - Security techniques - Code of practice for information security management) establishes guidelines and general principles for initiating, implementing, maintaining, and improving information security management in an organization. The objectives outlined provide general guidance on the commonly accepted goals of information security management. The standard contains best practices of control objectives and controls in the following areas of information security management: Security policy Organization of information security Asset management Human resources security Physical and environmental security Communications and operations management Access control Information systems acquisition, development and maintenance Information security incident management Business continuity management Compliance The control objectives and controls in this standard are intended to be implemented to meet the requirements identified by a risk assessment. It is intended as a common basis and practical guideline for developing organizational security standards and effective security management practices, and to help build confidence in inter-organizational activities. The revised standard (2005), integrates the latest developments in the field to maintain it as the international standard code of practice. This new version addresses the security of information in its widest sense, providing best business practice, guidelines and National Grid C-12 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

296 Attachment COMM 2-8(b) Page 160 of 160 general principles for implementing, maintaining and managing information security in any organization, producing and using information in any form. ISO/IEC 17799:2005 recognizes that the level of security that can be achieved purely through technical means is limited. The required level of security established through assessing the levels of risk and associated costs through breaches of security, against the costs of implementing security should always be driven by appropriate management controls and procedures. Information security management requires, as a minimum, participation by all employees in the organization. It may also require participation from shareholders, suppliers, third parties and customers. It also identifies the controls that form the starting point for information security. It covers the critical success factors, the organization of information security, asset management, human resources, physical and environmental security, communications and operations management, information systems acquisition, development and maintenance, incident management, business continuity management and compliance. National Grid C-13 Proprietary National Grid ELSSI Phase II Systems Architecture Report August 2005

297 Attachment COMM 2-8(c) Page 1 of 103 feel Enterprise Level Substation Systems Integration Study Report National Grid USA 25 Research Drive Westborough, Massachusetts April 7, 2005

298 Attachment COMM 2-8(c) Page 2 of 103 Table of Contents Executive Summary... 1 Cost Benefit Analysis Summary and Overall Recommendations... 1 ELSSI Phase 1 Background... 3 Synopsis of Current Situation... 4 Gaps to Best Practice and Estimated Financial Savings to NGRID... 6 Important Concepts of Scalability, Deployment, and Mapping of Benefits Introduction Utility Evolution Organization of Report Methodology References Issues and Opportunities for Enterprise Use of Operational and Non-Operational Data Data Mart Existing Conditions Impacts to Business Users Known Initiatives Best Practices Gaps and Priorities IED Penetration Existing Conditions Impacts on Business Users Known Initiatives Best Practices Gaps Data and Diversity Existing Conditions Impacts to Business Users Known Initiatives Best Practices Gaps and Priorities SCADA Penetration Existing Conditions Impacts to Business Users Known Initiatives Best Practices and Gaps Asset Management and Condition Based Maintenance Best Practice and Gap Analysis Benefit Cost Analysis ELSSI BCA Methodology ELSSI Candidate Functions T&D Planning/Design Candidate Functions National Grid i Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

299 Attachment COMM 2-8(c) Page 3 of 103 Table of Contents 3.4 Fault Location and Trending Candidate Functions Condition Based Maintenance (CBM)/Maintenance Management Candidate Functions Operations and Control Candidate Functions Opportunity Matrix Benefits Transformer And Circuit Breaker Inspections Based On Condition Capital Asset Management And Planning Targeted Dispatching Dynamic Equipment Rating Voltage/VAR Control Feeder Reconfiguration and Load Balancing Intelligent Alarm Processing Outage Planning & Scheduling Costs Benefit Cost Analysis BCA Conclusions Vision and Recommendations Vision for the Future Data Mart Vision Communications Vision Recommendations Cost Benefit Analysis and Overall Recommendations Appendix A: Completed Interview Templates for Each User Group...A-1 Appendix B: Questionnaires & Analysis of Questionnaire Responses... B-1 Appendix C: Interview Schedule & List of Interview Groups... C-1 Appendix D: Information Request Form & Results...D-1 Appendix E: Data Requirements Matrix... E-1 Appendix F: Priority Matrix for Data...F-1 Appendix G: Initial Indications of Information Maturity...G-1 Appendix H: Project Reference Materials...H-1 List of Exhibits: Figure 1-1: All Utilities are Fundamentally the Same Figure 1-2: The Drivers for Utility Evolution Figure 2-1: Analysis of Data from Multiple Sources Figure 2-2: Data Mart Vision Figure 2-3: User Access to Data Mart Figure 2-4: Data Mart Action Pyramid National Grid ii Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

300 Attachment COMM 2-8(c) Page 4 of 103 Table of Contents Figure 2-5: Example Substation One-Line Diagram Figure 2-6: Theoretical Replacement Rates Figure 2-7: Number of Refits Required during IED Rollout Figure 2-8: SCADA Observability Figure 2-9: Best Practices CBM Model Figure 2-10: Financial Performance versus Risk Figure 3-1: ELSSI Impact On Each Benefit Area Figure 3-2: ELSSI Benefits By Opportunity Type Figure 3-3: Ranking of ELLSI Candidate Functions Figure 3-4: Opportunity Matrix Figure 3-5: Estimated Benefits of Condition Based Inspections Figure 3-6: Estimated Benefits of Condition Based Asset Management Figure 3-7: Estimated Benefits of Targeted Dispatching Figure 3-8: Estimated Benefits of Dynamic Equipment Rating Figure 3-9: Estimated Benefits of Improved Volt/VAR Management Figure 3-10: Estimated Benefits of Improved Feeder Reconfiguration and Load Balancing Figure 3-11: Estimated Benefits of Intelligent Alarm Processing Figure 3-12: Estimated Benefits of Outage Planning and Scheduling Figure 3-13: Foundation Cost for Transmission Substation Figure 3-14: Foundation Cost for Distribution Substation Figure 3-15: BCA Results for National Grid Transmission Substation Figure 3-16: BCA Results for National Grid Distribution Substation Figure 3-17: Incremental BCR for Transmission Substations Figure 3-18: Incremental BCR for Distribution Substations Figure 4-1: Different Levels of User Access Figure 4-2: Drivers, Benefits and Consequences Figure 4-3: ELSSI Communications Vision National Grid iii Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

301 Attachment COMM 2-8(c) Page 5 of 103 Executive Summary Cost Benefit Analysis Summary and Overall Recommendations KEMA believes that National Grid should develop overall ELSSI strategies, plans, and architectures aimed at closing the key gaps identified above, especially Condition Based Maintenance, Asset Management/Utilization, Fault Location, and Intelligent Substation Alarming, Outage Trending, and Fault Data Retrieval. The estimated savings from these ELSSI exploitations are, as noted, $25M to $30M a year. The cost and benefit numbers behind these recommendations are extremely rough as there was no engineering analysis done of the existing infrastructure, maintenance budgets, capital budgets, or deployment costs. To amplify on the overall recommendation stated above KEMA recommends the following specific activities as next steps: Recommendation 1 Develop an ELSSI technical architecture including IEDs, communications, data models and integration, and data marts as soon as possible. This will have the benefit of ensuring that ongoing IED deployment, applications procurement, and so on all will be compatible with the final ELSSI plan as it is developed and that retrofit costs of infrastructure, systems, and software are avoided. This is an inexpensive step to take at this time the costs occur when the integration systems and communications are deployed. The longer National Grid waits to take this step, the more difficult it will be to realize the benefits of ELSSI and the more expensive it will be to retrofit an architecture. Due to the high IT component and inter-departmental reach of the data mart, National Grid s IT department should play a significant role in helping to develop this architecture. Recommendation 2 Analyze the T&D infrastructure more carefully to refine the estimates of savings from CBM and AM strategies and to identify targeted equipments / stations for deployment and exploitation. This will allow development of the ELSSI deployment plan aimed at CMB and AM. To avoid the costs associated with doing this at an equipment-specific level, it can be done in classes of equipment as in 230kV power transformers of type XYZ with historic loadings >90% over 100 hrs / year. While this may appear to be a chicken and egg situation, transformers to be monitored could be selected based on engineering information including transformer age and history. Deciding on the overall time frame and rate of IED deployment is a high priority so as to eliminate the costs and inefficiencies associated with today s deployment methods. This step is key to determining what deployment rate and strategy National Grid will implement. As with National Grid 1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

302 Attachment COMM 2-8(c) Page 6 of 103 the architecture design, the cost of this step is low compared to implementation and the benefits great. As an adjunct to this effort, KEMA recommends a discussion of current CBM and AM plans to validate our observation that probabilistic risk analysis is not being planned and to explore the potential benefits to National Grid in more detail. Recommendation 3 Analyze outage records to identify the parts of the system most likely to benefit from district-wide availability of electronic fault location and optimized dispatch, in order to identify the SAIDI improvements and crew cost savings potential. This will allow the evaluation of ELSSI for these regions and the development of an ELSSI plan. Recommendation 4 Develop an implementation of ELSSI-based CBM, including processes and steps required to integrate it fully within National Grid s strategic and tactical asset management programs. ELSSI and some of the other recommendations given above will provide the base for an effective CBM function which has estimated benefits of $10 - $12 million per year for National Grid. However, if CBM is maximally coordinated with the rest of the portfolio planning process, it will provide additional benefits beyond those savings, including further cost reductions and increased business flexibility at lower risk. To provide those additional benefits, CBM must be implemented not as a stand-alone optimization but in a way that integrates it into the rest of National Grid s asset management and spending/resource commitment decision processes. Recommendation 5 Identify the substations where intelligent substation alarming can be deployed. (Note: this requires IED deployment but depending upon SCADA penetration and RTU spare capacity, may not require communications deployment) Recommendation 6 Wavewin (Fault Event Diagnosis and Analysis) will be a key part of ELSSI deployment and essential to realizing the benefits of ELSSI. In the development of an ELSSI architecture Wavewin compatibility should be addressed early so that Wavewin enhancement and build-out can begin in parallel with the ELSSI strategy work. Securing funding for the Wavewin initiative is a critical piece of making ELSSI successful. Recommendation 7 There are a number of secondary benefits of ELSSI that were identified by various departments. One example is support of trending analysis of distribution outages to support tree trimming program decision-making. National Grid has, not surprisingly, identified tree trimming as the most significant contribution to distribution reliability, and National Grid 2 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

303 Attachment COMM 2-8(c) Page 7 of 103 enabling better decision making should result in both cost savings and improved reliability. Recommendation 8 Increase the rate of IED rollout substantially. Analyze the suitability of National Grid s service territory for a district that could be used for a more widespread deployment of IEDs to explore benefits and provide a test bed of data to be used in supporting and enhancing data mart development. This would also allow development of new dispatching procedures, which could be kept consistent within that district. Recommendation 9 It is time to replace the EMS system in New York. National Grid owns, maintains and operates a 9,000-mile transmission system serving five states and a service area of 29,000 square miles. National Grid provides transmission services for more than 310 generators, five electric distribution companies, and 53 municipal electric companies and interconnects with a dozen other transmission providers. National Grid believes this can best be accomplished within an independent transmission company structure which can achieve many of the Federal Energy Commission's goals of reducing transmission costs for customers and encouraging efficient transmission operation, yet the EMS in New York is one of the oldest transmission Energy Management Systems in the United States. The ability of the existing NY EMS to support new initiatives such as ELSSI is limited and thus it imposes an opportunity cost on NGRID so long as it is not replaced. Having a definite plan and time frame for its replacement would enable other initiatives such as ELSSI to establish plans, budgets, and schedules without the uncertainties or limitations imposed by the NY EMS. ELSSI Phase 1 Background National Grid engaged KEMA, Inc. to prepare an analysis of National Grid s current deployment and usage of real time and engineering data from substation Intelligent Electronic Devices (IEDs) across the enterprise, compare the current situation with best practices, and develop a high level vision of how National Grid might modify current deployment plans and strategies for IEDs and how it might make best use of the data accessible across the enterprise to improve business processes. As part of the study, KEMA conducted 48 interviews with approximately 110 National Grid employees from 25 departments in New England and New York that spanned all aspects of electric T&D engineering, maintenance, operations, asset management, metering, and construction. KEMA also talked to IT and communications personnel about existing computer and communication resources, as well as cyber security and to representatives of the Gas Distribution business in New York. The interviews focused on what field data National Grid 3 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

304 Attachment COMM 2-8(c) Page 8 of 103 the departments make use of today in what processes and tasks, and how better access to more data might allow them to improve their operations. KEMA then compiled the results of these interviews and gathered corrections and comments from the interviewees to produce final summaries, which have been verified by the National Grid participants. The information collected and the results of the interviews are attached as appendices to this report. KEMA then produced a summary of the current state of National Grid deployment of IEDs, the systems used to communicate with and disseminate data from the IEDs, and applications software/systems in National Grid that make use of this data (or could if it were accessible to them). The study charter was to then identify where National Grid could positively impact safety, reliability, and cost by changing its strategies for Enterprise Level Substation System Integration (ELSSI) and by modifying its engineering, maintenance, and operations business practices to take advantage of a changed ELSSI strategy. This was done at a high level by KEMA as the study time frame did not allow for comprehensive Business Process Re-Engineering, nor did it allow for a detailed engineering analysis of the state of the National Grid T&D infrastructure and current operations, maintenance, and engineering practices. KEMA relied on overview descriptions of the state of the infrastructure provided during the interview process, and on verbal descriptions of critical business processes obtained during the interviews. Based on this high level information, KEMA has prepared a qualitative assessment of where National Grid is compared to best practice in exploiting ELSSI within the electric utility industry. KEMA performed a high-level benefit cost analysis to identify areas for ELSSI implementation that would provide the largest potential opportunities for cost savings, productivity improvement, and T&D system performance improvement at National Grid. Order of magnitude estimates of the potential cost savings for National Grid were determined by extrapolating results from similar sized utilities that have either achieved these savings through actual implementation of ELSSI functions or have performed detailed studies to determine the estimated benefits. This report also provides recommendations for integration of some existing initiatives (a prime example is Wavewin) into an ELSSI plan and possibly accelerating these initiatives even though the study was unable to do the Business Process Analysis that would lead to a quantitative financial justification. Synopsis of Current Situation National Grid has somewhat disparate situations in New England and New York. In New England, the data communications to substations is still principally low bandwidth analog or digital lines as required to support metering and SCADA, and IED penetration is very low in both transmission and distribution substations (less than 5% of all protective relays are IEDs, with the balance being electromechanical or solid state electronic devices). New substations (almost entirely distribution) are constructed using IEDs almost exclusively. While fiber optic communications exist to some substations, almost all substations are still on low speed (9600 or 1200 bps) digital or analog National Grid 4 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

305 Attachment COMM 2-8(c) Page 9 of 103 communications (compared to minimum 56Kbps for substation communications as best practice.) There is no plan at this time to systematically eliminate non-ied protective relays on a fixed schedule. When electromechanical and solid state electronic devices fail, they are often replaced with a similar device, if it is not practical to install an IED. If a similar technology device is not available, an IED device is installed. Communication system extensions do not necessarily follow retrofit. The New England EMS system, while near end of life from the standpoint of vendor support, is reasonably modern and provides enterprise access via a number of mechanisms of varying convenience. There is also a PI Historian system that provides somewhat more widespread and convenient enterprise access, and which is widely accessed by the engineering and maintenance departments. Wavewin also provides access to IED operational and engineering data where IEDs exist using dial up telephone communications with the attendant usability and security concerns. In New York, the communications infrastructure is better developed; however IED penetration is also low and there is not the same standardized deployment strategy as exists in New England. The SCADA system is actually well beyond end of life in terms of obsolescence and support, and does not support enterprise access. The ERS system used analogously to PI does not provide similar enterprise support. In both locations, engineering and maintenance departments seem well aware of what ELSSI capabilities exist. Broadly speaking, however, (and there are exceptions to this statement) the low penetration of IEDs, the difficulties of access, and the lack of standardization in retrofit deployment have prevented the various departments from developing procedures that exploit the integration and information capabilities of the IEDs. Safety and reliability are the driving concerns in an environment where lack of uniform and consistent deployment and data access undermines operational faith in the systems. The lack of uniformity has created sub-specialization in some departments where one or two individuals specialize in handling stations based on what degree and type of automation exist. The slow deployment of IEDs guarantees that over time National Grid will always have multiple generations/types of equipment to maintain, with varying data models/interfaces/operating characteristics. This makes it difficult to achieve any process improvements in the departments charged with designing, installing, and maintaining the substation instrumentation, protection, and control equipment, and also continues the difficulties the other departments have in escaping manual processes. The report includes an analysis of this effect and a discussion of its impact on staffing and knowledge retention. This is a major risk area for National Grid long term. National Grid is also at risk for having to dramatically accelerate IED deployment triggered by increased failure rates in electromechanical equipment or unavailability of spares/replacements in the (near) future. In a number of cases, applications which might be expected to exploit ELSSI do not, due to issues of translating different data models, uncertainty about the quality of data in the various applications, and so on. Thus data is manually screened and selectively re-entered in these cases. National Grid 5 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

306 Attachment COMM 2-8(c) Page 10 of 103 Anecdotally, a number of departments expressed an inability to aggressively perform condition based maintenance, dynamic rating, or other asset management techniques due to lack of the necessary real time and historical data. Gaps to Best Practice and Estimated Financial Savings to NGRID ELSSI is actually new enough that no one utility has completely revamped their T&D operations to exploit ELSSI comprehensively. However, there are utilities that have achieved large savings in capital costs via asset management based on ELSSI, that have achieved SAIDI improvements by utilizing fault location data from the IEDs, that have captured savings from condition based maintenance, and so on. There are also utilities that have adopted enterprise standards for ELSSI, including IED equipment and deployment in substations, data models and data integration, and enterprise access. The key gaps to best practice that are most significant to National Grid are as follows: National Grid does not have an enterprise plan for ELSSI with a time frame for achieving a target deployment strategy. It is very difficult for the T&D departments to plan to exploit ELSSI without a definitive plan for ELSSI itself. An ELSSI architectural design including communications, data models, system integration, and data mart design and access does not exist. Without this design and consequent standards, departmentally driven decisions will guarantee a continued proliferation of manually integrated processes and systems. National Grid is unable to plan for and take advantage of aggressive Condition Based Maintenance (CBM) or Asset Management (AM) strategies due to lack of the real time and historic data accessible to the departments that would perform CBM and AM. Experience of other utilities suggests and the benefit cost analysis by KEMA confirms that National Grid would be able to realize CBM/CBI and Asset Management (maintenance and capital replacement) savings on the order of $259K/yr/transmission substation (reference Figures 3-15 and 3-16). The estimated cost for targeted deployment of ELSSI to achieve these savings is approximately $1.1M/substation, so that the deployment has a positive B/C ratio for this application. These estimates are extrapolated from benefits determined by other electric utilities without allowing for the age of the equipment or the relative load growth rates of the utilities. The cost estimate does include the cost of developing a data mart for IED and other operations data. It also does not include any additional benefits that might accrue from analysis of these data. National Grid does have an Asset Management initiative under way, and is beginning to utilize the software tool for enterprise asset project prioritization. However, the source data used to justify and classify projects is not robust, in many cases, due to lack of data retrieval and analysis National Grid 6 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

307 Attachment COMM 2-8(c) Page 11 of 103 capabilities. An ELSSI strategy needs to be part of any Asset Management program. Also, while the current initiative is a major step forward, we believe that the initiative can be more sophisticated in the future by embracing probabilistic risk analysis within the program. We were unable to obtain very much detailed information on the Asset Management strategies being developed, perhaps due to the newness of the initiative, and our comments are therefore somewhat tentative. Use of IEDs to perform fault location and electronic communication of this to field personnel during outage restoration has been demonstrated to have significant positive impacts on SAIDI (in the range of 4-7 minutes per event plus tangible savings in field crew overtime costs). A utility the size of National Grid might see cost savings on the order of $5K/yr/substation from an incremental (over and above IED foundation costs) investment of $4K/substation in ELSSI. National Grid is currently unable to take advantage of ELSSI for the purpose of fault location. Targeted deployment of IEDs to achieve intelligent substation alarming as opposed to simple trouble alarms could improve crew dispatching and improve response accuracy at relatively low cost. We believe that better equipment utilization (overrating and dynamic rating) will also in the future have a positive impact on congestion costs and purchased power costs as well as reducing dependence on purchased demand management / load shedding. It could produce significant savings in an environment of no transmission system upgrades / load growth / high oil & gas prices. KEMA calculated an estimate for this savings of $21K/yr/substation for an incremental investment of $13K/yr/substation. KEMA recommends that National Grid monitor this issue for now. Overall, the estimated costs and benefits for a targeted ELSSI strategy that includes approximately 100 substations are benefits of $30M/yr from an investment total of $110M. Important Concepts of Scalability, Deployment, and Mapping of Benefits Strategies for exploiting ELSSI affect productivity, costs, and system performance differently and in different combinations. They also require different deployment strategies to be achievable and effective. For example, equipment utilization (increased or dynamic ratings) will have the following effects: Improved reliability by avoiding severe overload and by achieving faster/more comprehensive restoration via temporarily overloading adjacent equipment Lower congestion/purchased power costs as noted above National Grid 7 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

308 Attachment COMM 2-8(c) Page 12 of 103 Lower replacement/new capital acquisition costs due to deferrals resulting from increased loading. Arguably, staff productivity will not be improved in some cases, worker productivity may even decrease due to increased workload monitoring the utilization strategy. It would be possible to achieve ELSSI exploitation on a targeted basis. IEDs could be deployed at identified power transformers and cables to allow dynamic ratings, and integrated with the appropriate engineering and operational systems. Transformers where loadings were not an operational issue may not achieve significant ELSSI benefits. On the other hand, other exploitation strategies, especially productivity related ones, are not as easily scalable. Fault location is an example at one level: To get the benefits of fault location on crew dispatch, an entire station at a minimum must have ELSSI deployed. In fact, it would be desirable to have ELSSI deployed throughout a District so that dispatching procedures did not become dependent on whether or not a station had ELSSI. Relay maintenance and fault follow-up and clearance are an even less scalable example: To re-engineer their processes to take advantage of ELSSI, it would be desirable to have most or all stations in a region be ELSSI stations. The scalability issue impacts the timing of ELSSI exploitation strategies and the timing of expected benefits. A cost benefit analysis of a blanket T & D ELSSI strategy suggests that benefits are in the range of $180M/yr from an investment of $950M; i.e. this strategy is not attractive based on the benefits analyzed and the enormous financial resources that are required. National Grid 8 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

309 Attachment COMM 2-8(c) Page 13 of Introduction 1.1 Utility Evolution Part of the challenge with this type of project is that it needs to consider the past, present and future in terms of utility data use while also recognizing that while all utilities are fundamentally similar, they also have many differences. Differences also exist within single utilities in terms of how data is collected, stored, shared and used. In the case of National Grid, this situation is compounded by the existence of two distinctly different operating geographies and regulators that have led to some of these differences, as well as differences within each region. The following figures illustrate the existence of some of these differences and ways in which the organization can operate in a more uniform manner. National Grid has already spent four years working on many such problems following the merger of New England Electric System, Niagara Mohawk, and Eastern Utilities Associates. While the future expectations may be similar for how to operate across the electric utility enterprise, how to realize that vision is founded on the recognition of the gaps with current practices, systems, and infrastructure. These gaps will vary depending on many factors, and some of the gaps may remain in the future, though diminished in size. The following figure shows some of the fundamental components of all utilities. At this level, all utilities have similar responsibilities to fulfill whether it be gas, electricity, water, waste management or some other commodity that is being provided, or even removed as in the case of waste management. As a natural monopoly there are issues that all utilities face, and which force a superficial similarity upon them. Figure 1-1: All Utilities are Fundamentally the Same National Grid 1-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

310 Attachment COMM 2-8(c) Page 14 of 103 These similarities start to become less clear as we look into them in more detail, even for two apparently similar utilities. The following figure shows how the fundamentals are influenced by different factors. Even with this detail most utilities are the same in as much as they share many common goals, challenges and drivers and share many similar accountabilities. Figure 1-2: The Drivers for Utility Evolution Once you start to break down these factors into more detailed components, the evolution of each utility starts to change to reflect the nature of these components. For this reason it will not be completely possible to eliminate gaps between different utilities, or between utilities that have been subject to different drivers and challenges such as New England Electric System, Eastern Utility Associates and Niagara Mohawk. By aligning strategy across old operating boundaries, the gaps can begin to be closed but issues such as performance based rates, geography, customer demographics, regulation, rate cases, and the multitude of existing systems will mean that while it will be possible to evolve consistency in some areas, it will take longer in others. These components are responsible for many of the gaps identified in this report, but even within New England and New York there are differences that exist. These differences need to be identified so that these differences can be eliminated by applying consistent strategy. Even then, some differences will persist but their impact on operations will be diminished or negated. For instance, even with enterprise wide standardization on a single energy management system, RTUs and IEDs, the way the IEDs and RTUs are deployed will vary depending on substation design and that will vary based on demographics as will the need for the level of SCADA observability. By standardizing on strategy and device choices the differences will be minimized and so long as the strategy allows for adaptation to meet regional requirements, significant conformity can be achieved over time. National Grid 1-2 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

311 Attachment COMM 2-8(c) Page 15 of 103 This report focuses on identifying gaps in three different areas: Data (IEDs and SCADA facilities) Data Marts Communications This is the first of many steps required to address uniform practices with respect to data collection and management of operational and non-operational data to support National Grid in achieving its objectives with regards to reliability and cost management in a safe and responsible way. 1.2 Organization of Report This report titled Enterprise Level Substation System Integration Study Report represents the results of the first phase of a multi-phase approach to define the operational and non-operational data requirements, business needs, benefits, vision, conclusions and recommendations for implementation at National Grid. This work was performed during February and March 2005 by KEMA, Inc., Burlington, MA. Phase 1 included 48 interview sessions with approximately 120 National Grid employees in New England and New York. This engagement concluded with a period of results analysis, the development of this final report and a final presentation to National Grid s managers and executives. The main body of this report is structured as follows: Section Name Description Executive Summary A brief executive summary of the findings and recommendations is provided in this section. Section 1 Introduction The introduction provides a description of the organization of the final report, the methods deployed and the references used. Section 2 Issues and Opportunities This section contains an organized discussion of KEMA s findings in four broad categories; data mart, IED penetration, data and diversity, SCADA penetration and asset management. Section 3 Benefit Cost Analysis The benefit cost analysis is a high level, order of magnitude summary of benefits and estimated costs that are used to determine the economic justification for ELSSI and to prioritize the ELSSI applications for subsequent decision and implementation. National Grid 1-3 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

312 Attachment COMM 2-8(c) Page 16 of 103 Section Name Description Section 4 Conclusions, Vision and Recommendations This section contains a summary of the conclusions that KEMA has drawn from our work, a conceptual vision for the future and recommendations to achieve the vision. The main report is supported by a substantial amount of base material that was captured with the greatest potential for use in subsequent phases. All of the appendices are contained on CD-ROM. The structure of the appendices and the disposition of the contents of each section are as follows: Section Name Description Appendix A Completed Interview Templates for Each User Group This Appendix represents a deliverable as defined in KEMA s proposal. It contains two documents in Microsoft Word format. Section A.1 contains a complete listing of KEMA s interview notes organized by meeting while Section A.2 contains National Grid s edits to KEMA s notes for those meetings where National Grid provided feedback. Appendix A also contains a summary spreadsheet that outlines each meeting by code, name, date, time, location, KEMA s notes to National Grid and whether or not National Grid provided edits to KEMA s notes. Appendix B Questionnaires and Analysis of Questionnaire Responses This Appendix represents two deliverables in one document as defined in KEMA s proposal. The first part is the complete questionnaire document that was distributed to National Grid s candidate meeting attendees. The second deliverable, Analysis of Questionnaire Responses represents National Grid s actual responses to questionnaire prior to the interview meetings. Appendix C Interview Schedule This section contains the schedule of interviews as initially laid out for New England and New York. The schedules represent a required deliverable as defined in KEMA s proposal. Appendix D Appendix E Information Request Form Data Requirements Matrix This section contains the foundation for the high level benefit cost analysis. The first part of the section represents the base Information Request Form (IRF) while the second part of the section contains the form as completed and provided to KEMA by National Grid. The IRF is a deliverable as defined in KEMA s proposal. This section contains the Data Requirements Matrix as gathered from the interviews. This draft represents the first of three normal passes required to complete the matrix. National Grid 1-4 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

313 Attachment COMM 2-8(c) Page 17 of 103 Section Name Description Appendix F Priority Matrix for Data This section contains a matrix that summarizes priorities for data usage that is to be considered when implementing a data mart. Appendix G Initial Indications of Information Maturity A presentation that discusses information maturity at National Grid. This is a deliverable that helps to frame the issues surrounding meta modeling required to make a virtual data mart implementation successful. Appendix H Reference Material This section contains the base reference materials that were provided to KEMA by National Grid. 1.3 Methodology KEMA deployed a standard, but compressed nine-week version of a traditional four-month information gathering, analysis and recommendation effort to complete the work within National Grid s 2005 fiscal year. The project was supported by tasks in the following order: Project Kickoff Review Questionnaires Identify Potential Users of IED Operational and Non-operational Data Survey All Identified Users (Gap Analysis) Priority Assessment High Level Business Case Development 1.4 References A number of reference materials were provided by National Grid to use as guides through the development of this report. This foundation material included spreadsheet and text documents containing information on IEDs and their penetration within National Grid, IED data usage spreadsheets and substation designs including IEDs. Reference material is provided in Appendix H. National Grid 1-5 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

314 Attachment COMM 2-8(c) Page 18 of Issues and Opportunities for Enterprise Use of Operational and Non-Operational Data 2.1 Data Mart This section examines issues and opportunities surrounding the storing and sharing of enterprise level data, specifically the issues of ease of access to information and the existence of multiple independent sources for data that currently exist within the National Grid organization. While National Grid has focused significant investments in developing enterprise data marts, these have been in the areas of corporate systems such as Asset Management, Human Resources, Customer Service, Financials, etc. in a federated model that includes both individual and consolidated data marts. Data relating to the operational systems such as SCADA/EMS has not been an emphasis from a corporate data mart perspective. Although there are some systems that manage this data, they do not necessarily appear on enterprise IT system diagrams and have not been included in integration efforts to date. While the function of each system is suited to most of its users needs, their largely standalone disposition limits their ability to provide substantial enterprise benefits. The following matrix indicates partial data flows between some of the key systems. Down the side are data collection systems, while across the top are data dissemination systems. In the intersecting cells are the data that are passed between the two types of systems. From this it is possible to see the effects of replacing one system (e.g. ERS) with another that replaces all or only part of its functionality. It highlights the areas of risk that this might create and the need for coordinated migration strategies. National Grid 2-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

315 Attachment COMM 2-8(c) Page 19 of 103 Data Collection Systems PI New England EMS ASRS MV-90 All SCADA Data & Quality Codes; Alarms & Events Table 2-1: Data Collection and Dissemination New England Data Dissemination Systems New York IS-400 IDS Lodestar GIS PAIRS TOA ERS Internet GIS IDS PAIRS TOA NGRID One lines, Alarms & Events SOE & Alarm Data Customer Interruptio ns Manual Entries DFR Data, Digital Relays Wavewin (in the future) DFR Data, Digital Relays Revenue Metering Data Substation HMI Local IED Data Manual Entries Feeder Config. Data Outage and incident data DFR Data, Digital Relays Scheduled Outages New York EMS SCADA Data PowerOn MV-90 Revenue Metering Data Manual Entries Feeder Config. Data GEMS Gas Meter Readings Archived SCADA data DFR Data, Digital Relays (1) RTU with SOE Capability Customer Interruptio ns DFR Data, Digital Relays DFR Data, Digital Relays Outage and incident data Scheduled Outages National Grid 2-2 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

316 Attachment COMM 2-8(c) Page 20 of Existing Conditions Operations Data Limited access to operational data in both New England and New York. Access to current status of the electric or gas systems requires SCADA access. This may be required for alarms, information about tags, sequence of events, and short-term historical loads. While access should be limited on a needs basis, those that do need access should have an easy to use interface available. While access to this data for users with an operational requirement may have to continue to use existing solutions, access to this data by non-operational staff (i.e. those that require access to historical data) a centralized data mart would provide a better solution. Difficulty of data access for the occasional users due to the unique user interfaces, data models, and access to systems. This is because most systems were designed for specific user bases and to fulfill specific requirements. These systems often have excellent data but non-intuitive interfaces which make it difficult for occasional users to retrieve the data they need. In many cases people simply call known users of the system to request the data they need. Islands of information. A pattern of islands of information makes it difficult for departments to improve processes much less to improve cross-department processes. The technological islanding acts to maintain departmental silos and discourage cross department streamlining. Accuracy of information is as important as availability of information. If the quality of data is poor, users will check the data if they have concerns. This is wasted effort that should be addressed by improving data quality at source. If users are not aware of data quality problems poor data may be used based on the reasonable assumption that it is good. One area of confusion occurs where a data path becomes forked and users downstream of the different forks cannot validate their data since they are intrinsically different. Removing manual data paths and centralizing data with systems of record eliminates these problems in theory. In practice they will continue to exist but over time a user friendly and accessible central data mart should decrease these inconsistencies significantly. Faster diagnosis means faster restoration (improved reliability). The ability to connect key substations with control rooms and further provide meaningful data through the use of IEDs in a timely manner should provide the ability to diagnose and therefore restore National Grid 2-3 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

317 Attachment COMM 2-8(c) Page 21 of 103 many faults more quickly. This includes benefits not only in terms of hard performance measures but also in improved customer relations (and increased expectations) Revenue Metering Data Individual business processes for settlement data based on older proprietary technology. Both New England and New York use a DOS based version of MV90. This is due to be upgraded. For archiving of meter data, New England uses a Lodestar system and New York uses an in house system (ERS) based on an old Lodestar system. There are inconsistencies across organizations in data processing methods. Also, the validation, editing, and estimation (VEE) of data are performed within MV90. While MV90 has tools to perform these tasks they could also be performed downstream in a consolidated system. National Grid reports performance differently in New England and New York based on different regulatory requirements yet in Transmission a common set of performance measures are used to evaluate overall performance and create internally consistent methods. In the same way, although some estimation rules may be market specific, a centralized VEE capability would allow not only the application of consistent corporate standards where permitted but better auditability. Several departments expressed the concern that post-deregulation and the loss of direct metering data from generation, it is difficult for National Grid operations to observe its total native load directly (i.e. old paradigm of Load derived from Generation plus Interchange not supportable today). Issues around energy trading and FERC 2004 make this a sensitive issue that restricts the availability of this data Incident Data Maintained in separate databases with no automated method for correlating the data. Now that National Grid s Interruption and Disturbance System (IDS) is deployed in both New England and New York it is assumed that a mechanism has been implemented to perform data consolidation for analysis. Transmission is developing PAIRS2 (Phase 2 of the Performance And Incident Reporting System) to consolidate all transmission outage data in a single database. PAIRS2 will use IDS as a source of outage data. Lack of time synchronization is an issue that was raised in several questionnaire responses, representing a need for system-wide time synchronization of Sequence Of Event (SOE) data. This results in current actions to correct time stamp errors in data transmitted by relay IEDs and results in the inability to accurately diagnose root cause of problems and the associated labor that corresponds with attempts at diagnosis. Time synchronization should be a conceptual design issue. National Grid 2-4 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

318 Attachment COMM 2-8(c) Page 22 of 103 Consolidating and combining information from various sources is needed to help diagnose disturbances involving more complex protective relay operations, especially when relay misoperation is suspected. A centralized / virtual data mart would fulfill this requirement Enterprise Data Outage trending was a requirement expressed by numerous groups and in responses to the questionnaire. National Grid currently uses fault location data and other information to identify portions of the service territory where many outages of a given type have occurred. There is also an initiative to install IEDs to get more information on distance to fault for transmission lines with high numbers of momentary disturbances. This type of initiative will enable National Grid to target areas where a higher than average number of faults of a given type have occurred and allow outage trending/correlation with lightning and other types of data. Figure 2-1: Analysis of Data from Multiple Sources In the preceding figure the numeric labels indicate processes to correlate data from different sources to better understand the behavior of the system and represent analysis that National Grid already performs. The ability to analyze these types of data (and the sharing of the data) is something that should be rolled out to a wider audience: 1. Correlation of asset and fault data 2. Correlation of outage and weather data 3. Correlation of fault and weather data 4. Correlation of asset and outage data National Grid 2-5 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

319 Attachment COMM 2-8(c) Page 23 of Correlation of outage and fault data 6. Correlation of asset and weather data Even with separate data stores, it is possible to combine two sources together to look for correlations and related trends etc. National Grid currently performs analytical studies on various combinations of data (as represented above). An integrated virtual data mart for operations (field) data would permit multi-dimensional analysis to uncover more information about root cause etc. It is important to remember that these tools provide the ability to uncover areas for performance and reliability benefits that beyond those covered by normal utility operations (e.g. tree trimming, animal guards, grounding etc.). They are not a substitute for the basics. If activities such as tree trimming, animal guards, grounding etc. get cut back too far, the analysis is likely to reveal areas where those cutbacks have introduced new problems with performance Impacts to Business Users Operations Data Limited access to operational and non-operational data in both New England and New York partly due to limited IED deployment and partly due to lack of easy data access for occasional users. Faster diagnosis means faster restoration (improved reliability, SAIDI). Another issue raised more than once is that the accuracy of information is as important as availability of information. This is also discussed in Section Revenue Metering Data Data from interval meters is stored in a Lodestar system in New England and in ERS in New York. In both regions interval data is collected using Itron's MV90 system. National Grid is considering the implementation of a single platform for all energy data used by MDS, though this appears to be focused on time series interval data only. KEMA assumes that any reconciliation between retail and wholesale would still require the supply of retail load from CIS/CSS for dial meters. It was not clear from the interview sessions what format the meter data from large Gas customers is stored in and how the implementation of a meter data repository would impact the large Gas customer meter data that is currently stored in ERS, assuming that a new meter data repository would result in ERS being retired. National Grid 2-6 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

320 Attachment COMM 2-8(c) Page 24 of 103 The adoption of a single meter repository no doubt has potential benefits but the specification needs to be sufficiently well thought out that the incorporation of more input systems (e.g. non-mv90) can be easily accommodated to support rollout of the system in the event of any future mergers. ERS is shared by Meter Data Services (revenue meter data), Planning (historical electric SCADA data) and Gas (meter readings for large customers) Incident Data Consolidating and combining information from various sources is needed to help diagnose disturbances involving more complex protective relay operations, especially when relay misoperation is suspected. A small percentage of the IED data is collected using Wavewin. Wavewin itself is implemented at relatively few substations Enterprise Data The need for a common simplified User Interface for occasional users was cited in several questionnaire responses. The issue raised was too many systems with separate data access procedures. As a result of this it was felt that sharing of data is sometimes cumbersome because many people collect various data in separate documents (spreadsheets) but there is little consolidation of data. Consequently respondents felt that it is difficult to remember all the places to get data because there are so many. Individual districts sometimes collect data for use only in that region e.g. some inspection data. This type of data silo prevents easy comparisons of data and may result in a lack of synchronization (fidelity) of data between systems. Respondents felt that National Grid should aim for one system with browser access Known Initiatives Operations Data The licensing agreement for PI is being expanded to a corporate agreement covering both New England and New York. The number of licensed tags is also being expanded under the new agreement. A Gas Energy Management System (GEMS) replacement project will be initiated in the near future and an Electric EMS upgrade/replacement project is under consideration for New York and New England. National Grid 2-7 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

321 Attachment COMM 2-8(c) Page 25 of Revenue Metering Data A RFP document is being issued for a combined meter and settlements data repository system to replace both Lodestar and ERS. If the case for extending this repository to another region cannot be adequately made, the justification for proceeding has to be questioned. If a case can be made, however, it is important that this work is pushed ahead with Enterprise Data Cascade (New York) and EAP (New England) are being replaced with Maximo for Asset Management. The current plan is to implement Maximo 5.2 by May Handhelds will be used to upload into Maximo when docked. An Automation Study Team was created in In order to get input and direction from all stakeholders in National Grid s U.S. business, a Substation Integration Steering Committee was then formed that included Control Center Operations, Substation Operations and Maintenance, Substation Engineering & Design, Protection & Control Engineering, Relay Operations, and Distribution Planning. This proved to be a good forum for cross-company debate on what integration technologies should be used, and valuable in gaining stakeholder ownership of the design as it developed. The Substation Integration Committee, now called the Transmission & Distribution Technology Integration Steering Committee (TISC), has since expanded its scope beyond substation integration technology to include other similar technologies that can be applied to the transmission and distribution network, such as distribution automation. The deployment of integrated substation design (LPS03 Low Profile Secondary, 2003) to capture more operational data from IEDs for sharing within National Grid is an example of TISC in action Best Practices Corporate wide access to a centralized data mart maintained by IT, where IT provides user access tools, training, documentation, and access security. Note the data mart can consist of multiple component databases; however, user access is through a common user interface supported by IT-developed tools (e.g., data query templates based on application needs). Every organization has many databases. Many of these databases support necessary niche business solutions, which provide critical functionality within the organization. These specialized systems often provide company specific solutions to business challenges that National Grid 2-8 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

322 Attachment COMM 2-8(c) Page 26 of 103 represent a competitive advantage or differentiate the organizations business style and this diversity has to be managed. There should only be one owner of each type of data and if each application needs a consistent view and we have a single owner, that owner ought to be able to provide data as a service to those that need to use it. To have a consistent view of data you need reference standards that provide guidelines and standards to provide a common frame of reference for data and terminology. Table 2-1 shows a matrix of systems that broadly categorizes various systems as either collecting or disseminating data. In terms of approaching the design of a data mart for sharing enterprise-level data this figure provides a good starting point for identifying transmission and distribution data (including Gas) sources that represent the dynamic state of the electrical and gas networks over time. The following figure shows that this virtual data mart would include mechanisms to use data in other systems such as PI, Wavewin, and any new repository for meter data. Additional provision would be required to manage elements of the data not provided for by these other systems and to manage user access, permissions, custom reports etc. (represented by ELSSI Data in the figure). Figure 2-2: Data Mart Vision Virtual data marts are ideally suited for enterprise applications such as ELSSI that need to bring in data from multiple sources. National Grid 2-9 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

323 Attachment COMM 2-8(c) Page 27 of 103 A virtual data mart would utilize middleware to pull data from native repositories scattered around the enterprise and present it in a single view, as though it were all in one database. It also provides the additional benefit of being a one stop shop. So long as the user is looking for data related to metering or system operation, they do not need a list of five or ten applications and specialized training to search for the required data. Data can be retrieved by going to a single (virtual) data mart. This provides a single contextual view for enterprise users to view substation (and related) data that is normally dispersed. The difference is that duplication of data is reduced or eliminated, and no additional permanent repository needs to be developed except to cover gaps in data storage (if necessary) and ELSSI specific information pertaining to users, reports etc. By not duplicating and copying information, a virtual data mart also helps solve data ownership concerns by making sure the data is always under the control of the original data owner although the overall concept and ELSSI specific data need to be owned by a specific business area. A Service-Oriented Architecture (SOA) is a component model that inter-relates the different functional units (called services) of an application such as the virtual data mart discussed here through well-defined interfaces and contracts (see later in this section) between these services. The interface is defined in a neutral manner that allows services, built on a variety of such systems, to interact with each other in a uniform and universal manner. The benefit of such a loosely-coupled system lies in its agility and the ability to survive evolutionary changes in the structure and implementation of the internals of each service, that make up the whole application. These changes might be based upon the needs of the business to adapt to changing policies, business strengths, business focus, mergers and acquisitions, industry standing, and other factors that influence the nature of the business. Service-oriented architectures are not new, but are an alternative model to the more traditionally tightly coupled object-oriented models that have emerged in the past. SOA itself is an abstract concept of how software should be put together. It relies on the more concrete ideas and technologies implemented in extensible Markup Language (XML) and Web services using Web Services Definition Language (WSDL), to exist in software form. The SOA concept does not exactly define how services specifically interact, just how services can understand each other and how they can interact. Web services have specific guidelines on how messaging between services needs to interact; that is, it is the tactical implementation of an SOA model most commonly seen in Simple Object Access Protocol (SOAP) messages delivered over HyperText Transfer Protocol (HTTP). By describing interfaces in WSDL, services have moved to a more dynamic and flexible interface system than older interface definitions. It is required to define how the overall application performs its workflow between services and it is necessary to find the transformation point between the operations of the business versus the operations of the software used in the business. Thus, an SOA is able to relate the commercial processes of a business to its technical processes, and map the workflow relationships National Grid 2-10 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

324 Attachment COMM 2-8(c) Page 28 of 103 between the two. To be effective, it is necessary to define the policies of how relationships between services should transpire, often in the form of service-level agreements and operational policies. These contracts then facilitate the inter-relation of the services (in the cloud in Figure 2-2). This is the same cloud shown in Figure 2-3 which illustrates some of the potential user groups that might benefit form a virtual data mart. Figure 2-3: User Access to Data Mart Its function and/or audience are what define a data mart. A data mart is an implementation of a data warehouse with a restricted scope of content and support for analytical processing, usually serving a specific function or business area; it could also be across multiple business process and multiple departments if they happen to have similar reporting requirements or a particular data analysis problem domain such as enterprise level substation data integration. A data mart to support information about the electric and gas transmission and distribution systems will need to support several fundamental types of business application requirements. This section discusses those elements and proposes a solution for how to accommodate the diverse and potentially conflicting requirements. National Grid 2-11 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

325 Attachment COMM 2-8(c) Page 29 of 103 Figure 2-4: Data Mart Action Pyramid The preceding figure depicts the path of data (in blue, on the left) from its initial acquisition (at the top), through immediate or operational use, through storing for delayed use and historical analysis, through to integration and sharing of data at the enterprise level (at the bottom). It also shows the type of actions required (in green, on the right) at each stage in the data path. The following table summarizes the types of actions required at each stage: Table 2-2: High Level Data Action Summary Data Path Systems Action Description Acquire SCADA, MV90, Wavewin Improve EMS and MV90 both need improving to keep pace with technology. Use Store EMS, GEMS, MV90, IDS, PI, TOA PI, ERS, TOA, PAIRS, Wavewin, IDS Improve Update As above but also IDS needs improvements to satisfy both New England and New York. TOA needs to integrate with Primavera and provide easier access to scheduled Transmission outage data. PI is to be rolled out into New York. The operational systems here both acquire and provide functionality to use the data. Systems need updating on an evolutionary basis. Share Data Mart Design Requirements need to be developed from both functional and non-functional perspectives. From a non-functional perspective, involving National Grid IT department is a key requirement. National Grid 2-12 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

326 Attachment COMM 2-8(c) Page 30 of Gaps and Priorities Operations Data Historical SCADA Data: Access to historical SCADA data is required for many reasons. In New England this data is stored in PI. PI is a good choice for archiving point data but is not designed to support non-operational data. In New York, historical SCADA data is available within EMS for 28 days at scan rate, and thereafter is available in ERS summarized as average values over 15 minute intervals. When PI is rolled out into New York it is assumed that SCADA data will be archived at scan rates. PI is not a very userfriendly system and some mechanism needs to be provided to access historical SCADA data for infrequent users. Infrequent users of systems typically call up someone more familiar with the system to get the required data. This was an issue encountered several times during the interviews. ERS was not mentioned in this respect but IS400 was described as taking a long time for the application to open and PI was cited for cryptic point names Revenue Metering Data The initiative to develop a centralized repository for combined revenue metering and settlements data is a positive step in normalizing the procedures for data access security, maintaining audit trails and version controls despite the different business settlement rules. The new system should provide enhanced capabilities for aggregating, sorting, and filtering the metering data based on user requirements. Data drill down capabilities should also be enhanced for identifying and correcting data variances. This system should be considered as a critical component for an enterpriselevel data mart Incident Data Native IED Data Formats: The use of native IED data formats is required to preserve data integrity for complex data types such as oscillography and other file type data structures that may be associated with manufacturer provided diagnostic tools. Preserving the integrity of the data format allows the use of these tools when analyzing the data Enterprise Data Single Point Of Reference: During the interview and questionnaire processes, many participants stated that they needed more access to data. Many also stated that any new system should be intuitive to use and that there was not enough time to keep up to date with all the systems they used. Yet despite the requirements for more access to data it National Grid 2-13 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

327 Attachment COMM 2-8(c) Page 31 of 103 was also stated many times that there were too many current systems used to collect data. Due to these potentially conflicting requirements KEMA recommends that National Grid consider implementing a virtual data warehouse (see Figure 2-2). Different Levels Of User Access: At each level in Figure 4-1 different types of users will need access to different data. Also as you move down the pyramid access to data, and the information created from it, will be required by more diverse user bases. Different users will also require different ways to analyze similar data. Some may want to perform trending and analysis on aggregated data while others may want to look at detailed data. Also, those users looking at aggregated data may have different aggregation requirements in terms of the data that is being aggregated or the time periods for data to be analyzed and/or aggregated over. Access to data will also have to be restricted and perhaps constrained by factors such as geography, market area (ISO), and user base (FERC 2004 restrictions) and internal/external user considerations. An additional consideration is where to perform the aggregation of data for analysis. Without losing access to historical records in the IEDs themselves the solution is to keep both detailed and aggregated data in a data mart for analysis. The issue then becomes what level of denormalization is applied to the data within the storage schema. By keeping multiple levels of data within the data mart, National Grid can modify the aggregation approaches over time and the trade off becomes the effort needed during Extract Transform Load (ETL) procedures or batch aggregation routines versus the impact of ad-hoc aggregation. So long as both methods are supported the approach can be modified and the best approach will be determined by careful analysis of user habits once the data mart is implemented and once the users have become familiar with its capabilities and the data within it. Operational and Non-Operational Data: The data mart will need to provide access to nonoperational data such as waveforms etc. that are stored in their native format, and operational point/sequence type data. The operational data may be collected through points defined in SCADA or may be retrieved by some other system such as Wavewin. Meter Reading Data: A central repository allows for a single point of validation and provides consistency of approach. It also creates a single point for archival of vital data, required for Sarbanes-Oxley compliance, and creates a single system of record for financially important data. National Grid has prepared a draft RFP to create a new repository for meter data, which was provided to KEMA for analysis. It is assumed that the sentiments listed here are echoed in this RFP. A repository for meter data should National Grid 2-14 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

328 Attachment COMM 2-8(c) Page 32 of 103 support multiple interval sizes, dial meter readings, multiple units of measure, multiple commodity types and many more requirements than can be listed here. A central repository of meter data (and other) allows analysis to take a holistic view and allows comparison between regions using for example non-market specific common performance indicators. Multiple Commodity Types: It is critical to consider generalities, possible future business changes, and changes over time when designing a data mart. For any data associated with a type, it is suggested that an extra other type always be provided for, with a data driven flag for identifying the type. At a minimum it provides for extensibility of the model for unanticipated changes. Currently, SCADA and metering values are managed for both gas and electricity. It cannot be ruled out that other commodity types might be required in the future, for instance water, steam etc. Also, since ERS and other systems are used for providing support for rate cases and other planning activities it is important to consider the association of rules or algorithms over time. This is also applicable for metering etc. where rules for filling data gaps, estimating load from profiles etc. may change over time and it is necessary to be able to find the correct rules from a temporal perspective (i.e., version controls) when performing regression testing or rerunning old settlement data. The data accessed through a data mart such as described here will potentially be provided and used by many different departments. This creates an ownership issue that needs to be resolved before the data mart is developed. Since it crosses many areas and since it has enterprise benefits it is recommended that National Grid s IT department take a significant role in this work. (See Recommendation 1, page 1.) 2.2 IED Penetration In 1998, the first-generation integrated substation standard was completed and since that year, 22 new distribution substations were designed to the standard, including four completed in 2003 in record time to meet summer load relief. In 2000, elements of the distribution standard were used in the design of two transmission substations that were being refurbished. With the technology foundation established and successes realized, plans for a more ambitious second-generation standard were initiated in The second-generation design is based on a breaker-and-a-half station, which typically is employed in areas of high load density in National Grid s U.S. service territory. This design has been implemented in two distribution substations in New England and there are three more in progress. Some of the techniques of the new design are also being incorporated in five Transmission substations that are now in the design stage, including one in New York at Clay station. National Grid 2-15 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

329 Attachment COMM 2-8(c) Page 33 of 103 National Grid is also working on an integrated station design that could be deployed in areas of low load density in National Grid s U.S. service territory, which typically use a straight bus substation primary configuration in New York. The straight bus configuration does not require as much control logic as the breaker-and-a-half design, so location of the control logic elements may be moved from one device to another, for example, from the PLC to the relays themselves. With respect to types of IEDs, National Grid has implemented a limited set for a variety of purposes with primary focus on protective relays, reclosers and voltage regulation, but includes meters and equipment condition monitoring (mostly transformer monitors) Existing Conditions IED Penetration The percentage of IED penetration across the corporation is approximately 4% of protective devices on the transmission system and 1% for the distribution system. The balance of National Grid s protective devices are electromechanical and solid-state electronic devices (not IEDs). Within New England there are approximately 2500 pole mounted reclosers with embedded IED-type controllers. They operate based upon local conditions and are therefore standalone. New York has instituted a recloser installation program that resulted in approximately 200 installations on worst performing feeders (at the time). Today, the program still exists but has effectively stalled due to funding cuts. Today, automated reclosers are installed only as equipment is replaced or new stations are added. There are 23 Digital Fault Recorders (DFRs) employed which track data prior to and following faults in the form of oscillographic waveforms. The majority of DFRs are implemented at bulk power (230kV and above) stations in New England. While in New York where there are fewer than 10, there is interest in increasing the number for greater penetration to more quickly and accurately identify fault cause and location. There are no funded programs to implement new IEDs with the exception of replacements due to equipment failure or construction of new substations (e.g. LPS03, and at construction of new Transmission substations). There are 22 existing LPS98 substations and 2 LPS03 substations with six LPS03 substations scheduled for the 2006 calendar year. National Grid 2-16 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

330 Attachment COMM 2-8(c) Page 34 of Communications The majority of functioning IEDs installed across National Grid support protective relays and metering, and they are accessible via dialup telephone lines. Existing dialup telephone facilities were referred to as unreliable in some interview sessions and operate at speeds of 9600bps (digital) or 2400bps making it difficult to access IEDs for purposes of obtaining large data files. This also has the effect of tying up the line and making it unavailable for any other users. While 9600bps is functional for operational data transfer, this rate is insufficient for more than one function, two-way communications, and nonoperational data transfers from the substation to the corporate environment. One hundred of the line reclosers are Cooper Form 6 and are connected using the Telemetric cellular telephone service where operational data can be viewed on the Telemetric website. Approximately 80 reclosers are accessible via 900Mhz MAS radio. While MAS radio is not used extensively for the electric business, it is used to support IED communications in gate and regulator stations for the gas distribution system. Dialup communications fall outside of the realm of the NERC CIP Standards, unless they connect to Critical Cyber Assets as defined in NERC Standard CIP-002, in which case they need to implement some form of protection on the dialup access point. The VeriLink devices used in New York (see Section on page 2-33) look like they qualify as firewalls under the NERC CIP Standards though National Grid are currently working with Verilink to implement firewalls in these devices (reference: interview session 43). Gas distribution falls outside of IT Cyber Security s area of responsibility Data Acquisition, Storage and Applications The Wavewin application polls the DFRs, receives event data from them and stores it locally for analysis. While most DFRs are accessible by dial up phone lines with data uploaded to remote PCs, they can also be accessed on site via physical connections. Data from those DFRs that are not connected to the communications network require site visits and physical connection to laptops in order to obtain data. Some equipment condition monitoring IEDs (Hydran and Qualitrol transformer units) have been installed and monitor condition data continuously while the data goes to individual users PCs and on a shared drive, but not to a shared repository. When an alarm is activated, the alarm indication may be collected by SCADA but not the actual data point. Some of the monitors are not connected to any communications facilities. National Grid 2-17 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

331 Attachment COMM 2-8(c) Page 35 of 103 In locations where IEDs do not exist, time-consuming site visits are mandatory. At sites with IEDs, poor communications frequently prevent remote access to them to obtain data. This circumstance necessitates site visits to physically connect computers to IEDs to download the data. However, due to the remote nature of many of the stations, they are not always visited on a routine basis. At the same time, data continues to be loaded into the buffers of those IEDs. The combination of infrequent site visits with data that is continually loaded into the storage buffers of the IEDs results in lost data because the buffers fill and new data overwrites older data. The result is valuable line performance information that is lost. Similarly, when dial-up data comes back to an individual PC or a shared server environment, it cannot be shared easily. More importantly, its use cannot be controlled and it cannot be updated on a regular basis. Current maintenance practices for protective relays and other station equipment is primarily time-based, whether the equipment needs it or not. Bulk station intervals are prescribed by NPCC. Upon receiving time prescribed maintenance, technicians capture the results manually on log sheets at the stations, followed by manual key entry into AIMMS by office personnel. These data are used to diagnose patterns and problems to trigger maintenance activities. The net result is only a snapshot of a condition that is often aged. Transformer oil samples are drawn manually into bottles and sent to a laboratory to test for impurities and dissolved gas content, which are signs of impending failure or need for maintenance. The net is result is only a snapshot of a condition that is often aged Impacts on Business Users Communications Low-speed, unstable communications result in hard copy report transfers; phone calls between engineers, technicians and operations, and site visits for problem analysis, and periodic data uploads through physical connections. Low-speed lines are not sufficiently stable and fast enough to reliably move data from the IED to its destination in the corporate environment. The destination within the corporate environment is normally local PCs or data shared on shared drives. Rather than direct wiring I/O to RTUs, IEDs alleviate the need for special I/O cards and hard wiring as long as all the equipment being monitored are microprocessor based. National Grid is still installing RTUs. If there is even one electromechanical device that National Grid 2-18 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

332 Attachment COMM 2-8(c) Page 36 of 103 needs to provide data to an RTU, the remote I/O is still needed. What is needed is a communication processor that can act as an RTU (e.g. NovaTech Orion) and National Grid has not yet qualified an RTU with comm. processor functionality yet Data Acquisition, Storage and Applications The ability to access protective relays and DFRs enables measurement of problems soon (seconds or minutes) after they occur versus time consuming site visits. For example, when fault location information is readily available to engineers, operators and technicians, they can detect outage locations so that more immediate and accurate corrections can be made. IEDs are capable of supplying information electronically that is not available from other sources. (e.g. fault location, circuit breaker contact wear indicator (I2t)). By supplying data electronically it can be transferred and stored in a centralized data repository that is accessible throughout the enterprise. Equipment condition monitoring IEDs will supply information that is useful in determining early warning signs of impending failures or results in delaying maintenance to avoid unnecessary costs. By storing IED-generated (operational and non-operational) data in one location, it will be accessible by anyone who has authorization to access it. Combined with maintenance information from AIMMS, it can provide valuable insight into overall asset conditions and condition-based maintenance. Meter IEDs will capture power quality data that can be used to detect problems such as sags, swells, spikes and flickering lights. Availability of this data will provide a means for National Grid to be proactive against customer complaints. Data gathered from IEDs is stored locally on PCs or on shared drives and therefore is not easily shared. Since the data is not centrally maintained use of the data cannot be controlled (except through access to the shared drive), and it cannot be updated routinely or confidently Known Initiatives The Feeder Automation Team is exploring the expansion of Telemetrics as well as other distribution automation options to expand the penetration of feeder automation. Part of the expansion is consideration of a connection between the Telemetric website and EMS so that feeder information can be monitored from the EMS (and ultimately via the data warehouse). National Grid 2-19 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

333 Attachment COMM 2-8(c) Page 37 of 103 Wavewin has been implemented on a limited basis to automatically obtain information from DFRs and microprocessor relays. The idea is to bring back fault event data by polling over a phone line on a timed basis without any manual intervention. This should be expanded to get data from other substations and different types of IEDs (i.e., equipment condition monitors). (See Recommendation 6, page 2.) There is an initiative to correlate lightning strike data with power system outages in order to pinpoint areas that are especially prone to this type of outage with the intent of reducing outages Best Practices IED Penetration IEDs, particularly relays, reclosers and condition monitors, are implemented in new and existing stations through funded programs at steady or fast rates. IED manufacturers and models are standardized. Standardization provides an operating and maintenance environment that is familiar to engineers, operators and technicians. Similarly, interfaces to information systems are standardized and easier to maintain, operating conditions can be standardized and schemes can be implemented easier, as well as updated HMIs. Finally, operational and non-operational data formats are standardized to simplify storage, maintenance and analysis requirements. Protective relay IEDs are capable of providing oscillographic data and maintenancerelated information such as I 2 t at locations where DFRs are not available Communications A best practice is continuous monitoring of devices in the field and bringing data back to a central repository where it can be viewed and analyzed on demand. On demand is based upon limited time delays in transferring data from IEDs to both EMS (operational) and the data warehouse (operational and non-operational). The primary benefit to operations, maintenance and reliability is to integrate IEDs so that they monitor and deliver data continuously to both EMS (operational) and a data mart. Data will no longer be lost at the station, there is relief on the need to manually collect data and to ensure that the data is available on demand. Data will not be overwritten, a continuous stream of data will be provided and it will be stored in a central location accessible to authorized personnel. National Grid 2-20 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

334 Attachment COMM 2-8(c) Page 38 of Gaps Best practice for substation communications is network based communications with a minimum data throughput rate of 56kbps. Routable protocols, such as DNPi (DNP3 over TCP/IP) are commonly used to handle substation data acquisition requirements. With substations there is one (or more) physical LAN, which is connected to the corporate WAN using a router (demarcation point). Protection from unauthorized access to the Corporate WAN and the station LAN is provided by configuring firewalls within NERC CIP Standards criteria. Consistent with the LPS03 design objectives, standard model types should be employed for IED equipment to simplify designs, operations, data formats, communication protocols, interface formats and maintenance for retrofits. Even if the substation design needs to be changed based on local conditions etc. the types of IED to be used and data to be retrieved from them should be standardized. The concept embodied by LPS98 and LPS03 is a good one and a design cycle of 5-6 years feels right from a subjective analysis perspective. Data concentrators should be employed as a key part of IED strategy and communication strategy to optimize communication and data traffic. There are several ways to interpret what is meant by data gap. We not only have to consider where data is not being collected, but also where it is being collected but not used. From the perspective of the potential end user the reason why the data is not available is moot a gap still exists. The following diagram is taken from National Grid s RFP that was issued for this project and shows a typical one-line for a substation with a breaker and a half scheme. This type of scheme is prevalent in New England. National Grid 2-21 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

335 Attachment COMM 2-8(c) Page 39 of 103 Figure 2-5: Example Substation One-Line Diagram Line1 Line2 #1 TR 24/32/40 MVA 115/13.2kV #2 TR 24/32/40 MVA 115/13.2kV 4A1 4A2 N.O. 4A3 4B1 L1 4B2 L2 4B kv #1 Bus L3 N.O. L4 4C1 4C2 4C kv #2 Bus N.O. 4E1 L5 L6 4E3 #1 CAP BANK #2 CAP BANK Hydran and Qualitrol transformer monitors are capable of providing temperature information. The LPS03 design does not include an interface for Qualitrol temperature monitors and that data is not being presented in the HMI. Information from Hydrans goes to individual user s PCs via a telephone dialup mechanism and can be copied to a shared drive but it does not go into SCADA or into PI. Since continuous monitoring is not provided, alarms detected by the sensors will not be detected automatically. Qualitrol data is collected on site except for one or two sites, which have telephone dial up capability. Improvements have been made on LPS03 for transformer temperature monitoring for trending IED Penetration 96% of the transmission relays are not IED type devices and 99% of the distribution relays are not IEDs. IEDs that are installed should be connected to the communications infrastructure. The types of IEDs installed should go beyond protective relays to include DFRs, equipment condition monitoring devices, and LTC/regulator controllers. Other IEDs for equipment condition monitoring, such as substation battery monitors and bushing monitors, may be added in the future. The pace at which IEDs should be installed should be increased based upon the results the analysis in Section National Grid 2-22 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

336 Attachment COMM 2-8(c) Page 40 of 103 Engineering and Maintenance: Information is currently being gathered manually and instead could be uploaded continuously without the need for manual data entry Communications Communications infrastructure needs to be fast, reliable and secure. The minimum speed should be 56Kbps. There needs to be multiple logical channels for different types of data; operational, non-operational and remote access. A network-based solution is recommended. Communications should be extended to include certain devices on feeders such as regulators, reclosers and capacitor banks Data Acquisition, Storage and Applications Operational and non-operational data needs to be gathered on a continuous basis and stored in a location where it is readily accessible to authorized users. Operational and non-operational data combined with maintenance information will offer greater potential for condition-based maintenance. Additionally, by avoiding storage on individual PCs in favor of a centralized data store, there will be more information available for more and better uses. Therefore, the storage of these data should be standardized, centralized and accessible to authorized personnel. Outage analysis initiative should be extended to correlate other trends to outages. The Wavewin concept should be extended to include maintenance type devices and not just fault event data. 2.3 Data and Diversity This section examines issues and opportunities surrounding several areas where diversity impacts the way that work is performed and the ways that data are collected, stored and shared within the National Grid organization. Please refer to the sections on Consolidation and Sharing Data, and IED Deployment Issues & Opportunities for additional related discussion Existing Conditions Operations New England (electric), New York (electric and gas distribution) use three different SCADA systems. This diversity of energy management systems is not unexpected since Gas and Electricity have different requirements and the two EMSs being used on the electric side are due to historical differences between New England and New York that pre-date the merger. The EMS in New England has already been standardized between National Grid 2-23 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

337 Attachment COMM 2-8(c) Page 41 of 103 the former NEES and EUA companies to a common system and database that now manages what were previously two different energy systems (with two different SCADA technologies). On the electric side, whether a common database or separate databases are preferred, the path to bring both New England and New York to a common technology platform should be pursued. (See Recommendation 9, page 3). Given the differences that existed historically, as described above, the use of separate systems for archiving SCADA data is not unexpected. In New England SCADA data is archived to PI (at scan rate) and occurs in near real time (within a few seconds). In New York, scan rate data is available from EMS for 28 days after which the data is archived to ERS. When the data is archived to ERS it is transformed into 15-minute average values. Gas meter readings from large customers are also stored in ERS but Gas SCADA is archived to a separate database (with web access). When new points are added to the New York EMS it is a manual process to add the point to ERS. If this is not coordinated ERS will reject data from the new point until it has been set up in ERS. Also, the EMS data has to be shifted by one minute to fit ERS due to differences when intervals start and finish in each of the systems. In New England there is a function that updates PI automatically when new points are added to EMS. Since existing Planning applications in New York may be designed specifically for using 15 minute averaged data intervals, the effect of replacing ERS with a meter data repository and PI may have unforeseen implications that should be assessed. Non operational data is retrieved and stored in Wavewin or is collected manually on site or by dialup from individual PCs. Wavewin has the capability of communicating with multiple IEDs to extract data using proprietary commands specific to each type of IED. It is anticipated that Wavewin will be relied upon more and more to collect data from IEDs. (See Recommendation 6, page 2). Observability varies by control center, as discussed during interview sessions. These figures (also discussed in Section ) were verbal estimates of observability provided to KEMA and ranged from 35% in New York Western Region to 85% in New York Central Region. New England and New York Eastern Region were both similar (in the 50% - 60% range). The variation of observability for New York was unusual but as per NERC Policy 9D the observability required is dependent on the specific system and its operating criteria. Given that operating criteria were stated as being consistent across all three New York control centers, the question has to be raised as to whether the estimated figures are accurate or whether the nature of the system varies significantly by region. If no significant variation exists then the question needs to be asked as to whether the observability in New York Western Region needs to be increased. National Grid 2-24 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

338 Attachment COMM 2-8(c) Page 42 of Applications Systems General One issue that was raised in several questionnaire responses and interview sessions was the issue of needing to access many different systems to get the desired data. With infrequent use of these systems, staff find it difficult to effectively navigate these systems in search of data. Many engineering systems are designed for very specific uses and provide tremendous value to those people that rely on them day to day. These systems are developed around specific requirements and generally meet them very well. For people who require infrequent access to many systems it is a challenge to stay current in all of the systems. To be familiar with two to three systems in detail is more a manageable target but limits availability of data to that stored in those systems. A centralized/virtual data mart (as discussed in Section 2.1.4) provides the opportunity for engineers to potentially focus their application familiarity on fewer systems where they have a need for specialized functionality and provides a single point to retrieve all other data needed for analysis. This would meet the requirement for more data, also raised in several questionnaire responses and interviews. A virtual data mart, if properly designed and implemented, will address the paradox of needing a new system yet having too many systems already. Another issue raised by multiple respondents is that any new system should be intuitive and easy to use. Meeting this requirement is vital if a new data mart is to help provide greater access to data while facilitating more in depth familiarity with key systems. Part of the challenge with identifying differences is recognizing that differences exist in terms of how data is collected, stored, shared and used. In the case of National Grid, this situation is compounded by the existence of two distinctly different operating geographies and regulators that have led to some of these differences, as well as differences within each region. Thus there are many areas where diversity of processes exist and have been driven by: Different rate cases. Different regulators. Different funding. Different markets in New York and New England. Different geography. National Grid 2-25 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

339 Attachment COMM 2-8(c) Page 43 of 103 Different substation design. Etc Impacts to Business Users Operations Differences between regions in New York were apparent both in terms of equipment, procedures and terminology. Alarm information provided by SESCOA units are not available for New York Eastern region when control is transferred to Central Region overnight. There were also references made to confusion over different terminology when dispatching traveling operators in Eastern Region from Central Region control in New York. Slow rollout of IEDs creates a mixed base of electromechanical and multiple varieties/generations of IEDs to be supported and used Applications Systems Two other issues that were expressed were that data quality needs to be improved and that data cannot always be trusted. These issues mask the fact that the majority of current data is acceptable, however the comments raised may be indicative of infrastructure issues, business process issues or other issues. Without more analysis of specific instances of these types of comments there is not much that can be stated here other than it should be a goal of National Grid to identify and fix the quality of data by addressing the root cause. Rather than constantly patching data collected by a flawed process, the process itself should be fixed. Where patches to data are unavoidable they should be performed as close to source as possible to avoid the same (or worse, different) edits being performed downstream by different people or departments, leading to data inconsistency and quality issues. In cases where it is a business decision to perform validation and correction of data downstream (e.g. the new meter data repository proposed by MDS), it is important to control (restrict) access to data upstream of these changes. KEMA has seen this type of downstream VEE implemented previously based on justification similar to that in National Grid s RFP, and also where the scale of VEE was better suited to a back end system in order to prevent the VEE requirements from overwhelming the upstream data collection system. National Grid 2-26 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

340 Attachment COMM 2-8(c) Page 44 of Enterprise A virtual/centralized data warehouse could potentially create the ability for the development of management or executive dashboard capabilities for the purpose of providing a view of current or recent operational status, in addition to any current monthly, weekly, daily reports already being provided (e.g. from ERP) so long as the benefit is clear. There is no need to share information where it does not provide benefits. The downside of sharing more information is that it takes more time to digest. This leaves less time for action. If the new information is replacing older information sources, provides educational value, or allows more effective actions to be taken then sharing that information provides clear benefits Known Initiatives Operations New or upgraded capabilities for Energy Management Systems are being considered for New England and New York Electric systems (discussed above) and a new Gas SCADA system is also being considered. See Section 2.4 for more information on SCADA Applications Systems Enterprise A single system to store historical SCADA data does not exist. However, the current plan is for PI to ultimately perform that role (discussed above). A single system to store meter data does not exist. An RFP is in the process of being written and whose scope is planned to address this issue (discussed above). The overall objective of National Grid s integrated substation design efforts has been to reduce the total life-cycle cost of the secondary systems within substations while increasing their functionality. However, as the number of substations deployed with this integrated technology has increased, and substation information is becoming more readily available, attention needs to be given to the way that the available information is managed. Opportunities to mine this information using software agent technology are becoming increasingly available which will enable applications to be developed to use the data and to improve operations around the enterprise. Abilities to more quickly and more accurate diagnose faults for protection engineers and operations engineers, as well as providing smarter analysis tools for asset managers are possible hence National Grid s Enterprise Level Substation System Integration project. National Grid 2-27 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

341 Attachment COMM 2-8(c) Page 45 of General Revenue metering is being added between Transmission and Distribution in New York. This is not a separate initiative but is being performed as other projects are done, with metering being added to the scope of work where possible Best Practices Standardize on a set of IEDs for use across all regions. Become familiar with their use and define standard data to be retrieved from each device type. Perform a fact modeling exercise to develop a common terminology for the electric business. A fact model focuses on how the knowledge underlying business operations are organized such that it indicates what you need to know in order to do what you do. This tells you how to structure your knowledge about business process based on a standard vocabulary. An additional benefit of creating a focus on structuring how people think and communicate about business processes is that it helps to clarify alternative ideas about how business capacity itself can be best structured to satisfy business goals. A fact model is a blueprint for basic business knowledge. This should be a key step in implementing Recommendation 1, page Gaps and Priorities Operations IED training for relay technicians must not be overlooked. There is a risk of a scarcity of trained personnel as more LPS03 stations are implemented if training does not keep pace with implementation. This also requires a role change for Supervisors who have not come up through the ranks working with this type of equipment. If National Grid replaces 5 substations per year and generates a new standard design every 5 years, there will be over 50 standard designs by the time stations have been fitted with IEDs. This creates potential safety, maintenance, training, and supply chain issues (spares). This is because such a slow rate of deployment could result in the use many different IED types but with relatively few of each type being used. If the life of an IED is 15 years, the replacement rate will need to grow by 5 a year every 15 years just to maintain the same rate of replacement. These figures are meant to provide the reader with a point of reference. IED life may be longer (or shorter). The model can be adjusted to reflect changes in assumptions but the fact remains that slower rollouts will cause issues. Even at 5 new stations per year, after National Grid 2-28 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

342 Attachment COMM 2-8(c) Page 46 of cycles of installations it will be necessary to be refitting 90 substations per year at the end of this period. At 5 new stations per year it will take 260 years (theoretical but unrealistic) to deploy IEDs to all substations. In doing so it will be necessary to perform a total of 11,480 station refits due to the need for continuing refurbishment as IEDs expire. Any rollout that lasts longer than the life of an IED will incur compound rollout costs, not unlike paying compound interest on a credit card bill. At 5 new stations per year, with an IED life of 15 years the first stations will get refitted 18 times before the last one is completed. The fact is that 5 new stations per year is not even the equivalent of making minimum payment using the credit card analogy. Another way to approach the problem is to assume that electromechanical devices will not be available for too long but will still get fitted for another 10 years (probably longer in reality) and have a lifespan of up to 40 years. You end up at a replacement rate of approximately 25 substations per year, for a replacement schedule of approximately 50 (10+40) years but it is necessary to consider if it is desirable to be installing old devices for the next 10 years and still maintaining this type of device for another 40 years afterwards. Yet another way to back into a replacement rate is to assume that electromechanical device support may not be unavailable (to a greater extent) in 35 years. That makes the base replacement rate 37 stations per year. The following figure shows the number of years to fit IEDs in all substations (red line on chart, left vertical axis scale) and the total number of substations that would end up getting refurbished allowing for multiple (repeated every 15 years) refurbishments (blue line, right vertical axis scale) for a given rate of IED penetration (number of substations, horizontal axis). National Grid 2-29 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

343 Attachment COMM 2-8(c) Page 47 of 103 Figure 2-6: Theoretical Replacement Rates Figure 2-7 is very similar to Figure 2-6, the difference being the quantity shown on the right hand vertical axis and represented by the blue line in each figure. Again, the numbers were chosen to provide a statistical view of the number of times that the first substations would be refitted with IEDs based on an assumed 15-year lifespan. The step function exists since it was also assumed that a station is completely refitted with IEDs when a refit occurs. These figures are just meant to provide a reasonable view of possible replacement rates. Figure 2-7: Number of Refits Required during IED Rollout National Grid 2-30 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

344 Attachment COMM 2-8(c) Page 48 of 103 Many of these assumptions can and should be challenged and modified as with any model. IED life may be changed, different IED types may have different life spans, refits may be based on device type rather than be performed by station (for stations already fitted with IEDs). Also, individual or type based IED refits that do not involve other changes at the substation will be performed more quickly so once a significant number of substations (e.g. 30%) are fitted with IEDs a change in policy for refits is probable. This in turn will affect future replacement rates and may tend to shorten the refit cycle and it is expected that future programs are likely to be a combination of wholesale refits and selected device type refits. An optimum range would seem to be about new stations per year, which will take years for the replacements. (See Recommendation 8, page 3.) There are several reasonable methods to determine an appropriate rate for deploying IEDs. Some are forward looking/proactive and based on combinations of strategic/enterprise goals, some are tactical and focused on solving localized or widespread/geographical operational issues, and some are simply backed into i.e. assume that eventually there is no choice but to fit IEDs. At some point the need to replace electromechanical devices will be so overwhelming that it becomes a massive undertaking, unless the rate of IED deployment is increased soon Applications Systems Perform detailed inventory of engineering and operational systems. Gather more information about use of operations/engineering data and data flows and develop requirements for IED/operations data mart. (See Recommendation 1, page 1.) Decreasing the number of electromechanical devices will result in less recalibration of relays etc. and a corresponding reduction (or change) in maintenance work. Some of this reduction will possibly be offset against work required for IED replacement or software patches. 2.4 SCADA Penetration Existing Conditions National Grid collects SCADA data from three independent systems that support New England electric T&D operations (NE EMS), New York electric T&D operations (NY EMS), and New York gas distribution operations (GEMS). The NE EMS is an ABB SPIDER system, commissioned in 1997, that currently supports 425 RTUs with SCADA master stations located in Westborough, MA and Lincoln, RI. The NY EMS is a Stagg system, commissioned in 1985, that currently supports 518 RTUs with master National Grid 2-31 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

345 Attachment COMM 2-8(c) Page 49 of 103 stations located in Albany, Buffalo, and Syracuse, NY. The GEMS is a Fisher Controls system, Commissioned in 1986, that currently supports RTUs with a master station in Syracuse, NY. Both the NY EMS and GEMS systems are based on obsolete technology without any support from their original system vendors. National Grid has used third-party products and contractors, as well as their in-house technical staff to maintain and upgrade these systems. The NE EMS is a relatively new system that has limited support from ABB, while ABB is transitioning to a new baseline system (Network Manager) that will eventually integrate technology from two separately supported product lines (i.e., SPIDER and RANGER) SCADA Data Approximately 49 bulk stations in New England and 161 in New York represent approximately 40-50% coverage of SCADA. The majority of the SCADA facilities are conventional RTUs with mostly direct-wired points. RTU manufacturers include GE, Siemens (Telegyr), Qualitrol (Hathaway/Systems Northwest), Telvent (formerly Tejas Controls) and Advanced Control Systems. IED penetration at bulk stations is approximately 4%. Figure 2-8: SCADA Observability SCADA observability in New York is summarized by 85% CRCC observability, 50% ERCC observability and 35% WRCC observability. The most significant problems associated with outage durations are old equipment followed by lack of SCADA coverage. Lack of SCADA coverage is a factor in prolonged outage durations though customer call-in will provide assistance. National Grid 2-32 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

346 Attachment COMM 2-8(c) Page 50 of 103 ERCC has dial up alarms in about stations via SESCOA The SESCOA system is not working in WRCC and CRCC. SESCOA provides alarms at these stations during the day but not at night because operations are transferred to CRCC. SESCOA produces a paper feed, date and time and a code that tells what equipment had a problem. It does not describe what the problem was (e.g., for transformer alarms). It also provides a return to normal alarm, as well as breaker open and close notification. There is no way of knowing the actual problem at the station until a traveling operator gets to the site and diagnoses the problem, which can take a lot of time in remote regions. The SCADA system allows dispatch to monitor and control the power apparatus at substations. This includes opening and closing circuit breakers and raising and lowering transformer and regulator tap settings and switching substation capacitors on and off. SCADA monitoring provides real-time measurements of load, voltage on every piece of equipment in the station. The SCADA system for distribution is capable of monitoring momentary interruptions followed by successful reclose when the feeder breakers are monitored Communications Some of the bulk stations have no communication facilities connected to them but have IEDs installed, so the IEDs cannot be used to supply valuable operational and nonoperational information to operators, engineers, planners and technicians without a site visit and manual data collection. Poor or lack of communications represents part of the cause of a substantial amount of travel to and from stations for problem diagnosis, particularly for problems that could normally be diagnosed through normal SCADA coverage. SCADA/RTU communications in New England are provided over leased serial telephone lines. Some of the SCADA/RTU communications in New York utilize VeriLink devices as local communication hubs. Most SCADA communications occur over 1200 bps analog lines or 9600 bps digital lines Data Acquisition, Storage and Applications For V&O inspections, Maintenance and Operations relies on snapshot data for inspections, which must be acquired manually through site visits. Planning needs accurate peak load information to determine load schemes with consideration given for National Grid 2-33 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

347 Attachment COMM 2-8(c) Page 51 of 103 both temporary switching and erroneous data. For stations without SCADA, Planning relies on Maintenance and Operations when it performs V&O inspections to capture snapshots of data. The snapshots simply represent information gathered at the station when it is visited. Any data in between visits is lost. Planning then supplies these data to FeedPro for applications like load growth and feeder ratings, the results of which are estimates. Because reclosures are capable of providing only amp readings and reclosure status, Distribution Planning would like to obtain any information about pole mounted capacitors that are controlled by EMS, but very few are controlled in this manner. Alarm points at substations are grouped into one substation trouble alarm that is sent to the EMS. Operations would like specific alarm point information rather than a general station trouble alarm. Both ABB IS400 and PI provide operational and alarm data for New England where ERS is the system of record in New York. PI has 200 licensed users for 25 concurrent licenses in New England and is planned for expansion into New York. ERS is a historical Lodestar product maintained by National Grid, so licensing is not a problem but future upgrades are not available. ERS also contains metered data that is ultimately used to supply the New York ISO. In some cases, SCADA brings in more data than is strictly needed for operations and it would be helpful to have the equipment data (non-operational) that is not necessary for operations to be captured and stored in a data warehouse. It is understood that this nonoperational data comes with additional security requirements. There are dead zones where the EMS State Estimator s observability is poor. As a result, the state estimator cannot identify bad data due to PT or CT scaling constant errors and/or calibration inaccuracies. It is difficult to accurately diagnose faults. IEDs could provide distance to fault, magnitude of fault, and relay target data as points that could go into SCADA directly without using WaveWin and from there could be provided to distribution dispatching and field personnel. Most of the RTUs installed in bulk stations can be called to obtain SOE data. While all stations with DFRs have GPS clocks, stations with other IEDs are in the process of receiving them, while other bulk stations that have no SCADA coverage are not covered by SOE. Therefore, when a misoperation occurs it becomes difficult to analyze the cause National Grid 2-34 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

348 Attachment COMM 2-8(c) Page 52 of 103 when dealing with data that is not time synchronized because timestamps, where they exist, can vary up to 25 milliseconds. (NOTE: This was a major difficulty experienced in analyzing the August blackout and a subject of industry initiatives to improve.) In order to reduce momentary interruptions, information about them at substations and line reclosure locations would be useful, which means SCADA penetration and the quality of communications would have to increase. While Hydran and Qualitrol transformer ECM units can bring a lot of non-operational data to SCADA, care has to be taken in determining what non-operational data is needed (if any) by SCADA, and how and where it should be stored and accessed. The SCADA support groups are wary of having the SCADA systems become a data mart in themselves; by transferring the data from SCADA to a data mart (or PI) these concerns can be overcome. Even better, the data should go from the IEDs direct to a data mart Process and User There are times where C&MS is called to fix problems on transmission lines because C&MS has special skills that other first responders do not, or because C&MS crews are available and in the area. However, because fault location information is often inaccurate, determining the location of the problem is time consuming and inefficient first response. Incidents in New England are often first addressed by C&MS through morning reports that reflect events as old as 12 hours for maintenance purposes. First responders are aware immediately. Morning reports are prepared by dispatchers from IDS extracts, which include SOE if the impacted stations have SCADA coverage, and are ed to maintenance regional managers. If SOE data is not part of the IDS extract then site visits are required to obtain event data for problem diagnosis. Sometimes, calls from troublemen or pager messages are the first notice of a problem. The net result is that each region has different ways of addressing outages because of differences in equipment age (most significant problem) and lack of penetration of SCADA. This is a process that requires frequent direct communications with the dispatcher to help diagnose problems. In New York, substation construction and maintenance personnel rely on manually prepared load sheets captured by traveling operators. Maintenance has access to EMS but does not have access to alarms. Instead, they rely on the morning reports that reflect old and summarized information, not detailed alarm information, which would be helpful. New York s Field Operations is responsible for both gas RTU maintenance as well as electric. The gas SCADA is an old Fisher system that is tentatively scheduled for National Grid 2-35 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

349 Attachment COMM 2-8(c) Page 53 of 103 replacement in 2007 and the electric SCADA/EMS is being considered for replacement soon. Manual collection of load data has suffered in recent years as a result of downsizing. As a result, there has been a corresponding reduction in the number and consistency of manual load data collection Impacts to Business Users SCADA Penetration The percentage of SCADA penetration at substations is 50% (estimate based upon verbal input from several interviews) and is maintained at this level. Stations without SCADA cause a substantial amount of manual documentation, verbal communications, travel and time to diagnose events. The low SCADA penetration limits the usefulness of applications such as state estimation due to the lack of observability and redundancy of measurements required for estimation and bad data detection. The aging SESCOA system provides notice of alarms but because it cannot confidently diagnose events, the alarms trigger site visits whether they are needed or not. SESCOA is deployed in each region but it is only being used in ERCC Communications Where the communications infrastructure is good, it is used and expressed as such by business users. Where communications are insufficient or non-existent, gaps are made up by hardcopy reports, ed digital reports, direct communications between first responders or maintenance crews and engineers and control centers, and labor and time intensive trips to sites for event diagnosis. Slow or non-existent communications combined with lack of IEDs eliminates the incentive for data to be pushed to the corporate environment for use by others Data Acquisition, Storage and Applications PI provides valuable information to business users in New England, as does ERS in New York. However, a data warehouse is needed to store the large volumes of non-operational data that should not be passed through SCADA. While the expansion of PI in New York is planned, PI is not a system where data is accessed easily. Distribution Planning makes decisions on incomplete sets of data, such as non-coincident peak load readings, because continuous monitoring is not available at most distribution National Grid 2-36 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

350 Attachment COMM 2-8(c) Page 54 of 103 substations. Distribution SCADA is available at only approximately 40% of the distribution substations. If SCADA data could be transferred into AIMMS, then maintenance groups could analyze the number of operations against maintenance and repair information with the intent to fix potential problems before more problems occur. This would provide a mechanism to move from traditional time-based inspections and maintenance to condition-based inspections and maintenance. SOE points from different stations will have inconsistent time stamps resulting in the inability to determine the actual sequence of operations related to an event due to lack of time synchronization from a common source such as GPS. Distance to fault information can be used to cut down on fault investigation and patrol time. By reporting fault location to the dispatcher, the dispatcher will be able to direct the field crews more precisely he fault location. An expansion of the volume of data captured by IEDs should be carefully planned so that business users are not overwhelmed with data that they do not need. IED deployment should be prioritized by substation and device and information that is extracted should be useful and consistent Process and User The semi-automated process of producing morning reports with partial pictures of events causes time consuming site visits and subsequent problem diagnosis by analyzing data obtained from the site. The process required to support alarms from the SESCOA system is labor and time intensive. Knowing an event has occurred is valuable to control centers but the nature of the alarm remains unknown, causing a traveler to be dispatched to the location whether the actual problem that caused the alarm is warranted or otherwise. The V&O inspection process is largely manual and often requires on site physical computer connections to download data, whereas in a more modern environment data can be sent on a consistent basis to a data mart thereby having the potential to change the nature of the inspections and at the same time providing information to inspectors in advance of their inspections. National Grid 2-37 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

351 Attachment COMM 2-8(c) Page 55 of Known Initiatives The current Fisher Controls gas SCADA in New York is old and a new gas SCADA replacement is being considered for fiscal year There is consideration being given to upgrading or replacing EMS in both New England and New York. A RFP is in the process of completion. There is consideration being given to replacing ERS in New York with a meter data repository. As a result, an RFP for a meter data warehouse is under development. There are plans to extend the use of PI to include New York Best Practices and Gaps SCADA penetration at the transmission level should be nearly 100%. Distribution level SCADA penetration should include distribution substation transformers and feeder breakers at all heavily loaded substations and substations that serve critical customers such as hospitals, major industrial and commercial customers and airports. At stations where a single station trouble alarm is available it should be possible for operators to obtain more detailed alarm information through dial up, if necessary. A mechanism for obtaining more refined data will result in better dispatching decisions. The crews with the appropriate capabilities and equipment will be dispatched. All time series and event data should be captured in one historian system such as PI. Organizations, other than control center operations, with need for near real-time data should have access to SCADA data using PI. Combined with increased SCADA penetration, this will eliminate the process of manually retrieving SCADA data for delivery to engineering and maintenance groups. The data models and object IDs used in GIS, Outage, SCADA, and planning applications should be made common across the enterprise. Mismatches between outage device IDs and GIS is a frequent cause of integration failure. Additional SCADA load measurements should be provided to support system operations in the following areas: State Estimator observability and bad data detection Targeted areas for frequent or seasonal load transfers National Grid 2-38 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

352 Attachment COMM 2-8(c) Page 56 of 103 Service restoration / feeder reconfiguration Seasonal studies for outage planning and scheduling It is time to replace the EMS system in New York. (See Recommendation 9, page 3.) The EMS in New York is one of only three such remaining systems of this vintage. Both of the other two systems still running are being replaced or are in the process of being readied for replacement. 2.5 Asset Management and Condition Based Maintenance Best Practice and Gap Analysis Traditionally, utilities performed maintenance on a scheduled basis based on manufacturer s recommendations or long term experience, required maintenance activities were scheduled on a periodic basis. Many utilities have migrated most of their scheduled maintenance activities to a type of condition based maintenance (CBM) scheduling but where the conditions are themselves monitored on a scheduled periodic basis. (Example Inspect transformer oil on an annual or quarterly basis; use the results to determine when and why type of maintenance is required.) By contrast, some expensive and sophisticated equipment in other industries (gas turbines, jet engines, certain locomotives) are subject to continuous condition monitoring and true event / condition based maintenance triggers, as well as usage based maintenance scheduling, based on hours of operation and harshness of operating conditions, etc. as opposed to calendar based. This allows the maintenance decisions to incorporate actual usage out of a statistically derived life expectation between maintenance cycles, and to capture environmental and other factors, which result in condition alterations, that vary from the statistical norm. The more expensive the equipment or the greater the consequences of failure, the more justifiable investment in such CBM monitoring, diagnostic, and decision support systems becomes When an enterprise operates a fleet of equipment of varying type classes, the maintenance / replacement decisions themselves can be subject to global optimization where the economic impacts of failures on the enterprise as a whole are modeled and resources allocated to maintenance and replacement projects so as to best achieve enterprise goals. This approach, when applied to management of the entire fleet owned by the enterprise, is often called Asset Management (AM). These approaches use extensions of financial engineering models for probabilistic risk management and real options decision support. There are two critical components of these methodologies for CMB and AM strategy development: National Grid 2-39 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

353 Attachment COMM 2-8(c) Page 57 of 103 Figure 2-9: Best Practices CBM Model The first, as shown in Figure 2-9 is a probabilistic model of the breakdown and failure modes for individual equipment. For each piece of equipment, a type or class model is developed that has a probabilistic model of failures, which incorporates observations on equipment condition, maintenance history, loading, service, and usage history. The probabilistic model can start with manufacturer s data but ultimately has to incorporate observed equipment performance to calibrate the model. The particular model s developments, observed conditions, and monitored conditions are specific to each type of equipment. As noted earlier, the justification for investments in monitoring equipment and diagnostic systems is based on the value of the equipment and/or the impact that failures and breakdowns have on the enterprise s business performance as a whole. The failure impact of this type of model itself can be a probabilistic function of the consequences of equipment failures; the amount of revenues lost; congestion costs imposed; the impact on SAIFI and SAIDI, and so on, which all functions of loading, weather, and outage duration which are themselves uncertain. Given a certain observed condition and probability distribution function of failure / performance there will be several maintenance / asset management alternatives: do nothing (run to failure or to a change in conditions), or various maintenance possibilities from minimal service to extensive refurbishment, replacement, or even replacement with an upgraded capability unit. Each of these options has a cost and a predicted impact on the enterprise in terms of system impacts (congestion, for instance) and performance (reliability, load served, etc). National Grid 2-40 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

354 Attachment COMM 2-8(c) Page 58 of 103 The asset management problem then becomes a (complex) probabilistic optimization problem of maximizing some performance or financial attribute subject to operating criteria and requirements, resource constraints, performance constraints (targets), and financial constraints. Figure 2-10: Financial Performance versus Risk For instance, an enterprise might undertake to optimize over a period its asset base s expected costs (maintenance and replacement) subject to constraints on predicted performance (reliability, failures expected, etc.) and to resource constraints (no more than N units of a given type under maintenance at a time, total available crew hours per year are X). When the objective function (predicted asset costs) and the constraints (reliability) are themselves probabilistic functions, this becomes a risk reward problem or what the financial engineering community refers to as a portfolio optimization problem (Figure 2-10). When the cost/reward models and the constraint models are static (snapshots in time) the problem is well understood and there are numerous tools available to perform the analysis. When the models and the constraints are not static but must be modeled over time the problem is more complex and specialized applications must be used, or the problems transformed into more amenable forms. As applied to utility asset management, the problem can be used to minimize cost; minimize spending (cash out flow); optimize the condition of the system at an end horizon; or optimize operating reliability over the period (e.g., SAIDI). Whichever value is taken as the objective, the others can become hard or soft targets or constraints. More sophisticated models will incorporate decision factors such as whether to do all desirable maintenance in a substation once a major activity is scheduled. National Grid 2-41 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

355 Attachment COMM 2-8(c) Page 59 of 103 The utility industry has seen limited amounts of optimal maintenance scheduling in the past. For instance, it was routine prior to deregulation to schedule generator maintenance in this manner, and deregulated generation fleet operators still do that. What is new today is, first, the use of stochastic and constantly updated models of equipment failure and performance; second, the integration of these models into an enterprise model; and third, the extension of the portfolio optimization framework to all of T&D asset management so that maintenance decisions are coordinated and optimized in company with all other investment and spending decisions of the organization. National Grid 2-42 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

356 Attachment COMM 2-8(c) Page 60 of Benefit Cost Analysis This section of the report describes the high level ( order of magnitude ) benefit cost analysis (BCA) performed by KEMA for the ELSSI recommendations. This analysis compares the projected hard dollar (monetary) benefits that could be achieved by implementing the recommended ELSSI architecture at National Grid Company against the estimated costs to implement the recommended substation, communication and data mart facilities. The estimated ELSSI costs and benefits for National Grid were derived from results at other electric utilities that have either implemented ELSSI or have performed a detailed engineering analysis of the costs and benefits. 3.1 ELSSI BCA Methodology This ELSSI BCA provides a comparison between the monetary benefits achieved by implementing the proposed ELSSI architecture versus the cost to implement this architecture. A per substation analysis was performed that is, the benefits per substation were compared with the costs per substation The benefits and costs are significantly different for transmission substations and distribution substations, so somewhat different BCA results can be expected for each type of substation. Therefore, separate analyses have been performed for a typical National Grid transmission substation and a typical National Grid distribution substation. The general approach used by KEMA is summarized below: Prepare list of candidate ELLSI application functions: This is a list of ELSSI application functions that appear to provide benefits to National Grid and appear worthy of further consideration and analysis. This list of ELSSI Candidate functions includes items identified during end user interviews as well as industry best practices. Generate opportunity matrix to rank and prioritize the candidate ELSSI application functions: The opportunity matrix lists the candidate ELSSI functions and their potential impact on a number of benefit categories of interest to National Grid. Each candidate function was scored in each benefit category and the scores were totaled for each function to identify the highest priority ELSSI functions. These functions will undergo benefit cost analysis in the following steps. Compute rough estimates of benefits in dollars per substation per year. These estimates are based primarily on actual results achieved or computed (using much more rigorous analysis) by other similar utilities. These results have been scaled to reflect National Grid. Only those items considered hard monetary benefits (i.e., benefits that affect National Grid s bottom line ) were included in the analysis. National Grid 3-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

357 Attachment COMM 2-8(c) Page 61 of 103 Compute rough cost estimates for ELSSI on a per substation basis Cost estimates include procurement, implementation, operating, and maintenance costs for all modifications and additions to existing substation Intelligent Electronic Devices (IEDs), substation automation/integration platforms, communication infrastructure, and data management (data mart) facilities. Cost estimates used for the analysis were based on price quotations received from qualified system vendors and installation cost estimates prepared by other electric utilities on recent similar KEMA projects. Data warehouse costs, which are shared by all substations, are assumed to be approximately $10,000 (initial one-time cost) per substation, with approximately $1,000 per year operating and maintenance cost per substation. Initial one time costs (procurement and installation) were converted to an equivalent annual expenditure using appropriate present worth formulas. Compute Benefit-to-Cost ratio (BCR): KEMA computed BCR by dividing the annual ELSSI benefit by the equivalent annual cost to implement, operate, and maintain the system. A BCR greater than one indicates that the investment is economically viable. 3.2 ELSSI Candidate Functions This section covers the ELSSI candidate functions identified during and following the end user interviews. The candidate ELSSI functions are those items that appear to provide benefits to National Grid and appear worthy of further consideration and analysis. The ELSSI candidate functions have been grouped into the following four main categories: T&D Planning/Design Opportunities Fault Investigation and Trending Opportunities Condition Based Maintenance/Maintenance Management Opportunities Operation and Control Opportunities National Grid 3-2 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

358 Attachment COMM 2-8(c) Page 62 of T&D Planning/Design Candidate Functions Item Description Nature of Benefit Capital Asset Management/Planning Base decisions regarding major capital expenditure expenditures for T&D equipment on equipment condition rather than age. Use equipment condition metric based in part on significant events: o Number & magnitude of major through-faults o Number of operations under heavy load or through fault conditions o Other factors that have a major affect on the remaining life of the equipment. Information used to develop these metrics would be acquired from IEDs and other sources of field data. Voltage/VAR Control Develop and implement optimal control strategies for feeder voltage and VAR control. Use integrated field data to o Tune engineering feeder models used to examine optimal control strategies, o Identify worst performing feeders. o Implement the actual optimal control actions. Phase balancing Monitor individual phase loading measurements obtained from IEDs and other sources of field data. Use information to target investments in feeder reconfiguration to those feeders where significant imbalance exists. Reduction of dollars spent on capital improvement projects Defer or eliminate capital expenditures by freeing up capacity on existing equipment through reduced electrical losses. (Note: Supply cost savings due to reduced electrical losses does not provide $ benefits to National Grid (savings passed to customers)) Reduce electrical losses to free up capacity on existing T&D facilities Improve feeder voltage profile (power quality) for the benefit of National Grid customers. Load Forecasting Integrate accurate peak load measurements obtained from IEDs and other sources of field data with switching orders Identify peak loading anomalies resulting from temporary load transfers and adjust peak load measurements accordingly Reduce capital expenditures for T&D capacity additions by ensuring that peak load data measurements is accurately scrubbed to eliminate anomalies due to temporary switching conditions. National Grid 3-3 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

359 Attachment COMM 2-8(c) Page 63 of 103 Item Description Nature of Benefit System protection/fault Use actual field measurements obtained from Improve Study tuning substation IEDs to: Volt VAR Management verify and/or improve the quality of electrical models. Identify inconsistencies between feeder models and actual feeder facilities Use actual measurements obtained from substation IEDs to: SAIDI/SAIFI improvement by avoiding overtripping due to lack of coordination between protective devices. Avoid customer complaint resulting from low voltage or other power quality problem resulting from incorrect model. Improve voltage profile along the feeders Circuit reconfiguration and load balancing Verify and improve electrical models Support studies to identify feeders with poor power factor and/or voltage profile Enable volt/var control based on actual feeder measurements Use actual ampere and real power measurements obtained from substation IEDs to: Identify circuits where load on each phase is significantly out of balance Identify opportunities for transferring load from heavily loaded feeders to more lightly loaded adjacent feeders Avoid customer complaint resulting from low voltage at feeder extremities Reduce electrical losses Reduce peak load on heavily loaded circuits in order to defer capital expenditures for capacity addition. Improved voltage profile by achieving better balance the load between individual phases. Reduce electrical losses 3.4 Fault Location and Trending Candidate Functions Item Description Benefit Targeted Dispatching Outage trending - trees Integrate fault location information acquired from IEDs with Outage Management System (OMS) displays used by Dispatchers to "pinpoint" fault location. Use fault location info to direct field crews more accurately to the probable fault location. Integrate fault location information acquired from IEDs with outage cause data reported by repair crews to identify trends in outages involving tree contact. Labor savings due to reduction of fault investigation/patrol time by first responders. SAIDI (outage duration) reduction due to corresponding reduction in service restoration time. Achieve maximum reliability improvement benefit per dollar spent on tree trimming by targeting worst performing areas. National Grid 3-4 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

360 Attachment COMM 2-8(c) Page 64 of 103 Item Description Benefit Outage trending - equipment Integrate fault location information acquired from IEDs with outage cause data reported by repair crews to identify trends in outages involving equipment failures. Achieve maximum reliability improvement benefit per dollar spent on equipment improvements by targeting worst performing equipment by type and location. Outage trending - protection Outage trending - customer Integrate fault location information acquired from IEDs with outage cause data reported by repair crews to identify trends in outages protective equipment failures (misoperations). Integrate fault location information acquired from IEDs with outage cause data reported by repair crews to identify trends in outages caused by problems in customer equipment. Achieve maximum reliability improvement benefit per dollar spent by targeting worst protection system by type and location. Achieve maximum reliability improvement benefit per dollar spent identifying and mitigating problems in customer equipment. National Grid 3-5 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

361 Attachment COMM 2-8(c) Page 65 of Condition Based Maintenance (CBM)/Maintenance Management Candidate Functions Item Description Benefit Transformer inspections based on condition Distribution inspections based on condition CB inspections based on condition Acquire information from substation IEDs to determine key parameters indicating the operating condition of the substation transformer: dissolved gas and moisture content in oil number of tap changer operations, LTC "travel" (tap position drag hands) temperature differential between main tank and LTC compartment. Integrate IED information with maintenance management system to prioritize maintenance expenditures. Acquire information from protective relay IED (fault location, magnitude, and type) following permanent and temporary faults (successful reclosing) to identify distribution feeder locations that should be inspected for possible damage. Integrate this information with the maintenance management system to prioritize maintenance expenditures. Acquire information from substation IEDs to determine key parameters indicating the operating condition of high voltage circuit breakers: breaker operating time contact interrupting duty (I2t) number of re-strikes gas/air pressure Integrate IED information with maintenance management system to prioritize maintenance expenditures. Reduce expenditures on transformer maintenance by targeting equipment for maintenance based on equipment condition rather than elapsed time since last maintenance. Cost savings with scheduled versus emergency repairs. Reduce expenditures on distribution feeder inspections by targeting specific locations where temporary and permanent faults may have occurred. Reliability improvement (SAIDI and SAIFI) by inspecting/repairing problems at locations where temporary faults have occurred to avoid future permanent outage. Reduce expenditures on high voltage circuit breaker maintenance by targeting equipment for maintenance based on equipment condition rather than elapsed time since last maintenance. Cost savings by performing repairs on scheduled versus emergency basis National Grid 3-6 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

362 Attachment COMM 2-8(c) Page 66 of 103 Investigate relay & control misoperations Acquire information from protective relay IEDs and Digital Fault recorders: relay targets fault location and magnitude oscillographs sequence of events reports Provide this information to relay engineers and others for analysis. Restore critical equipment to service as quickly as possible following tripout (SAIDI improvement). National Grid 3-7 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

363 Attachment COMM 2-8(c) Page 67 of Operations and Control Candidate Functions Item Description Benefit Dynamic equipment rating Determine ratings of critical equipment (transformers, high voltage lines) based on measurements of actual conditions: Average increase of 2% - 3% versus fixed seasonal rating assumption. Power quality monitoring Ambient weather Oil temperature Recent loading history rather than conservative seasonal assumptions. Acquire information about harmonic content, voltage sags/swells, and other voltage distortions from IEDs Reduction of congestion charges. Reduce amount of load shedding during emergencies occurring at peak load. Defer capital expenditures to increase capacity. Proactive response to power quality problems before customer complaints occur. Provide this information to power quality engineers for investigation and possible corrections. Reduce damage claims due to voltage quality. Intelligent processing alarm Acquire detailed "non-operational" information about alarms to supplement consolidated "station trouble" alarms provided to system operators Improved use of crew resources by ensuring that dispatched crew is prepared to detail with the specific problem that has occurred. Outage Planning & Scheduling Determine load forecast for the proposed work time using integrated: weather forecast information measurements of actual ambient conditions equipment loading similar day historical information Information used to determine whether scheduled outage is possible without overloading other T&D facilities. Labor savings by avoiding sending the wrong crew to address a substation problem. Reduced maintenance and construction costs due to Better crew deployment Fewer false starts Job interruption due to unanticipated loading conditions. 3.7 Opportunity Matrix Strategies for exploiting ELSSI affect productivity, costs, and system performance differently and in different combinations. The opportunity matrix presented in this section lists the candidate ELSSI National Grid 3-8 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

364 Attachment COMM 2-8(c) Page 68 of 103 functions and their potential impact on a number of benefit categories of interest to National Grid. This section provides a qualitative assessment of the benefits provided by the ELSSI candidate functions. Each candidate function was scored in each benefit category and the scores were totaled for each function to identify the highest priority ELSSI functions. Benefit cost analysis was performed on the highest priority candidate functions identified in the opportunity matrix. To assist in weighing the expected impact of each ELSSI opportunity (provide a subjective assessment of each ELSSI candidate function), KEMA prepared an Opportunity Matrix (reference Figure 3.4) that illustrates the relative impact of each item. Scores between 0 and 3 were assigned to each item for each benefit area base on a consensus of the KEMA team members as follows: A score of 3 indicates that the ELSSI candidate function will have a major positive impact on the respective benefit area. Scores of 1 or 2 indicate less impact on the benefit area. A score of 0 indicates that the ELSSI function has no significant impact on the benefit area or may actually have a negative impact on the item. The following conclusions can be drawn from the Opportunity Matrix: 1. The ELSSI functions are expected to have the most significant (positive) impact on labor productivity, service reliability, and equipment inspection and maintenance processes. Figure 3-1 below shows the total impact (sum of the scores) of the ELSSI candidate functions for each benefit area. As seen in Figure 3-1, ELSSI has the highest total impact on labor (productivity), reliability improvement, and inspection and maintenance processes. Figure 3-1: ELSSI Impact On Each Benefit Area Labor Rework Reliability Restoration Time to Complete DSM/Purch. Congestion cost Losses Pwr Quality Insp. & Maint. Replacements New Expansion Score Benefit Area National Grid 3-9 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

365 Attachment COMM 2-8(c) Page 69 of Figure 3-2 shows the expected ELSSI impact by major category. As seen in this figure, the highest ELSSI benefits can be derived in the area of Condition Based Maintenance (CBM) and Maintenance Management, followed by T&D Planning, Fault Location, and Operation and Control, in that order. These results are shown in Figure 3-2. Figure 3-2: ELSSI Benefits By Opportunity Type T&D Planning/Design Fault location information CBM/ Maintenance management Operation and Control 3. Figure 3-3 shows the total scores for each ELSSI candidate function, with functions ordered by decreasing total score. Candidate ELSSI functions having a total score of 6 or higher were included in KEMA s BCA. National Grid 3-10 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

366 Attachment COMM 2-8(c) Page 70 of 103 Figure 3-3: Ranking of ELLSI Candidate Functions CB inspections based on condition Trf inspections based on condition Targeted Dispatching Capital Asset Man.Planning Dynamic equipment rating Outage Planning & Scheduling Intelligent alarm processing Phase balancing Voltage/VAR Control Investigate relay & control misop Outage trending - trees Power quality monitoring Outage trending - protection Outage trending - equipment Distr inspections based on condition Outage trending - customer System protection/fault study tuning Load Forecasting National Grid 3-11 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

367 Attachment COMM 2-8(c) Page 71 of 103 Application Productiv ity Figure 3-4: Opportunity Matrix Category Productivity Performance Equipment Costs Performance Equipment Cost Total Labor Rework Reliability Restoration Time to Complete DSM/Purch. Congestion cost Losses Pwr Quality Insp. & Maint. Replacement s New Expansion T&D Planning/Design Capital Asset Man.Planning Voltage/VAR Control Phase balancing Load Forecasting System protection/fault study tuning Fault location information Targeted Dispatching Outage trending - trees Outage trending - equipment Outage trending - protection Outage trending - customer CBM/ Maintenance management Trf inspections based on condition Distr inspections based on condition CB inspections based on condition Investigate relay & control misop Operation and Control Dynamic equipment rating Power quality monitoring Intelligent alarm processing/more detaile Outage Planning & Scheduling National Grid 3-12 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

368 Attachment COMM 2-8(c) Page 72 of Benefits This section of the report describes the high level ( order of magnitude ) estimates of benefits for the highest ranked candidate ELSSI functions identified in the previous section, which are listed below: Transformer and circuit breaker inspections based on condition Capital asset management and planning Targeted dispatching Dynamic equipment rating Voltage/VAR Control Phase balancing Intelligent alarm processing Outage Planning & Scheduling The following sections describe the process used to calculate the estimated dollar benefits for each of the above functions. The estimated benefits for each were derived primarily by extrapolating results of actual field implementation or rigorous engineering analysis reported by other similar sized electric utilities Transformer And Circuit Breaker Inspections Based On Condition Based on KEMA s previous experience at other utilities, the time interval between major inspections for substation transformers and high voltage circuit breakers can be increased by providing continuous equipment monitoring of transformers and circuit breakers using substation IEDs. For this benefit cost analysis, it was assumed that deploying continuous equipment monitoring would enable National Grid to double the current intervals between inspections. The net result of fewer major inspections is a reduction of labor costs for performing these inspections (material cost is zero for each test). Benefit calculations were performed using the following information supplied by National Grid in response to KEMA s Information Request Form (IRF): National Grid 3-13 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

369 Attachment COMM 2-8(c) Page 73 of 103 Input Data Supplied by National Grid Data Item National Grid Input Interval between transformer inspections 6 12 years Labor cost to do transformer inspection $700 - $1200 Material cost to do transformer inspection $0 Interval between circuit breaker inspections 6 9 years Labor cost to do circuit breaker inspection $600 - $900 Material cost to do circuit breaker inspection $50 - $200 The expected benefits achieved by reducing the number of major inspections through equipment condition monitoring are shown in Figure 3-5 below. Figure 3-5: Estimated Benefits of Condition Based Inspections Transmission Substations Distribution Substations $2,000 per year $1,000 per year Capital Asset Management And Planning Based on KEMA s previous experience at other utilities, the expected number of major equipment failures per year in high voltage circuit breakers and transformers can be reduced through continuous monitoring of equipment condition IEDs. One utility in central US reported the savings identified in the table below after 1.5 years of actual field experience with this approach. Item Initial Annual Annual Ben. B/C Existing CBM System + ( 00) IED-driven CBI years 1-2 IED-driven CBI projected final CBI-driven CBM (delta figures) IED-driven CBI and CBM total $4,000 $1,840 $9,500 $3,800 $14,300 $1,200 $450 $685 $1,200 $1,785 $2,800 $660 $1,300 $2,750 $1, * ** * Includes early hits but early costs ** Expected to improve due to asset base aging model National Grid 3-14 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

370 Attachment COMM 2-8(c) Page 74 of 103 A second utility in central US reported the following figures following 5 years of actual operation using substation IEDs for asset management and planning. Item Initial Annual Annual Ben. B/C Targeted IED- CBI&M as done $2,780 $380 $ Targeted IED- CBI&M planned $2550 $440 $ Systmwd IED-CBI&M (est) $12,850 $2080 $3, To compute the rough estimate of benefits for National Grid, it was assumed that the number of high voltage circuit breaker major failures could be reduced by 75 percent and the repair costs for each incident would be reduced by 50 percent by detecting the problem in an incipient stage. For substation transformers, it was assumed that the number of major failures in substation transformers could be reduced by 75 percent and the repair costs for each incident would be reduced by 40 percent by detecting the problem in an incipient stage. The estimated savings in repair and replacement costs is computed by multiplying the difference in failure rates with and without ECM by the repair or replacement cost. Furthermore, it was assumed that transformer and circuit breaker life could be extended by 2 years (i.e., equipment replaced deferred by 2 years) by deploying continuous equipment condition monitoring. Input data supplied by National Grid in the Information Request Form (IRF) that was used to perform the benefit calculations is listed below. Input Data Supplied by National Grid Data Item National Grid Input High voltage circuit breaker cost $55,000 MTBF for high voltage circuit breakers years Bulk power transformer cost $2,250,000 MTBF for bulk power transformer 9 years Distribution sub transformer cost $650,000 MTBF for distribution substation transformers 34.5 years The projected National Grid results are shown in Figure 3-6. Figure 3-6: Estimated Benefits of Condition Based Asset Management Transmission Substations Distribution Substations $257,000 per year $34,000 per year National Grid 3-15 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

371 Attachment COMM 2-8(c) Page 75 of Targeted Dispatching Having additional power system disturbance data (relay targets, fault location and magnitude, etc.) will assist field crews in investigating and locating the fault and thereby reduce both investigation time and feeder patrol time. This reduces the average customer outage duration, which in turn results in reduction of lost revenue and reduction of customer outage costs. Progress Energy Carolinas reported the following savings achieved through targeted dispatching techniques using fault location information from IEDs: Item Initial Annual Annual Ben. B/C Use of Subst. IEDs to improve distribution restoration all $1,666 $285 $1, Credited with 7 minutes total SAIDI improvement Progress Energy Florida reported the following savings: Item Initial Annual Annual Ben. B/C Use of Subst. IEDs to improve distribution restoration all $3,760 $610 $2, Expected 3-5 minutes total SAIDI To calculate the rough estimate of benefits to National Grid, it was assumed that fault investigation and patrol time would be reduced by 75% - 90% by taking advantage of fault location information available from IEDs. Note that this application requires specialized software that uses probabilistic methods to compute fault location using the raw information supplied by the IED. The estimated benefits for National Grid are shown in Figure 3.7. Figure 3-7: Estimated Benefits of Targeted Dispatching Transmission Substations Distribution Substations $5,000 per year $15,000 per year National Grid 3-16 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

372 Attachment COMM 2-8(c) Page 76 of Dynamic Equipment Rating The dynamic equipment rating function will provide an accurate assessment of the loading limits for power transformers and transmission lines based on ambient environmental conditions, internal oil and winding temperatures, recent loading history, and other such factors. This will enable National Grid to exploit the normal rating and emergency short-term ratings of the power transformer or circuit without overloading the equipment. Continuous calculation and monitoring of dynamic equipment ratings will enable National Grid to squeeze the maximum capacity out the existing power equipment without excessive heating that can cause insulation to become brittle with excessive loss of life for the transformer. By calculating the power transformer rating dynamically based on environment and equipment conditions, excessive heating which causes greater that normal loss of life, can be avoided. Results reported by several utilities that have deployed dynamic equipment ratings are shown in the following figure. Item Initial Annual Annual Ben. B/C Australian Wind Farm Tie $425 $125 $1, Large US Utility Pool Tie $1,400 $450 $ Southern US best 3 lines $2,800 $750 $7, best 10 $29,000 $7,000 $15, European Cable Project $1,800 $217 $ South American Intertie $1,850 $200 $1, For National Grid Company, it was assumed that dynamic equipment rating would enable National Grid to increase the maximum rating of its equipment by approximately 3 percent over the present maximum loading. This additional capacity would enable National Grid to defer expenditures on capital additions by several years. The rough estimates of benefits for National Grid s transmission and distribution substations are shown in Figure 3-8. Figure 3-8: Estimated Benefits of Dynamic Equipment Rating Transmission Substations Distribution Substations $21,000 per year $7,000 per year National Grid 3-17 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

373 Attachment COMM 2-8(c) Page 77 of Voltage/VAR Control Improving the feeder power factor during peak load and/or off- peak periods will enable National Grid to reduce the electrical kilowatt-hour losses that must be purchased to serve a given load. Furthermore, by improving the power factor during peak load conditions, National Grid will be able to reduce the peak demand on the distribution feeders. Another benefit is improved voltage profile along the feeders, which will provide operators with increased flexibility in operating the feeders, especially when load needs to be transferred during power system contingencies. Results reported by several utilities who have used IED data to improve the voltage profile on their feeders and obtain better control over reactive power flow on their feeders are shown in the figure below. VAR Dispatch Voltage Control Annual savings BCR Annual savings BCR Mid Sized Investor Owned Utility 2, , Large Investor Owned Utility 18, , Mid Sized Investor Owned Utility 9, , The rough estimates of benefits for National Grid s transmission and distribution substations are shown in Figure 3-9. Figure 3-9: Estimated Benefits of Improved Volt/VAR Management Transmission Substations No Benefit Distribution Substations $20,000 per year Feeder Reconfiguration and Load Balancing Continuous monitoring and recording of load on each feeder will enable National Grid to identify opportunities where lad can be transferred from a heavily loaded feeder to a less heavily loaded feeder. Monitoring the load on individual phases of each feeder will enable National Grid s planning engineers and operators to identify feeders requiring reconfiguration (i.e., shift load from heavily loaded phases to a less heavily loaded phase. The benefits of using substation IED data for feeder reconfiguration and load balancing include deferral of capital expenditures for capacity addition, reduced electrical losses, and improved voltage profile and operating flexibility. Rough estimates of benefits for National Grid are shown in Figure 3-10 below. National Grid 3-18 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

374 Attachment COMM 2-8(c) Page 78 of 103 Figure 3-10: Estimated Benefits of Improved Feeder Reconfiguration and Load Balancing Transmission Substations No Benefit Distribution Substations $15,000 per year Intelligent Alarm Processing Intelligent alarm processing using detailed information obtained from substation IEDs will enable dispatchers to learn as much as possible about the nature of a station trouble alarm before dispatching a crew to the scene. This will ensure that the dispatched crew is the correct crew for the type of trouble being experienced. The results of one eastern US electric utility are shown in the figure below. Item Initial Annual Annual Ben. B/C Targeted Alarms as done $2,780 $380 $1, Targeted Alarms as plan $2,550 $440 $1, Systemwide plan $12,850 $2,080 $3, Rough estimates of the benefits to National Grid from implementing Intelligent Alarm Processing are shown in Figure 3-11 below. Figure 3-11: Estimated Benefits of Intelligent Alarm Processing Transmission Substations Distribution Substations $1,000 per year $1,000 per year Outage Planning & Scheduling Scheduling work on existing distribution circuits requires careful planning to ensure that service to customers impacted by the proposed work is not interrupted during the work activities. Load data obtained from substation IEDs can be combined with other substation data (e.g., ambient temperature and weather information) and then compared with other similar days in the past to develop accurate load forecasts (estimates) for the proposed work time. This information can be used to determine if work can be performed at the proposed time or must be deferred. National Grid 3-19 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

375 Attachment COMM 2-8(c) Page 79 of 103 Benefits that can be achieved using the ELSSI function include reduction of interrupted work activities (and wasted crew time) due to higher than expected load, more accurate determination about whether a work site must be restored to normal conditions at the end of each day for multi-day work activities, and more accurate determination about when live line work is needed. Rough estimates of savings for National Grid are shown in Figure 3-12 below. Figure 3-12: Estimated Benefits of Outage Planning and Scheduling Transmission Substations Distribution Substations $15,000 per year $30,000 per year 3.9 Costs The ELSSI cost per substation include one-time costs (equipment purchase, installation, etc.) plus ongoing (annual) O&M expenses for the equipment. KEMA has assumed that the annual O&M costs are a fixed percentage (3% to 10%, depending on type of equipment) of the initial one time cost. Cost estimates for typical National Grid transmission and distribution substations are shown in Figures 3-6 and 3-7. These costs are determined as follows: 1. Substation Integration and Automation Platform: The substation integration and automation (SI&A) platform includes all facilities needed to integrate substation IEDs, interface with directwired I/O points, perform local automation functions (e.g. automatic bus failover) and user interface functions, and interface with the control center. It is assumed that National Grid would purchase these facilities from a qualified system vendor. The SI&A costs shown in the Tables 3-13 and 3-14 are based on actual system vendor firmpriced proposals that have recently been received by KEMA as part of substation automation projects at other electric utilities. The estimate includes: Hardware, including a data concentrator, programmable logic controller (PLC), PC for local HMI, substation network components, and GPS time synchronization units. Software, including a local human machine interface (HMI), PLC ladder logic, IED protocol profiles, databases, and communication software System integration, the cost for a system vendor the connect the hardware components integrated unit, including establishment of suitable communication links to each substation IED National Grid 3-20 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

376 Attachment COMM 2-8(c) Page 80 of 103 Miscellaneous vendor services, such as testing, training, documentation, warranty, shipment, etc. Installation, including the cost for National Grid to install and interconnect the substation integration and automation components under the technical guidance of the system vendor 2. Substation IEDs: It has been assumed that the National Grid substations in question are equipped with electromechanical (e/m) or solid state electronic (non-ied) protective relays that would have to be replaced with IEDs. The cost estimates are based on installed cost figures received during a recent project from a major electric utility in north central US. The per panel cost estimates are listed in the table below. To determine the cost of IEDs for National Grid substations, the costs per panel were multiplied by the number of lines, transformers, and circuit breakers in the average National Grid substations. Component IED Implementation Costs Per Panel Cost per Panel 345kV line $70, kV line $30,000 Bulk power transformer $60,000 Distribution substation transformer $40,000 Distribution feeder $25,000 Equipment Condition IEDs $15,000 per IED 3. Communication Costs: For the purposes of this high-level benefit cost analysis, KEMA assumed that National Grid would utilize commercial facilities (frame relay, digital cellular service, etc.) for handling communications between the substations and the control center. The ongoing service charges for using these facilities are estimated to be approximately $1,000 per month per substation. However, National Grid may be able to negotiate a significantly lower price for this service. Note that the cost of this communication facility will be offset by a reduction in existing leased line charges for EMS SCADA communications that are currently used at many National Grid substations. 4. IT/Data Mart Costs: The data mart portion of the ELSSI architecture is shared by all substations that are included in the system. Therefore, a portion of the cost to implement the recommended central data mart facilities was assigned to each substation. It was assumed that the one time data mart cost per substation would be $10,000 and ongoing O&M would be $1,000 per substation. National Grid 3-21 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

377 Attachment COMM 2-8(c) Page 81 of Costs Per ELSSI Application: Each ELSSI application function has its own incremental cost to implement that function beyond the IED foundation costs described above. Incremental costs associated with each ELLSI function are identified in the benefits section for each function. Figure 3-13: Foundation Cost for Transmission Substation Foundation Costs Initial Cost Annual O&M Cost Substation Integration and Automation Platform - Hardware (includes spare parts) $ 54,000 $ 1,620 - Software $ 17,000 $ 1,700 - System Integration $ 49, Misc vendor services (see note 1) $ 40, Installation $ 70,000 IEDs (quantities in parenthesis) - 345kV lines (4) $ 280,000 $ 8, kV lines (8) $ 240,000 $ 7,200 - Bulk power transformers (2) $ 120,000 $ 3,600 - Distribution transformers (0) Distribution feeders (0) IEDs for ECM (11) $ 165,000 $ 4,950 Communication facilities - Equipment $ 2, Service charges (see note 2) -- $ 12,000 IT/Data mart - Cost per substation $ 10,000 $ 1,000 TOTAL $ 1,048,000 $ 40,470 National Grid 3-22 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

378 Attachment COMM 2-8(c) Page 82 of 103 Figure 3-14: Foundation Cost for Distribution Substation Foundation Costs Initial Cost Annual O&M Cost Substation Integration and Automation Platform - Hardware (includes spare parts) $ 51,000 $ 1,530 - Software $ 17,000 $ 1,700 - System Integration $ 49, Misc vendor services (see note 1) $ 40, Installation $ 70,000 IEDs (quantities in parenthesis) - 345kV linesn (0) kV lines (4) $ 120,000 $ 3,600 - Bulk power transformers (0) Distribution transformers (2) $ 80,000 $ 2,400 - Distribution feeders (6) $ 150,000 $ 4,500 - IEDs for ECM (6) $ 90,000 $ 2,700 Communication facilities - Equipment $ 2, Service charges (see note 2) -- $ 12,000 IT/Data mart - Cost per substation $ 10,000 $ 1,000 TOTAL $ 680,000 $ 29,430 Notes: 1. Includes documentation, training, warranty, shipping, and other miscellaneous items supplied by the vendor 2. Assumes commercial facility such as frame relay or digital cellular service will be used 3.10 Benefit Cost Analysis Benefit to cost ratios (BCRs) were determined by dividing the estimated annual benefits by the estimated annual costs. The results for transmission and distribution substations are shown in Figures 3-15 and As seen in these figures, the rough estimate of BCR for transmission substations is 1.79 and the rough estimate of BCR for distribution substations is This indicates that ELSSI is financially viable (benefits exceed costs) in both types of substations. The results also indicate that investing in ELSSI architecture at transmission substations is significantly more attractive financially than investing in ELSSI at the distribution substations. National Grid 3-23 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

379 Attachment COMM 2-8(c) Page 83 of 103 Figure 3-15: BCA Results for National Grid Transmission Substation Area Initial K$ Annual K$ Benefit/Yr K$ B/C Ratio Implement IED Foundation $1,048 $40??? Targeted Dispatch + $4 $1 $5 3.5 IED-driven CBI & CBM + $25 $1 $ Dynamic Equipment Ratings $13 $3 $ Integrated Volt/VAR $0 $0 $0 -- Feeder reconfiguration/phase balancing $0 $0 $0 -- Outage Planning and Scheduling $1 $1 $ Intelligent Alarming $1 $1 $1 0.9 $1,092 $47 $ Figure 3-16: BCA Results for National Grid Distribution Substation Area Initial K$ Annual K$ Benefit/Yr K$ B/C Ratio Implement IED Foundation $680 $35??? Targeted Dispatch + $1 $1 $ IED-driven CBI & CBM + $12 $1 $ Dynamic Equipment ratings $5 $1 $7 4.5 Integrated Volt/VAR $10 $1 $ Operations planning/switch management $1 $1 $ Feeder reconfiguration/phase balancing $1 $1 $ Intelligent Alarming $1 $1 $1 0.9 $711 $36 $ Figures 3-17 and 3-18 show the incremental BCRs for individual ELSSI application functions at transmission and distribution substations. The incremental BCRs illustrate the relative payback of individual ELLSI functions. As can be seen in Figure 3-17, IED-driven condition based inspections and maintenance has the highest payback by far, following by outage planning and scheduling. At distribution substations, outage planning and scheduling is the highest payback item, followed by CBI/CBM, targeted dispatch and feeder reconfiguration. National Grid 3-24 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

380 Attachment COMM 2-8(c) Page 84 of 103 Figure 3-17: Incremental BCR for Transmission Substations Targeted Dispatch + IED-driven CBI & CBM + Dynamic Equipment Ratings Integrated Volt/VAR Feeder reconfiguration/phase balanceing Outage Planning and Scheduling Intelligent Alarming Figure 3-18: Incremental BCR for Distribution Substations Targeted Dispatch + IED-driven CBI & CBM + Dynamic Equipment ratings Integrated Volt/VAR Operations planning/switch management Feeder reconfiguration/phase balanceing Intelligent Alarming National Grid 3-25 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

381 Attachment COMM 2-8(c) Page 85 of BCA Conclusions The Benefit Cost analysis shows that the benefits provided by the ELSSI application functions are sufficient to justify the investment at both transmission and distribution substations. ELSSI is significantly more beneficial at transmission substations. Not all substations are alike in terms of ELSSI benefits and costs. To ensure the maximum payback on investment, ELSSI should be targeted at substations that provide the highest benefits at the lowest possible cost. National Grid 3-26 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

382 Attachment COMM 2-8(c) Page 86 of Vision and Recommendations 4.1 Vision for the Future Data Mart Vision Ideally, there will be only one system responsible for collecting data from a device although more than one system may be responsible for disseminating the data based on the requirements of its users and any transformations required. Where data is collected by one system and disseminated by many it is important to identify where the data is being stored after collection. This may be in the collecting system, a separate database or one of the disseminating systems but it is important that one system is designated as a system of record. If deviations in the data are discovered between systems, the data that is correct is that stored in the system of record. It is important that any errors are fixed in the system of record. This may require changes to be made in that system or in the processes that collect and feed data to that system. This ensures that all users who use that data receive the same data and that errors get fixed upstream of the users. A key action for National Grid is to develop a matrix of system information for all engineering and operational systems. This matrix should include: The name of the system A short overview of functionality. The business owner (if there is one) Users of the data (by functional area). Sources of data by type. Outputs from the system (interfaces with other systems) Known problems or issues with the system In building this matrix it is important to categorize the data into classifications that can be associated with each system to facilitate a data map that shows data flows between these systems and what the data flow frequencies and dependencies are. It is important to take into consideration the different types of users that may want to access IED information. The following figure illustrates four of these access levels: National Grid 4-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

383 Attachment COMM 2-8(c) Page 87 of 103 Figure 4-1: Different Levels of User Access Substation: Data may be accessed at the substation through a local data store that acts as a concentrator for IED data. This system periodically passes data back to a central location but provides vital access to equipment health and history data. This level of access was requested in several interviews [004, 010, 026, 036, 045] IED User Base: Data may be accessed by engineering and operations users remotely without the need to visit the substation. This may include relay technicians, field support, substation maintenance, Meter Data Services etc. Enterprise Users: This includes users who do not have an as immediate need to see data as users identified above and who are located in other areas of the enterprise. These users are typified by a need to look at historical data in order to identify trends, limiting values, unusual variations and summarize or drill down into the data. This may include Meter Data Services, Planning Functions, Substation Design, Control Center staff etc. External Customers: This includes users who are external to National Grid s core functions such as commercial and industrial customers, system operators (potentially), independent marketers, neighboring utilities, and National Grid staff subject to FERC 2004 restrictions. National Grid 4-2 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

384 Attachment COMM 2-8(c) Page 88 of Communications Vision With the advent of IEDs, National Grid has shown through its new substation design LPS03 and its predecessors that IED capabilities play a major role in prudent device automation and increased information exchange. However, along with automation of this nature and potential magnitude comes the increased need for data and corresponding upward pressure on the communications infrastructure. While National Grid has kept pace with its corporate WAN capacity by employing large bandwidth fiber and supporting solutions, the level of communications to electric and gas infrastructure (substations) is much lower. Figure 4-2: Drivers, Benefits and Consequences There are clear, inherent benefits (Figure 4-2) to upgrading National Grid s communications infrastructure to substations and there are corresponding consequences of doing so. However, none of this occurs until the benefits have been tied to the company s business drivers of safety, reliability and cost. On one hand, communications can be viewed as a sunk cost of doing business, while on the other hand, it can be viewed as a very positive change agent that opens doors to new, very powerful and different ways keeping the lights on. National Grid 4-3 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

385 Attachment COMM 2-8(c) Page 89 of 103 Figure 4-3: ELSSI Communications Vision Figure 4-3 depicts a conceptual vision for National Grid s supporting communications infrastructure. This conceptual vision is not designed to show how the end state should be engineered, but rather it is designed to the drivers, benefits and consequences of upgrading communications. These drivers, benefits and consequences result in another conceptual picture of how the fundamental operational (real time, shown in red), non-operational (shown in green) and remote access (shown in blue) data paths interact within the National Grid business environment. The following diagram depicts how these data paths move from the field to the office and how they re controlled Data Paths The three logical paths of data, operational, non-operational and remote access, will be passed along one physical network but through three logical paths. The first path is traditional operational data to SCADA that is extracted using an industry standard protocol, such as DNP. The second path, which can effectively reduce the volume of data passed to SCADA/EMS, is non-operational data that is continuously uploaded to a future data mart. Finally, remote access provides authorized users options for accessing secure systems and data from points within and external to the Corporate WAN Stations Stations will deploy LANs with connected IEDs that pass data through data concentrators to points on the Corporate WAN. Data concentrators reduce the traffic over the line and the time that the line is consumed National Grid 4-4 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

386 Attachment COMM 2-8(c) Page 90 of 103 with traffic, which will reduce the potential for lost data and increase the amount of historical data (e.g. faults) that can be captured from any IED inside the substation and used to support business decisions Transmission & Distribution One specific service, Telemetrics or similar service, can be expanded at National Grid s option to support event location and diagnosis along transmission and distribution lines while frame relay technology can be deployed to support existing communications Bandwidth and Remote Access Communication lines with a minimum bandwidth of 56Kbps, suitable to support two-way traffic and multiple applications, will also allow virtual connections to any IED from the central office, or remotely when necessary. Upgradeability of communication bandwidth for selected sites will be easier with frame relay. This is important for remote access problem diagnosis as well as the most efficient means of updating HMIs and potentially relay setting schemes. The bandwidth also sets a standard for obtaining a continuous stream of non-operational asset performance information necessary to support an asset management strategy Security The new communications strategy leverages the current LPS03 design and operates within NERC CIP Standards requirements. Access will be limited through user authentication and security strategies, such as VPN, and implementation of firewalls and security perimeters. Figure 4-3 shows the conceptual security elements; actual implementation may be accomplished through the use of dual-purpose devices. For example, the router and firewall shown in the Substation Environment may be implemented as a single router device containing access filter rule sets to control access. Additionally, not all elements will be initially required for NERC CIP standards compliance. The firewall shown on the telemetry data path into the EMS Environment is only required by the NERC CIP Standards if the communication uses a routable protocol (e.g., DNP/IP). If this is not the case, the design may be initially implemented without the firewall, with its addition when conversion to a routable protocol is implemented. 4.2 Recommendations Cost Benefit Analysis and Overall Recommendations KEMA believes that National Grid should develop overall ELSSI strategies, plans, and architectures aimed at closing the key gaps identified above, especially Condition Based Maintenance, Asset Management/Utilization, Fault Location, and Intelligent Substation Alarming. The estimated savings from these ELSSI exploitations would be between $25M and $30M a year, assuming that ELSSI is implemented at approximately 100 targeted substations. These savings would justify an investment of National Grid 4-5 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

387 Attachment COMM 2-8(c) Page 91 of 103 $100M to $150M in ELSSI over time (depending upon the savings level, schedule, and discount rate considered), which at an estimated $1.1M per substation deployed, would support a penetration of approximately 100 ELSSI substations across National Grid. This penetration rate would allow National Grid to plan on the less scalable exploitation strategies around productivity improvements as a second phase; the benefits from these strategies would then fund broader penetration in a virtuous cycle. The cost and benefit numbers behind these recommendations are extremely rough as there was no engineering analysis done of the existing infrastructure, maintenance budgets, capital budgets, or deployment costs. To amplify on the overall recommendation stated above KEMA recommends the following specific activities as next steps: 1) Continue to develop the ELSSI technical architecture by extending it to include IEDs, communications, data models and integration, and data marts as soon as possible. This will have the benefit of ensuring that ongoing IED deployment, applications procurement, and so on all will be compatible with the final ELSSI plan as it is developed and that retrofit costs of infrastructure, systems, and software are avoided. This is an inexpensive step to take at this time the costs occur when the integration systems and communications are deployed. The ELSSI architectural plan should include the following a) Develop Enterprise Standards for IED deployment / installation / configuration. i) Analysis of the inventory of electromechanical and older electronic protection and instrumentation infrastructure and development of a replacement plan based on the concept of eliminating types of equipment in a planned manner across the enterprise. ii) Development of a technology phase model that would allow National Grid to make decisions to evolve / update IED standards and to review the timing of the type elimination in a structured, enterprise fashion. This would provide an enterprise structure for managing generational cohorts of IED standards that would be active for a period of years (see Section ) and phased out or evolved in a managed fashion that incorporated considerations of maintenance, spares, vendor support, training, and staffing as well as technological cost benefits in an integrated formal structure. iii) Integration of the architectural plan with the CBM and AM plans developed (2 and 3 below) for determination of the best timing of managing the technology generational cohorts of IED deployment. National Grid 4-6 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

388 Attachment COMM 2-8(c) Page 92 of 103 iv) Development and adoption of a more formal plan for testing and pilot field testing of proposed new IEDs for incorporation in the IED standards and generational plan. b) Develop an enterprise communications plan that incorporates technology and policy issues around security, sharing of communications channels for operational, engineering, and metering information; how data communications to substations can be integrated with potential future dispatching / mobile workforce communications, distribution automation communications, and in particular how communications and security fit into a Substation Automation architecture standard for the enterprise. c) Develop a Substation Automation architecture standard that would provide for common data concentrator, substation computers, firewalls, and other substation automation and communications/control equipment that integrate IEDs and communications. d) Continue the work of the TISC to maintain a formal standards review process with owners and decision procedures that span across departments. e) Develop a plan for adopting evolving standards for substation data models (e.g. CIM extensions) and a Service Oriented Architecture and Event Driven Architecture model for ELSSI. These plans will have as key elements plans for migrating existing legacy applications and systems into this architecture in a cost effective way. The first level in the service oriented architecture adoption path is creating services from tasks contained in new or existing applications. A service oriented architecture provides the ability to design applications and systems that provide services to other applications through published and discoverable interfaces to be invoked over a network. This way of creating applications can provide a more powerful, and flexible programming model that can also reduce both development and maintenance costs and is fully compatible with National Grid s plans to extend the use of PI, to extend the use of Wavewin, and to replace the existing repositories for meter data with a single new system. f) Define enterprise Data Mart requirements and develop a plan for their implementation and governance. As part of this plan, and as a precursor to developing a service-oriented architecture based virtual data mart, develop a meta model for operations data and identify systems of record. 2) Analyze the T&D infrastructure more carefully to refine the estimates of savings from CBM and AM strategies and to identify targeted equipments / stations for deployment National Grid 4-7 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

389 Attachment COMM 2-8(c) Page 93 of 103 and exploitation. This will allow development of the ELSSI plan aimed at CMB and AM. To avoid the costs associated with doing this at an equipment-specific level, it can be done in classes of equipment as in 230kV power transformers of type XYZ with historic loadings >90% over 100 hrs / year. 3) Analyze outage records to identify the parts of the system most likely to benefit from district-wide availability of electronic fault location and optimized dispatch, in order to identify the SAIDI improvements and crew cost savings potential. This will allow the evaluation of ELSSI for these regions and the development of an ELSSI plan. 4) Identify the substations where intelligent substation alarming can be deployed. (Note: this requires IED deployment but depending upon SCADA penetration and RTU spare capacity, may not require communications deployment) a) Priority should be given to substations where (a) there is a history of trouble alarms requiring manual on site inspections and (b) where the stations are remote or otherwise situated such that significant time expenditures are required to investigate trouble alarms. b) When the substations already have SCADA RTUs installed and the RTUs have the capacity for additional digital points to be added, the implementation cost is driven by the cost of installing alarm sensors as needed. Where the RTUs require expansion then the cost will be also be a function of whether the RTUs can be expanded or would require replacement. Where there is no SCADA (which begs the question of how trouble alarms are received) then alternative communication strategies for trouble alarms only have to be considered. (e.g. wireless using public networks, etc; where there is no control, market sensitive data, etc to cause security concerns) 5) Develop a plan for implementation of ELSSI-based CBM, including processes and steps required to integrate it fully with NGRID s strategic and tactical asset management programs. There are really two recommendations here: A) develop a plan to implement CBM to the full extent that it makes a positive contribution to business performance, and, B) do so by integrating that CBM decision-making with NGRID s asset management process. a) ELSSI can provide the information base and verification system needed for an effective CBM function, whether that is implemented as a stand alone system (independent of other O&M or capital spending decisions) or as one part of the portfolio optimization within the overall corporate asset management program. National Grid 4-8 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

390 Attachment COMM 2-8(c) Page 94 of 103 Independent implementation of CBM would provide improvement in maintenance costs and effectiveness, which as discussed elsewhere are estimated at around $10 - $12 million per year. Thus, regardless of how it will implement it, NGRID should evaluate to what extent (on what equipment, in what way, prioritizing benefits by equipment type and area of implementation) CBM will contribute, and then develop a plan to make maximum use of its potential to improve maintenance effectiveness and contribution to its bottom line. KEMA recommends that NGRID look beyond a stand-alone CBM optimization process (A) and plan to also integrate that CBM process into NGRID s corporate asset management process. This would provide all the tactical savings of (A) above, but also permit NGRID to assess and optimize CBM s role as a part of its portfolio planning. KEMA believes that an integrated CBM-AM process using the portfolio techniques described in this report can be compatible with the use of its new project portfolio software for project prioritization. At an enterprise level, additional objectives (NGRID identifies these as citizenship ) must be considered as well as projects, which impact reputation, regulatory issues, or which are required by regulatory or investment community forces. Prioritization software can be used at this level, but the departments submitting projects to enterprise prioritization should make use of portfolio techniques to assess the financial and reliability impacts of projects quantitatively using risk adjusted methods. Based on results at other utilities and its knowledge of NGRID s current situation, KEMA believes integrated CBM-AM will provide substantial additional benefits and business flexibility. Portfolio planning is basically a process of determining where and how a utility will buy the performance it needs. For example, a utility can buy reliable customer service by installing large contingency margins, or through heavy dependence on automated switching and redundant circuits, or by maintaining extensive trouble call and restoration forces in the field at all times. Its portfolio reflects the way it diversifies and mixes those and other sources of performance - in this example reliability in order to obtain the spending-performance-risk profile it desires. Implemented in a way that optimizes it to the utility s needs, CBM improves the performance/cost ratio of the asset base s maintenance, and well as the asset base s performance itself, to the extent that that improvement will impact the strategic portfolio planning that revolves around where and how the utility buys the National Grid 4-9 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

391 Attachment COMM 2-8(c) Page 95 of 103 performance it wants. Utilities that implement effective CBM usually forego some spending in other areas of their portfolio because CBM means equipment care and service now provide a much better bargain and lower risk. To obtain those benefits, CBM must be implemented in a way that makes it compatible with, and integrates it into, the rest of NGRID s asset management and spending/resource commitment decision processes. Therefore, KEMA recommends that it give ELSSI (needed to drive CBM) and CBM a high priority in its technology and business process planning. National Grid 4-10 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

392 Attachment COMM 2-8(c) Page 96 of 103 Appendix A: Completed Interview Templates for Each User Group Section A.1 represents a complete set of KEMA s notes that were gathered during the interview meetings held in New England and New York while Section A.2 represents the feedback that National Grid provided to KEMA for some, but not all, of the interviews. Interviews are organized by meeting code as represented in the enclosed Microsoft Excel Spreadsheet. The spreadsheet identifies which interviews where KEMA received edits to our notes. Appendix A is contained in two separate documents in Microsoft Word format with notes and edits identified through the Track Change function turned on. A.1 KEMA s Interview Notes A.2 National Grid s Comments to KEMA Notes Both files are contained under separate cover. National Grid A-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

393 Attachment COMM 2-8(c) Page 97 of 103 Appendix B: Questionnaires & Analysis of Questionnaire Responses Two files represent appendix B. The first file is the base questionnaire that KEMA distributed to National Grid s interviewees, along with the completed questionnaires that were received by KEMA from National Grid. Both files are Microsoft Word documents and are contained under separate cover. National Grid B-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

394 Attachment COMM 2-8(c) Page 98 of 103 Appendix C: Interview Schedule & List of Interview Groups Appendix C includes the three Microsoft Word files containing the original listing of interview groups along with the original interview schedules for New England and New York. These files are contained under separate cover. National Grid C-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

395 Attachment COMM 2-8(c) Page 99 of 103 Appendix D: Information Request Form & Results Appendix D is represented by the Information Request Form as completed by National Grid for KEMA for the Benefit Cost Analysis. This file is contained under separate cover. National Grid D-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

396 Attachment COMM 2-8(c) Page 100 of 103 Appendix E: Data Requirements Matrix Appendix E contains the first draft of the Data Requirements Matrix, which represents the first pass of data requirements as a result of Phase 1. This file is contained under separate cover. National Grid E-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

397 Attachment COMM 2-8(c) Page 101 of 103 Appendix F: Priority Matrix for Data Appendix F, Priority Matrix for Data is a report contained under separate cover. National Grid F-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

398 Attachment COMM 2-8(c) Page 102 of 103 Appendix G: Initial Indications of Information Maturity Appendix G contains the final presentation that was delivered to National Grid on March 30, The presentation is provided in Microsoft PowerPoint format. This file is contained under separate cover. National Grid G-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

399 Attachment COMM 2-8(c) Page 103 of 103 Appendix H: Project Reference Materials Several files of information were provided to KEMA from National Grid and were used as foundational material during this study represent appendix H. These files are contained under separate cover. National Grid H-1 Proprietary Enterprise Level Substation Systems April 7, 2005 Integration Study Report

400 The Narragansett Electric Company d/b/a National Grid Commission 2-9 Request: Please provide a copy of the Chart on page 69 of the 2013 Electric ISR Proposal excluding PUC Major Storms. Response: The following chart provides the customer interruptions by cause for the period , excluding major storm events. CMI, 29% 2% 5% 5% 1%3% 5% 1%3% Rhode Island Customer Interruptions by Cause Excludes PUC Major Storms 2006 to % CI, 22% 6% 18% 17% 19% 5% 1% Events, 2% 7% 4% 24% 8% 7% 19% 3% 8% 11% 11% 7% 6% 7% 10% Adverse Environment Adverse Weather Animal Deteriorated Eqmt Human Element/Company Human Element/Other Intentional Lightning Substation Sub-Transmission Transmission Tree Unknown Prepared by or under the supervision of: Jennifer L. Grimsley

401 The Narragansett Electric Company d/b/a National Grid Commission 2-10 Request: Please provide a copy of the Chart included with the Company s Response to Commission DR 1-11 excluding PUC Major Storms. Response: The following chart provides the customer interruptions by cause for the period 2006 January 22, 2012, excluding major storm events. Rhode Island Customer Interruptions by Cause Excludes PUC Major Storms 2006 to 01/22/12 CMI, 29% 1% 5% 6% 1% 0%3% 22% CI, 18% 22% 7% 17% 19% 1% Events, 4% 2% 6% 24% 4% 7% 19% 4% 3% 9% 11% 6% 6% 1%3% 6% 8% 11% 6% 9% Adverse Environment Adverse Weather Animal Deteriorated Eqmt Human Element/Company Human Element/Other Intentional Lightning Substation Sub-Transmission Transmission Tree Unknown Prepared by or under the supervision of: Jennifer L. Grimsley

402 The Narragansett Electric Company d/b/a National Grid Commission 2-11 Request: Referencing National Grid s Response to Commission DR 1-11, page 2, please provide the definitions of each of the categories which clearly explains how each event is categorized. Please explain why there were no weather-related events for the period 2006 through 1/22/12. Response: Below are definitions for each of the categories used in response to Commission DR Adverse Weather as a cause category was discontinued in In deleting the Adverse Weather category, it was intended that interruptions that occur during adverse weather are categorized further as to cause, such as lightning, trees, or deteriorated equipment, for example. Information is obtained on the weather at the time of the interruption and included in the Company s Interruption Disturbance System (IDS). Adverse Environment includes the following: Moisture Interruption caused by dampness or water getting into the equipment and resulting in the equipment failing. Flooding Interruptions caused by flooding. Flying Debris Interruptions resulting from natural causes, e.g. flying or wind blown debris caused by windstorm, explosions, etc. but not vandalism. Contamination Interruption caused by tracking across contaminated insulators. Animal Interruption caused by contact with energized lines/equipment by an animal or bird. Deteriorated Equipment includes the following: Deterioration Interruption caused by corroding, rotting or aging of material, due to the normal process of time or accelerated by atmospheric conditions or other conditions. Insulation Failure Cable Interruption caused by failure of cable insulation. Use for cable failures when no other cause is apparent. Insulation Failure Other Interruption caused by dielectric breakdown of equipment insulation. Use for dielectric breakdown other than cable failures. This also applies to flashovers due to conductors off insulators. Human Element/Company includes the following: Construction by Company Interruption due to inadvertent contact during the construction or reconstruction of distribution facilities by utility system personnel. Construction by Company Contractor Interruption due to inadvertent contact during the construction or reconstruction of distribution facilities by contractors working for the company.

403 The Narragansett Electric Company d/b/a National Grid Commission 2-11 (continued, p2) Human Element/Company (continued) Control Trouble Any interruption caused by a relay, automatic throw-over device, supervisory control or other control device that causes an interruption device to operate. Relays not adjusted to established settings would be included. Distribution Transformer Overload Any transformer that fails as a result of loading above Distribution Standard. Also use when the secondary breaker in a self-protected transformer has opened and there is no evidence of trouble on the secondary system. Feeder Overload Interruption to customers caused by the operation of the circuit breaker due to a phase overload relay operation when no equipment trouble exists. Feeder Unbalance Interruption to customers caused by the operation of the breaker. Fire on Company Equipment Interruptions resulting from equipment or lines being damaged by fire due to failed equipment. Improper Application Interruption caused by the selection of the wrong piece of equipment for the task at hand; improper fuse size, improper relay setting, etc. Improper Installation Interruption caused by incorrect installation of equipment that would not be considered improper application. Example: the proper equipment not properly installed. Operating/Testing Error Interruption due to inadvertent operation of an energized device or an error committed during testing or switching. Other Company Activities Interruption caused by contact with energized lines and/or equipment by a crane, derrick, bucket truck or similar equipment operated by utility system personnel. Interruption caused by failure to properly insulate cable joints, splices, etc. or failure to apply insulation over joints or splices on spacer or aerial cable. Interruption caused by slack conductors. Interruption caused by conductors breaking due only to excessive tension. No evidence of ice, snow coating or excessive wind. Interruption caused by contact with energized lines/equipment by trees or limbs cut by company contractor. Human Element/Other includes the following: Human Contact Interruption caused by contact with energized lines/equipment by a human being who is not an employee or company contractor. Non-Company Activities Interruption due to inadvertent contact during the construction or reconstruction of distribution facilities by non-utility personnel. Interruption caused by contact with energized lines and/or equipment by a crane, derrick, bucket truck or similar equipment operated by non-utility personnel. Interruption caused by contact with energized lines/equipment by trees or limbs cut by customer or customer contractor. Vandalism Interruption caused by vandalism includes operation of switches by unauthorized persons, damage by gunfire, objects thrown onto lines and equipment, etc. Vehicle Interruption caused by a collision of a motor vehicle with distribution equipment.

404 The Narragansett Electric Company d/b/a National Grid Commission 2-11 (continued, p3) Intentional Any outage due to planned maintenance, 911 response, emergency repair work, and load shedding. Lightning A fault that occurs as a result of lightning either directly striking or inducing a voltage on a line or piece of equipment. Substation Any outage specifically associated with failed equipment at a substation. Sub-transmission Any outage caused by loss of supply on the sub-transmission system (under 69kV). Transmission - Any outage caused by loss of supply on the transmission system (69kV or greater). Tree includes the following: Tree Fell (whole tree) Interruption caused by broken tree due to natural causes. Tree Growth (tree or limb contact) Interruption caused by contact with energized lines/equipment, by movement or bending of trees or limbs due to wind, ice or snow loading. Tree Limb Interruption caused by broken limb due to natural causes. Vines Interruption caused by a plant that clings to energized lines/equipment. Unknown Interruption where diligent investigation fails to reveal the cause of the trouble. Prepared by or under the supervision of: Jennifer L. Grimsley

405 The Narragansett Electric Company d/b/a National Grid Commission 2-12 Request: Referencing the Report filed on or about November 22, 2011, in Docket No entitled Report on Tropical Storm Irene Preparedness, Damage Assessment and Service Restoration Efforts, please provide the following: (a) costs associated with transmission damage; (b) costs associated with replacing 81 distribution poles (Attachment 8); (c) costs associated with replacing 46 transformers (Attachment 9); (d) costs associated with replacing Wires listed on Attachment 10. Please separate between labor and materials. Response: Narragansett Electric has incurred a total of $2,137,000 for Hurricane Irene transmission storm damage. Attachment COMM 2-12 has been provided which contains capital costs for poles, transformers and wire, in response to parts b, c and d of this question. Please note that the quantities provided in the above referenced report were not final. Since that submission, the Company has recorded additional units through the work order as-built and close-out process which normally occurs after a major storm or any damage/failure project. The updated amounts have been included in the attachment. Prepared by or under the supervision of: Jennifer L. Grimsley

406 The Narragansett Electric Company FY 2013 Electric Infrastructure, Safety and Reliability Plan Commission 2-12 Attachment COMM 2-12 Page 1 of 1 Total Cost Quantity Total Labor Total Materials Total Cost b. Poles POLE, WOOD, 30' & UNDER POLE, WOOD, 31' - 40' ,907 25, ,885 POLE, WOOD, 41' - 50' 22 50,313 5,422 55,735 POLE, WOOD, 51' - 60' Total ,655 32, ,121 c. Transformers 10 KVA 6 19,844 4,914 24, to 50 KVA , , , KVA 1 3,307 4,144 7, KVA 1 3,307 7,299 10,607 Total , , ,847 d. Wires CABLE, NONMTL CVRD, 1/C, # , ,264 COND, BARE OR CVRD, < 1/0 60 2, ,653 COND, BARE OR CVRD, > , ,636 COND, BARE OR CVRD, > 4/ ,670 1,868 73,538 COND, BARE OR CVRD, 1/0 11, ,866 11, ,184 COND, SECNDRY CBL, 3/C, 1/0 6, ,307 13, ,190 SPACER CABLE, 1/C, 1/0 1,440 26,056 1,808 27,864 SPACER CABLE, 1/C, 336 2,013 49,804 5,472 55,276 SPACER CABLE, 1/C, ,734 2,071 17,805 Total 24, ,401 38, ,409

407 The Narragansett Electric Company d/b/a National Grid Commission 2-13 Request: Referencing the Report filed on or about January 27, 2012, in Docket No entitled Report on October 2011 Snow Event, Damage Assessment and Service Restoration Efforts, please provide information regarding the replacement of any equipment in a similar version to that provided with response to Commission DR Response: Narragansett Electric has currently identified a total of $36,000 for the October 2011 Snow Event transmission storm damage. The remaining unit cost information has been provided in Attachment COMM Prepared by or under the supervision of: Jennifer L. Grimsley

408 Attachment COMM 2-13 Page 1 of 1 The Narragansett Electric Company FY 2013 Electric Infrastructure, Safety and Reliability Plan Commission 2-13 Work order #: Total Cost Quantity Total Labor Total Materials Total Cost b. Poles POLE, WOOD, 31' - 40' 4 7, ,022 POLE, WOOD, 41' - 50' 1 1, ,176 Total 5 9, ,198 c. Transformers 10 KVA 1 3, , to 50 KVA 1 3,307 1,429 4,737 Total 2 6,615 2,248 8,863 d. Wires COND, BARE OR CVRD, < 1/ COND, SECNDRY CBL, 3/C, 1/ , ,560 Total 185 5, ,781

409 The Narragansett Electric Company d/b/a National Grid Commission 2-14 Request: On page 84 of the 2013 Electric ISR Proposal is the statement: The multiple safety and reliability goals of the I&M Program will be discernible by customers because the operating integrity of the distribution system will be raised and maintained at a relatively higher level. The validity of the I&M strategy has been demonstrated in New York during the past several years and the best practices from the Company s experience in New York have been incorporated into the roll out of the I&M Program in Rhode Island. Response: a) Please provide all studies and reports supporting this statement, particularly any adopted by the NY PSC. b) Please explain how the New York system may be different from the Rhode Island System. a) There are no studies or reports supporting the statement. The Company s Rhode Island Inspection and Maintenance program is a formalized and systematic program. The program in place today evolved from the feeder hardening programs in place in Rhode Island, New Hampshire and New York, and the mandated Inspection and Maintenance program in New York. The best practices referred to in the 2013 Electric ISR Proposal are regarding the implementation and rollout of the program in the Company s New York service territory, where the Company is in the last year of its first full five year inspection cycle. Outputs of this first cycle include the inspections database and the interface between this database and the work management system which have become more robust over the years of the program. In addition, the guidance documents provided to the inspectors have matured over the life of the program. The current guidance document is attached as Attachment COMM The guide notes changes from the previous version. b) In general, the New York and Rhode Island systems are similar, primarily comprised of 15 kv and 5 kv class overhead distribution systems of approximately the same vintages. As Rhode Island is significantly less rural than New York, the Rhode Island distribution system has shorter circuit lengths and higher customer densities than that of New York. The New York I&M program has a mandated 5 year cycle, and the Rhode Island cycle is currently a six year cycle. Prepared by or under the supervision of: Jennifer L. Grimsley

410 The Narragansett Electric Company d/b/a National Grid Attachment COMM 2-14 Please be advised that Attachment COMM 2-14 is 156 pages and the electronic version is approximately 28 MB. Due to the voluminous nature of this attachment, the Company is providing the Commission with three (3) copies and (3) CD-ROM s containing Attachment COMM 2-14.

411 The Narragansett Electric Company d/b/a National Grid Commission 2-15 Request: With regard to the proposed additional $367,000 in FY 2013 related to Post-Irene EHTM Tree Trimming, please provide a copy of any reports regarding any inspections leading to the reasonable assumption that a significant number of the remaining trees within striking distance of the line also have sustained damage. (2013 Electric ISR Proposal, p. 74). Response: There were no specific hazard tree inspections made on any of the nineteen (19) circuits selected for the Post-Irene hazard tree work. The assumption that these circuits need to be inspected for potential hazard tree conditions and mitigated for those same findings was drawn directly from the frequency of full tree failures recorded in the Company s interruption records from Tropical Storm Irene. The interruption records show that these nineteen (19) circuits experienced at least two and as much as four and one half (4 ½) full tree failures per mile during the storm. Prepared by or under the supervision of: Jennifer L. Grimsley

412 The Narragansett Electric Company d/b/a National Grid Commission 2-16 Request: Please provide the basis for the estimates leading to the proposed additional $367,000 in FY 2013 related to Post-Irene EHTM Tree Trimming. If the full $367,000 is not needed for this specified purpose, how will any excess funds be utilized? Response: The estimate of $367,000 is the product of the following three (3) multiplication factors: Company s historic average cost/tree ratio of $820 per tree removal for previous Rhode Island Enhanced Hazard Tree Management (EHTM) circuit work, one half the average trees per mile removed on those projects (1/2 of 10), and the total of number three-phase miles on the nineteen circuits chosen for the mitigation work. Thus, the formula is as follows: $820/tree x 5 trees per mile x miles = $367,000 rounded to the nearest thousand The $367,000 of proposed additional Post-Irene EHTM tree trimming is one of many components of the overall vegetation management activities included in the FY 2013 Electric ISR filing. The ISR proposal establishes an estimate of the costs to be incurred during FY 2013 to perform all vegetation management activities to maintain safe and reliable electric service for customers. Differences between estimates reflected in the FY 2013 Electric ISR and actual vegetation management costs may occur. Under spending of costs on certain proposed activities could provide an opportunity to conduct other beneficial vegetation-management activities during the fiscal year that were not contemplated in the current proposal or could serve to support over spending that may occur for other planned activities. For example, if after completing the mitigation work on the three-phase portions of the nineteen (19) identified circuits there are excess funds available then hazard tree removals may be performed on either the three phase sections of circuits adjacent to the original 19 circuits or on single phase portions of the same circuits. Any difference in total actual vegetation management costs at the end of FY 2013 as compared to the FY 2013 Electric ISR filing would be reflected in the annual reconciliation filing after the end of the fiscal year. Prepared by or under the supervision of: Jennifer L. Grimsley

413 The Narragansett Electric Company d/b/a National Grid Commission 2-17 Request: Given the amount of equipment replaced after Tropical Storm Irene, please explain how the future ISR Budget might be reduced. Response: The Company does not anticipate a significant reduction in future ISR budgets given the amount of equipment replaced after Tropical Storm Irene. As discussed in Commission 2-12, 172 poles, 80 overhead distribution transformers, and 24,293 feet (4.6 miles) of overhead primary wire were replaced in Tropical Storm Irene. This is a small percentage of this type of equipment, as in total there are 280,925 poles, 55,224 overhead distribution transformers and over 5,200 circuit miles of overhead primary in Rhode Island. Prepared by or under the supervision of: Jennifer L. Grimsley

414 The Narragansett Electric Company d/b/a National Grid Commission 2-18 Request: Referencing page 34 of the Responses to Division DR 2-3, please provide references to the specific line items comprising the $2.3 million included for UG Cable Replacements. Response: The Company s response to Division DR 2-3 states that approximately $2M has been included as specific line items in the budget, and the Company is continuing to identify additional candidates for replacement to account for the $1M in the line item, UG Cable Replacements (C OS IE UG Cable Replacement Program). The table below represents the line items in the FY13 ISR for cable replacement. FY Budget Other Proj Number Project Description FY13 Budget UG Cable C OS IE UG Cable Replacement Program 1,000,000 C A & 1102B Cable Replacement 80,000 C Cable Replacement 90,000 CD Fdr 1111 Inst Cable - Weybosset/Union Sts., Providence 400,000 CD Fdr 1135 Inst Cable - Eddy St., Providence 500,000 CD Fdr 1127 Inst Cable - Dyer/Dorrance Sts., Providence 400, W51_Carriage Drive_Replace Direct buried cable URD 489,000 PPM Grand Total 2,959,000 Prepared by or under the supervision of: Jennifer L. Grimsley

415 The Narragansett Electric Company d/b/a National Grid Commission 2-19 Request: Page 54 of the Responses to Division DR 2-9 states that the budget for the Warwick Mall substation is for a complete replacement of that substation. After further review, the Company believes it can reduce the scope significantly. Please provide the initial cost estimate and the reduced estimate. If the Warwick Mall substation costs are included in the FY 2013 Electric ISR Proposal, please provide the reference. Response: The initial cost estimate for Warwick Mall was $3.25M. The new cost estimate is $800K. These costs are not included in the FY 2013 Electric ISR Proposal, as this project is scheduled to start in FY Prepared by or under the supervision of: Jennifer L. Grimsley

416 The Narragansett Electric Company d/b/a National Grid Commission 2-20 Request: Page 108 of the Responses to Division DRs includes a chart which includes project descriptions and another description Flood Damage Avoidance Engineering Studies with increasing budgets through FY What is included in this description? Response: Included in the Flood Damage Avoidance Engineering Studies category are costs for engineering, design, and construction for the proposed flood mitigation projects being proposed by the Company. Attachment COMM 2-20 provides the original attachment referenced above in Division 2-9 (Asset Condition) and page two provides a project listing for projects included in the Flood Damage Avoidance Engineering Studies category. Prepared by or under the supervision of: Jennifer L. Grimsley

417 Attachment COMM 2-20 Page 1 of 2

418 Attachment COMM 2-20 Page 2 of 2