HOW WEBMETHODS IS USED TO SUPPORT THE ELECTRICITY AND GAS MARKETS IN AUSTRALIA.

Transcription

1 HOW WEBMETHODS IS USED TO SUPPORT THE ELECTRICITY AND GAS MARKETS IN AUSTRALIA. September 2017 WARREN VELLA PRINCIPAL INTEGRATION SPECIALIST SLIDE 1

2 TOPICS 1. Who is AEMO? 2. AEMO s Integration Objectives 3. The AEMO Integration Platform 4. AEMO s 4 Year Integration Journey 5. How AEMO achieves HA with the webmethods platform 6. How AEMO deploys changes rapidly 7. What s next for AEMO? SLIDE 2

3 WHO IS AEMO? SLIDE 3

4 WHO IS AEMO? As Australia s independent energy markets and power systems operator, AEMO provides critical planning, forecasting and power systems information, security advice, and services to the energy industry. ELECTRICITY AEMO operates Australia's National Electricity Market (NEM), the interconnected power system in Australia s eastern and south-eastern seaboard, and the Wholesale Electricity Market (WEM) and power system in Western Australia. GAS AEMO operates gas retail markets in NSW, ACT, QLD, SA, VIC and WA. AEMO also operates wholesale gas markets in south-eastern Australia. AEMO also operates the Natural Gas Services Bulletin Board (GBB), and is system operator of the Victorian Gas Declared Transmission System. SLIDE 4

5 NATIONAL ELECTRICITY MARKET (NEM) SLIDE 5

6 TWO FUELS ELECTRICITY GAS 40,000 KM / 5 min dispatch 200 Terawatt Hours 9 Million customers 7.7 billion dollars market value traded 4 hour schedules Peta Joules 4.8 Million customers 900 million dollars market value traded SLIDE 6

7 AEMO S INTEGRATION OBJECTIVES SLIDE 7

8 To improve is to change; to be perfect is to change often. W. Churchill SLIDE 8

9 AEMO S BUSINESS OBJECTIVES Promote efficient investment in and operation of Australia s electricity and gas markets for the long-term interests of consumers. Adapt to legislative and industry change quickly. Market participant and consumer satisfaction Ability to innovate SLIDE 9

10 AEMO S IT OBJECTIVES Shift from data processing and managing paper trails to management of automated processes and IT systems. The cornerstone of any automation platform is visibility Achieving agility through governance Enhance, improve and add value SLIDE 10

11 THE AEMO INTEGRATION PLATFORM SLIDE 11

12 THE AEMO INTEGRATION PLATFORM A single, centrally managed webmethods integration platform supporting both gas and electricity market systems providing: B2B Messaging User Portal CSV/XML Transformation Legacy System Inter-Operability Real-Time SLA Management API Management Platform SLIDE 12

13 FUNCTIONAL VIEW Participant A AEMO Participant B Transaction Management System Participant API Gateway* Participant A Endpoint API Hub Participant B Endpoint Participant API Gateway* Transaction Management System Message Management System Participant FTP Gateway* FTP Participant FTP Gateway* Message Management System SLIDE 13

14 EMMS SMB LOGICAL VIEW a Gas Retail Participants Electricity Retail Participants Other Markets Retailers DNSPs MDAs Others Retailers LNSPs MDPs Others Wholesale Settlements etc Protocols ebxml SOAP Services FTP New SMP Protocols Webservices FTP 3 7 MMS FTP Enterprise Interaction / Gateway 20 ebxml MSHs (Gas FRC - webmethods) Managed File Transfer New protocols Elec Gateways & Elec MSHs (webmethods) Other Markets FTP & File Servers (e.g. MIBB) Impacted SMP Data flow Gas Retail via ebxml 12 Gas Retail via FTP 8 18 MSATS Participant Server (Logical)* 23 EMMS Participant Server (Logical)* Elec Retail via Hokey Pokey Elec Webservices Other Markets a Impacted Systems a Enterprise Integration Universal Messaging & Integration Layer (webmethods) MSATS File Handlers & other File processors Gas Retail Application Gas FRC Retail Electricity Retail & Other Apps MSATS OPDMS HSM EMMS GSH GBB Other Market Apps DWGM STTM OTHERS MIBB SLIDE 14

15 VOLUME AND THROUGHPUT +650 TN Partners 12,000 messages p/minute or 200 p/second 184 production instances Integration Platform +2million API calls per day 50 production servers 5TB database SLIDE 15

16 KEY OPERATIONAL CHALLENGES Multiple SLA s Gas 99.9% Electricity 99.99% Different Maintenance Windows Outage cannot affect both fuels Different B2B Messaging Technologies Gas ebxml (SOAP) Electricity FTP and REST API s Large Data Growth GB s per day SLIDE 16

17 AEMO S 4 YEAR INTEGRATION JOURNEY SLIDE 17

18 AEMO S INTEGRATION AND API JOURNEY 2014 Renovate 2015 Integrate 2016 Strengthen 2017 Expand Re-build webmethods from 6.5 to 9.6 for SA, VIC, QLD and WA. Gas Retail Market Integration Integrate NSW & ACT B2B and B2M messaging Build ICC capability and upgrade platform from 9.6 to 9.12 Build and rollout API management platform capability and expand to electricity. Electricity Retail Market Integration SLIDE 18

19 API MANAGEMENT PLATFORM SLIDE 19

20 HOW AEMO ACHIEVES HA WITH THE WEBMETHODS PLATFORM SLIDE 20

21 THREE CORE OPERATIONAL VALUES Zero Impact Work Standards Strict Maintenance and Support Schedules Mature High Availability Practices SLIDE 21

22 ZERO IMPACT WORK STANDARDS No-outage maintenance work An issue on one fuel should not affect the other fuel The system cannot tolerate reduced capacity during maintenance work Upgrades to the database infrastructure must not force an outage to the applications SLIDE 22

23 STRICT MAINTENANCE AND SUPPORT SCHEDULES Platform Stability Security and Reliability Continuous Integration 24/7 on-call support The platform is upgraded every 12 months Product patches and fixes are rolled out every 3 months Codes fixes are rolled out on-demand using CI/CD practices 24/7 on-call support with end-to-end SLA monitoring Integration Competency Centre SLIDE 23

24 Site Transfer Automation Hot Standby MATURE HIGH AVAILABILITY PRACTICES Validate nearinstant recovery every 6 weeks Minimise manual deployment and transfer steps Applications in the inactive site are always running SLIDE 24

25 HOW DO WE EXECUTE SITE TRANSFERS? SLIDE 25

26 THE AEMO SITE TRANSFER Inbound Load ESB B2B ESB B2B API DB 5 min replication API DB Portals Market Systems Portals Market Systems NSW QLD SLIDE 26

27 THE AEMO SITE TRANSFER Inbound Load ESB B2B ESB B2B API DB 5 min replication API DB Portals Market Systems Portals Market Systems NSW QLD SLIDE 27

28 THE AEMO SITE TRANSFER Inbound Load ESB B2B ESB B2B API DB 5 min replication API DB Portals Market Systems Portals Market Systems NSW QLD SLIDE 28

29 THE AEMO SITE TRANSFER Inbound Load ESB B2B ESB B2B API DB 5 min replication API DB Portals Market Systems Portals Market Systems NSW QLD SLIDE 29

30 THE AEMO SITE TRANSFER Inbound Load ESB B2B ESB B2B API DB 5 min replication API DB Portals Market Systems Portals Market Systems NSW QLD SLIDE 30

31 THE AEMO SITE TRANSFER Inbound Load ESB B2B ESB B2B API DB 5 min replication API DB Portals Market Systems Portals Market Systems NSW QLD SLIDE 31

32 Integration Platform SITE TRANSFER BY ZONES API Management (API) API Enterprise Gateway, API Gateway Enterprise Service Bus (ESB) Integration Server, Universal Messaging Business to Business (B2B) Enterprise Gateway, Trading Networks, ebxml Portal My webmethods Server, API Portal Reporting Optimise, Mash, CentraSite, Command Central Continuous Integration (CI) Deployer, Asset Build Environment, Jenkins SLIDE 32

33 SITE TRANSFER BY BAYS Bay A Bay A Bay B Bay B NSW QLD SLIDE 33

34 SITE TRANSFER BY BAYS Bay A Bay A Bay B Bay B NSW QLD SLIDE 34

35 SITE TRANSFER BY BAYS Bay A Bay A Bay B Bay B NSW QLD SLIDE 35

36 SITE TRANSFER BY BAYS Bay A Bay A Bay B Bay B NSW QLD SLIDE 36

37 SITE TRANSFER BY BAYS Bay A Bay A Bay B Bay B NSW QLD SLIDE 37

38 HOW DO WE DEPLOY CHANGES RAPIDLY? SLIDE 38

39 CONTINUOUS INTEGRATION SLIDE 39

40 WHAT S NEXT FOR AEMO? SLIDE 40

41 WHATS NEXT FOR AEMO? Advanced automation, automatic transfers Quarterly patching cycles Extension of the API s to the Business to Market messages (B2M) Maturing continuous integration, for quicker deployments webmethods 10 upgrade, quick provision and tear downs Next wave: Building enhanced API services SLIDE 41

42 QUESTIONS? SLIDE 42