25years. MQ High Availability. Celebrating. David Ware IBM MQ Chief Architect. MQ Technical Conference v

Size: px

Start display at page:

Download "25years. MQ High Availability. Celebrating. David Ware IBM MQ Chief Architect. MQ Technical Conference v"

Brooke Perry
5 years ago
Views:

1 MQ Technical Cnference v MQ High Availability David Ware IBM MQ Chief Architect Celebrating 25years

Highly Available A requirement, r a capability, f a system t be peratinal fr a greater prprtin f

2 High availability Available A system is said t be available if it is able t perfrm its required functin, such as successfully prcess requests frm users. Highly Available A requirement, r a capability, f a system t be peratinal fr a greater prprtin f time than is cmmn fr ther, less imprtant, systems. A sliding scale 0% 100% Often, greater availability means greater cmplexity and cst 2

9999% ~ 18 days < 4 days < 2 days < 9 hurs ~ 1 hur ~ 5 minutes ~ 30 secnds 5 nines Likelihd f

3 Measuring availability Target Yearly utage Impacts n availability lying maintenance 95% 99% 99.5% 99.9% 99.99% % % ~ 18 days < 4 days < 2 days < 9 hurs ~ 1 hur ~ 5 minutes ~ 30 secnds 5 nines Likelihd f utages (meantime t failure, and speed f recvery) Operatinal errrs Overall availability is the cmbined availability f all cmpnents The platfrm The middleware The applicatins 3

4 Messaging system availability Asynchrnus messaging can imprve applicatin availability by prviding a buffer but the messaging system itself must be highly available t achieve that Redundancy Multiple active ptins available fr applicatins t cnnect Ruting Ability t rute messages arund failures Message availability Critical messages are nt lcked t a single runtime and quickly available frm elsewhere 4

5 Messaging system availability Asynchrnus messaging can imprve applicatin availability by prviding a buffer but the messaging system itself must be highly available t achieve that Redundancy Multiple active ptins available fr applicatins t cnnect Ruting Ability t rute messages arund failures Message availability Critical messages are nt lcked t a single runtime and quickly available frm elsewhere Required fr highest system availabilities Only required fr certain message flws 5

Messaging system availability MQ Asynchrnus messaging can imprve applicatin availability by prviding a buffer but the messaging system itself must be highly available t achieve that Redundancy

6 Messaging system availability MQ Asynchrnus messaging can imprve applicatin availability by prviding a buffer but the messaging system itself must be highly available t achieve that Redundancy Multiple active ptins available fr applicatins t cnnect Ruting Ability t rute messages arund failures Message availability Critical messages are nt lcked t a single runtime and quickly available frm elsewhere licatin client cnnectivity MQ Cluster ruting MQ HA 6

7 licatin client cnnectivity

Decuple the applicatins frm queue managers

managers, always cnnecting as MQ clients,

8 Decuple the applicatins frm queue managers licatins lcally bund t a queue manager will limit the availability f the slutin. Running applicatins remte frm the queue managers, always cnnecting as MQ clients, decuples the applicatin and system runtimes, enabling higher availability.

9 Decuple the applicatins frm queue managers Step 1 Cnnect the applicatin as a client Benefits: Ability t supprt slutins where a queue manager may fail-ver between systems (mre later). Separates applicatin system requirements frm the queue manager s, reducing maintenance cnflicts and therefre, availability. Restart times n either side can be reduced. Shuld be relatively invisible t the applicatin Dn t hardcde that cnnectin cnfiguratin! Use client aut-recnnect t hide a queue manager restart frm the applicatin Qmgr

10 Decuple the applicatins frm queue managers Step 2 Allw the applicatin t cnnect t a set f queue managers Benefits: licatins can cntinue t interact with MQ even whilst a queue manager is failing ver r unavailable during maintenance With multiple applicatins cnnected, nly a subset will be impacted by a queue manager utage Hw des yur applicatin find the queue manager? Netwrk ruting Cnnectin name lists Client Channel Definitin Tables The applicatin may need t re-evaluate hw it explits all f MQ s capabilities Message rdering may change if it is currently expected acrss cnnectins licatins may be reliant n transitry state: Dynamic queues and subscriptins Reply messages XA transactin recvery Qmgr Qmgr Qmgr This might nt wrk fr all applicatins

high availability and wrklad balancing f clients licatins simply cnnect t an abstracted queue manager name (which desn t need t be the actual queue manager name use

11 What d CCDTs enable? These prvide encapsulatin and abstractin f cnnectin infrmatin fr applicatins, hiding the MQ architecture and cnfiguratin frm the applicatin They als enable security, high availability and wrklad balancing f clients licatins simply cnnect t an abstracted queue manager name (which desn t need t be the actual queue manager name use an * ) CCDT defines which real queue managers the applicatin will cnnect t. Which culd be a single queue manager r a grup f licatin 1 cnnect: *QMGRP1 licatin 2 cnnect: *QMGRP2 licatin 2 cnnect: *QMGRP2 CCDT QMGRP1: QM1 QMGRP2: QM2 QMGRP2: QM3 QM1 QM2 Acrss a grup, selectin can be rdered r randmised and weighted QM3

Creating the CCDTs CCDTs can represent cnnectin details t multiple queue managers CLNTCONN channels are defined t identify the SVRCONN channels Create the s and define their SVRCONNs

purpse MQ V8+: Use runmqsc n t remve the need fr a queue manager A single CCDT fr yur MQ estate r ne per applicatin?

12 Creating the CCDTs CCDTs can represent cnnectin details t multiple queue managers CLNTCONN channels are defined t identify the SVRCONN channels Create the s and define their SVRCONNs Define multiple CLNTCONNs in a central place t generate the CCDT It desn t have t be any f the queue managers wning the SVRCONNs Pre-MQ V8: Yu needed a dedicated queue manager fr this purpse MQ V8+: Use runmqsc n t remve the need fr a queue manager A single CCDT fr yur MQ estate r ne per applicatin? A single CCDT can be easier t create but updates can be expensive Separate CCDTs make it easier t update when an applicatin s needs change Centrally define all the CLNTCONNs that represent all the s Take the CCDT and make available t the cnnecting applicatins

Accessing the CCDTs QMGR1 QMGR2 QMGR3 QMGR1 QMGR2 QMGR3 QMGR1 QMGR2 QMGR3 QMGR1 QMGR2 QMGR3 CCDT files need t be accessible t the applicatins cnnecting t MQ Either accessible thrugh the client s

13 Accessing the CCDTs QMGR1 QMGR2 QMGR3 QMGR1 QMGR2 QMGR3 QMGR1 QMGR2 QMGR3 QMGR1 QMGR2 QMGR3 CCDT files need t be accessible t the applicatins cnnecting t MQ Either accessible thrugh the client s filesystem User needs t manage distributin f CCDT files themselves Or remtely ver HTTP r FTP Available fr JMS/XMS applicatins fr a number f releases Added fr MQI applicatins in MQ V9 LTS CCDT QM1 QM2 QM3 HTTP server CCDT QM1 QM2 QM3? CCDT CCDT CCDT CCDT CCDT MQI MQCONN(QMGRP2) MQI MQCONN(QMGRP2) MQI MQCONN(QMGRP2) MQI MQCONN(QMGRP2) MQI MQCONN(QMGRP2) MQI MQCONN(QMGRP2) MQI MQCONN(QMGRP2) MQI MQCONN(QMGRP2) QM1 QM2 QM3 QM1 QM2 QM3

14 MQ Cluster ruting

Ruting n availability with MQ Clusters MQ Clusters prvide a way t rute messages based n availability In a cluster there can be multiple ptential targets fr any message.

15 Ruting n availability with MQ Clusters MQ Clusters prvide a way t rute messages based n availability In a cluster there can be multiple ptential targets fr any message. This alne can imprve the availability f the slutin, always prviding an ptin t prcess new messages. A queue manager in a cluster als has the ability t rute new and ld messages based n the availability f the channels, ruting messages t running queue managers. 1 1 Client Queue Manager Queue Manager Queue Manager Queue Manager Clustering can als be used t rute messages t active cnsuming applicatins. Service Service Service 15 Clustering is used by many custmers wh perate critical services at scale Available n all supprted MQ platfrms Sessin: Build Scalability and Availability int Yur Messaging Slutins with IBM MQ Clustering

16 MQ HA (message availability)

17 Message high availability The prblem The bjective Cnsider a single message Tied t a single runtime, n a single piece f hardware Any failure lcks it away until recvery cmpletes Messages are nt tied t a single anything In the event f a failure, there is a fast rute t access the message 17

Message high availability Active / active messages Active / passive messages Any message is available frm any runtime at any time Crdinated access t each message A failed runtime des nt prevent

18 Message high availability Active / active messages Active / passive messages Any message is available frm any runtime at any time Crdinated access t each message A failed runtime des nt prevent access t a message by anther runtime Messages are highly available, thrugh replicatin Only ne runtime is the leader and has access t the messages at a time A failure results in a new leader taking ver 18

19 Message high availability Active / active messages Active / passive messages IBM MQ fr z/os shared queues IBM MQ Distributed HA slutins Kafka replicatin RabbitMQ mirrring 19

MQ fr z/os shared queues Available with z/os parallel sysplex A tightly cupled cluster f independent z/os instances MQ Multiple queue-sharing managers grups are members (QSGs) f a queue sharing grup

20 MQ fr z/os shared queues Available with z/os parallel sysplex A tightly cupled cluster f independent z/os instances MQ Multiple queue-sharing managers grups are members (QSGs) f a queue sharing grup (QSG) Shared queues are held in the Parallel SysPlex Cupling Facility A highly ptimised and resilient z/os technlgy All queue managers in a QSG can access the same shared queues and their messages Cupling Facility Benefits: ü Messages remain available even if a queue manager fails ü Pull wrklad balancing ü licatins can cnnect t the grup using a QSG name ü Remves affinity t a specific queue manager Queue Sharing Grup 20

IBM MQ Distributed HA slutins Externally managed External mechanisms are relied n t prtect the data and prvide autmatic takever capabilities MQ managed The

21 IBM MQ Distributed HA slutins Externally managed External mechanisms are relied n t prtect the data and prvide autmatic takever capabilities MQ managed The resilient data and the autmatic takever is prvided by the MQ system new System managed HA Multi-instance queue managers MQ liance Replicated data queue managers 21

22 Externally managed HA

23 System managed HA The HA manager mnitrs the MQ system (e.g. a queue manager in a VM r cntainer), n detecting a failure it will start a new system, remunt strage and rerute netwrk traffic active HA manager passive Relies n external, highly available, strage (e.g. SAN) active passive A queue manager is unaware f the HA system Availability depends n speed t detect prblems and t restart all layers f the system required (e.g. VM and queue manager) Examples: HA Clusters Veritas Cluster Server, IBM PwerHA, Micrsft Cluster Server Clud platfrms IBM Clud, AWS EC2, Azure Cntainers Kubernetes, Dcker Swarm Sme systems can be relatively slw t restart Additinal cst f infrastructure Multiple mving parts t cnfigure and manage 23

24 Multi-instance queue managers All queue manager data is held n netwrk attached strage (e.g. NFS, IBM Spectrum Scale). Tw systems are running, bth have an instance f the same queue manager, pinted t the same strage. One is active, the ther is in standby. A failure f the active instance is detected by the standby thrugh regularly attempting t take filesystem lcks. The queue manager with the lcks is the active instance. active active active passive Faster takever, less t restart Cheaper less specialised sftware r administratin skills needed Wide platfrm cverage, Windws, Unix, Linux Only as reliable as the netwrk attached strage Matching the MQ requirements t filesystem behaviur can be tricky N IP address takever, use client cnfiguratin instead 24

25 MQ managed HA

IBM MQ liance A pair f MQ liances are cnnected tgether and cnfigured as an HA grup active Queue managers created n ne appliance can be autmatically replicated, alng

s bth appliances can run wrklad (active / active lad) Optinal IP address assciated with an HA queue manager, autmatically adpted by the active HA appliance single

26 IBM MQ liance A pair f MQ liances are cnnected tgether and cnfigured as an HA grup active Queue managers created n ne appliance can be autmatically replicated, alng with all the MQ data, t the ther liances mnitr each ther active Autmatic failver, plus manual failver fr migratin r maintenance Independent failver fr queue managers s bth appliances can run wrklad (active / active lad) Optinal IP address assciated with an HA queue manager, autmatically adpted by the active HA appliance single lgical endpint fr client apps active passive N persistent data lss n failure N external strage N additinal skills required 2018 IBM Crpratin MQ Technical Cnference v

Replicated Data Queue Managers Linux nly, MQ Advanced HA slutin with n need fr a shared file system r HA cluster MQ cnfigures the underlying resurces t make setup and peratins natural t an MQ user

27 Replicated Data Queue Managers Linux nly, MQ Advanced HA slutin with n need fr a shared file system r HA cluster MQ cnfigures the underlying resurces t make setup and peratins natural t an MQ user Three-way replicatin fr qurum supprt Synchrnus data replicatin fr nce and nce nly transactinal delivery f messages Active/passive queue managers with autmatic takever Per queue manager cntrl t supprt active/active utilisatin f ndes Per queue manager IP address t prvide simple applicatin setup Supprted n RHEL v7 x86-64 nly Nde 1 Netwrk Nde 2 Nde 3 Mnitring New in V9.0.4 CD / V9.1 LTS MQ Advanced fr Linux Synchrnus data replicatin 27 MQ HA Grup 2018 IBM Crpratin MQ Technical Cnference v

28 Replicated Data Queue Managers Linux nly, MQ Advanced HA slutin with n need fr a shared file system r HA cluster MQ cnfigures the underlying resurces t make setup and peratins natural t an MQ user Three-way replicatin fr qurum supprt Synchrnus data replicatin fr nce and nce nly transactinal delivery f messages Active/passive queue managers with autmatic takever Per queue manager cntrl t supprt active/active utilisatin f ndes Per queue manager IP address t prvide simple applicatin setup Supprted n RHEL v7 x86-64 nly Nde 1 Netwrk Nde 2 Nde 3 Mnitring New in V9.0.4 CD / V9.1 LTS MQ Advanced fr Linux Synchrnus data replicatin 28 MQ HA Grup 2018 IBM Crpratin MQ Technical Cnference v

29 Replicated Data Queue Managers Recmmended deplyment pattern: Spread the wrklad acrss multiple queue managers and distribute them acrss all three ndes Nde 1 Netwrk Nde 2 Nde 3 Even better, mre than ne queue manager per nde fr better failver distributin Use MQ Clusters fr additinal ruting f messages t wrk arund prblems MQ licensing is aligned t maximise benefits QM 1 One full IBM MQ Advanced license and tw High Availability Replica licenses (previusly named Idle Standby) 29 MQ HA Grup 2018 IBM Crpratin MQ Technical Cnference v

30 External/MQ managed HA with RDQM Disaster recvery CD MQ Advanced added the ability t build a lser cupled pair f ndes fr data replicatin with manual failver Data replicatin can be Asynchrnus fr systems separated by a high latency netwrk Synchrnus fr systems n a lw latency netwrk N autmatic takever means n need fr a third nde t prvide a qurum Nde 1 Nde 2 data replicatin IBM Crpratin MQ Technical Cnference v Think 2018 / March 21, 2018 / 2018 IBM Crpratin

31 Cst f a restart

Speed f failver System managed Multi-instance queue managers MQ liance Starting a whle VM can be slw Cntainers

failure Restart underlying system Start queue manager Recver messaging state Recnnect applicatins Message state

can significantly increase the time needed Replicated data queue managers Recnnecting applicatins The

32 Speed f failver System managed Multi-instance queue managers MQ liance Starting a whle VM can be slw Cntainers can b e much quicker Recvering data frm netwrk attached strage can be slw Reliant n strage cnfiguratin Detect failure Restart underlying system Start queue manager Recver messaging state Recnnect applicatins Message state recvery This time is very dependent n the wrklad f the queue manager High persistent traffic lad and deep queues can significantly increase the time needed Replicated data queue managers Recnnecting applicatins The applicatins must als detect the failure befre attempting t recnnect Make sure the channel heartbeat interval is set suitably lw 42

Multi Instance vs. RDQM The speed f the filesystem plays a large part in the speed f failver Misleading number due t reductin in lad thanks t slwer filesystem.

33 Multi Instance vs. RDQM The speed f the filesystem plays a large part in the speed f failver Misleading number due t reductin in lad thanks t slwer filesystem. Reality is that it ll be even slwer Tw lad factrs effect the restart time: Hw much inflight message traffic at the time f the failure The amunt f message data stred n active queues The faster filesystem pssible with RDQM can significantly reduce fail-ver times Even if the system is busy, a cntrlled switch-ver gives MQ the chance t fflad the inflight wrk Multi instance RDQM 43

34 What, where?

35 Which HA fits where Shared queues System managed Multi instance RDQM liance HA z/os Distributed platfrms ** Cntainers * MQ liance * This will depend heavily n the capabilities f the cntainer management layer ** RHEL x86 nly 45

36 Pulling it all tgether

37 Building a highly available system Decuple the applicatins frm the underlying MQ infrastructure Service requestr r event emitters MQ Service prvider r lng running cnsumer 2018 IBM Crpratin MQ Technical Cnference v

38 Building a highly available system High availability f the individual MQ runtimes shuld be baked int the design Remve any single pint f failure Make each queue manager highly available Have multiple equivalent queue managers 2018 IBM Crpratin MQ Technical Cnference v

Building a highly available system The applicatins shuld be designed and cnfigured t maximise the availability f the MQ runtime Cnnect instances f yur service

39 Building a highly available system The applicatins shuld be designed and cnfigured t maximise the availability f the MQ runtime Cnnect instances f yur service applicatins t mre than ne queue manager Use CCDTs and design applicatins t be able t cnnect t ne f many queue managers 2018 IBM Crpratin MQ Technical Cnference v

40 Building a highly available system Separating ut the MQ infrastructure int layers can be used t imprve availability further by minimising impact t end users Separate ut the requestr/emitter queue managers frm the service prvider queue managers Use an MQ cluster t wrklad balance and dynamically rute wrk t the service prviders 2018 IBM Crpratin MQ Technical Cnference v

41 Building a highly available system Reading material ibm.biz/mq_hubs ibm.biz/mqaas_red 2018 IBM Crpratin MQ Technical Cnference v

42 MQ Technical Cnference v Celebrating 25years 2018 IBM Crpratin 52

App Orchestration 2.6

App Orchestration 2.6 App Orchestratin 2.6 Terminlgy in App Orchestratin 2.6 Last Updated: July 8, 2015 Page 1 Terminlgy Cntents Elements f App Orchestratin... 3 Dmains... 3 Multi-Datacenter Deplyments... 4 Delivery Sites...