IBM MQ Latest and Greatest Enhancements

Transcription

1 IBM MQ Latest and Greatest Enhancements Jamie Squibb IBM UK 7 November 2017 Session JE

2 MQ Runs Exactly How and Where You Need It Cloud Azure Message Hub IBM Bluemix AWS Google On-premises Linux Windows Private cloud HPUX Solaris Distributed platforms IBM MQ Appliance 2

3 IBM MQ We are here MQ V8.0.0 MQ V MQ V MQ V MQ V9.0.0 MQ V9.0.1 MQ V9.0.2 MQ V9.0.3 MQ V9.0.4 IBM MQ Appliance M2000 IBM MQ Appliance M was the start of MQ s continuous delivery model Long Term Support for static feature content Continuous Delivery for the latest capabilities 3

4 End of Service for the old versions MQ V7.0.1 MQ AMQ V7.0.1 MQ FTE V7.0.4 MQ V7.1 MQ V7.5 MQ HVE V7.0.1 WebSphere MQ 7.1 End of Service (Distributed) was April 2017 End of Service (z/os VUE) was September 2017 End of Service (z/os) will be November 2017 WebSphere MQ 7.5 End of Service (Distributed) will be April 2018 MQ FTE V7.0.x, MQ AMS 7.0.x & MQ HVE EOS was September

5 Continuous delivery 5

6 Previously: Service and continuous delivery combined Version 7.5 Fixes Self contained new function (V8) Significant new features added at GA of each release LDAP authority checks PAM authentication Start events Version 8 Capped message expiry Security change config events DB passwords Q&A: 6

7 Today: Service and continuous delivery separated Version 9 LTS, MQ long term support CD CD Incremental new function. New delivery every few months. Supported for duration of LTS. No fixpacks. Fixes delivered on latest mods only. Only available on Linux, Windows, z/os, AIX and MQ Appliance CD You re welcome to mix and match 9.0.n CD Version 9+ LTS, MQ long term support Fixes only. No mid-service function. Same 5+3 service lifetime. LTS releases every couple of years containing a roll up of CD function CD CD Q&A: CD 7

8 IBM MQ V9 LTS (in a slide) 8

9 Time to complete MQ V9 Long Term Support Advanced Message Security IBM MQ Advanced AMS added a new high performance policy for encryption, not signing. 3x-4x performance gains are typical, if not more Support for non-ibm JREs Centralised hosting of CCDTs Clients can automatically retrieve a remotely located CCDT over HTTP or FTP when connecting. Enabler MQI App MQCONN(QMGR2)? HTTP server CCDT QMGR1 QMGR2 QMGR3 for dynamically changing MQ configurations and a QMGR1 QMGR2 QMGR3 simpler application setup System topics Distributed queue managers publish statistics and application trace data to dynamic subscribers 9

10 IBM MQ V9.0.x CD 10

11 IBM MQ 9.0.x CD releases MQ V8.0.0 MQ V MQ V MQ V MQ V9.0.0 MQ V9.0.1 MQ V9.0.2 MQ V9.0.3 MQ V9.0.4 IBM MQ Appliance M2000 IBM MQ Appliance M2001 V9.0.1 CD V9.0.2 CD V9.0.3 CD November 2016 March 2017 July 2017 NEW V9.0.4 CD November

12 MQ release-to-release changes Always read the What s new and changed sections of the Knowledge Centre to see what each release adds 12

13 Managing MQ 13

14 Managing your MQ estate Management of MQ is evolving into a cattle not pets model. Consistent configuration and operations across multiple queue managers Automated deployment Configuration as code Self service Collection and analysis of diagnostic data Simple integration into standard devops and automation tooling 14

15 Software MQ Web Console V9.0.1 CD In 2015 the IBM MQ Appliance introduced a web based admin console. MQ saw this across all CD platforms (Linux, z/os, Windows) Point a browser at the MQ installation to create and manage queue managers and their resources Provides a simple way to access MQ resources Introducing new roles such as read-only Will not see the full richness of capability from MQ Explorer Targeted at each individual installation, not a network of MQ systems 15

16 Behind the scenes MQ installs a copy of WebSphere Liberty Profile (WLP) for the MQ installation s use MQ masks WLP s complexity for those that don t need it A small subset of WLP s configuration is exposed MQ commands manage the WLP runtime strmqweb, dspmqweb, endmqweb System MQ Installation HTTP server (WebSphere Liberty Profile) Queue manager Queue manager Security options to suit your setup Basic, LDAP, SAF Web Browser Rest Interface MQ Console Queue manager Assign users to roles Full administrator Read only Fine grained MQ access control This enables an even more exciting capability for MQ - REST 16

17 RESTful Administration MQ has supported scripting and programmatic administration for many, many years, but it is a little proprietary MQSC and PCF are powerful, but can be hard to get going with. They also require MQ tooling to be in place to use (runmqsc, MQ clients) MQ is rolling out support for a RESTful administrative API to enable much of what s available today with MQSC and PCF, but in a more intuitive way to many Being over HTTPS also enables the embedding of MQ administrative operations into many environments and tools that previously would not be possible HTTP POST send JSON GET receive JSON PATCH send JSON DELETE V9.0.1 CD V9.0.2 CD V9.0.3 CD V9.0.4 CD MQSC DEFINE DISPLAY UPDATE DELETE 17

18 Using REST Based off underlying MQ capabilities such as PCF and control commands, but adjusted to adhere to RESTful practices V9.0.1 CD V9.0.2 CD V9.0.3 CD V9.0.4 CD URL represents target object for command HTTPS GET: HTTPS POST: { name : QUEUE.1, type : local, cluster : { name : CLUSTER1 } } HTTP Response: 201 { qmgr : [ { qmgrname : QMTEST01, status : running }, { qmgrname : QMTEST02, status : ended } ] } 18 JSON payload when defining/updating objects HTTP response indicates success/failure JSON payload returned when querying

19 MQSC for REST V9.0.4 CD Tailored RESTful support for individual MQ objects and actions are in the works However, to speed up full MQ admin support over REST we will be adding the ability to submit arbitrary MQSC commands over REST Gives complete MQSC coverage quickly Simple to convert existing scripts Does not benefit from improved usability HTTPS POST: { } type : runcommand, parameters : { command : STOP CHANNEL(CHANNEL.TEST) } { commandresponse : [{ completioncode : 0, reasoncode : 0, text : [ AMQ8019: Stop IBM MQ channel accepted. ] }], overallcompletioncode : 0, overallreasoncode : 0 } Stopping a channel 19

20 Enabling your whole estate for REST administration REST REST REST Option 1 Administer each MQ installation separately, they must all be on the MQ 9.0.x CD release HTTP Server QM QM HTTP Server QM QM HTTP Server QM QM Option 2 REST V9.0.4 CD Manage a network of systems through gateway entry points HTTP Server Not every queue manager will need to expose HTTPS endpoints QM Pre-9.0.x queue managers are able to be administered through those 9.0.x gateways QM QM QM 20

21 Managing diagnostic data V9.0.3 CD V9.0.4 CD MQ delivered improvements to the error log content Universal timestamps Message severity 05/10/ :22:01 - Process( ) User(another) Program(runmqchi.exe) Host(DUMMY) Installation(MQNI ) VRMF( ) QMgr(QM1) Time( T11:22:01.376Z) ArithInsert1(0) ArithInsert2(0) AMQ9542W: Queue manager is ending. EXPLANATION: The program will end because the queue manager is quiescing. ACTION: None. 21

22 Distributed error Log Enhancements V9.0.3 CD V9.0.4 CD Modifications to the error log content added to aid central log collection and error detection Universal timestamps added /11/ :13:44 PM - Process(429.1) User(bld-adm) Program(amqzxma0) Host(c5ab153621a5) Installation(Docker) VRMF( ) QMgr(QM1) AMQ5008E: Time( T11:13:44.705Z) Time( T11:13:44.705Z) AMQ5008E: An essential IBM MQ process 442 (zllcrit) cannot be found and is assumed to be terminated. Ability to add severity information to the message codes 22

23 High availability 23

24 High availability, it s no longer a luxury High availability is more important then ever. Users expect their messaging systems to always be available for work and for messages to reach their destination no matter what happens. QM1 QM1 MQ delivers HA through the ability to build horizontally scaled, active-active systems and typically activepassive HA of the data itself*, the messages. Traditionally active-passive HA has been achieved through HA clusters or multi instance queue managers. Both rely on highly available infrastructure to be setup and relied on. Shared storage The MQ Appliance changed this with a fully integrated HA solution, providing built in machine to machine data replication and failover. * z/os shared queues provide active-active HA of the message data 24

25 Data replicated HA for Software MQ on Linux The IBM MQ Appliance introduced shared nothing HA through direct data replication One way to build your own equivalent is to look to a DRBD based solution (guidance given ) But we re working on an HA solution to provide this capability built directly into IBM MQ Advanced 25

26 26 Replicated Data Queue Managers A Linux only HA solution with no external dependency No requirement for a network filesystem or for an HA cluster Leverages the proven data replication and monitoring capabilities of DRBD and Pacemaker MQ configures the underlying resources to make setup and operations natural to an MQ user Three-way replication for quorum support Synchronous data replication for once and once only delivery of messages Active/passive queue managers with automatic takeover Per queue manager control to support active/active utilisation of nodes Floating IP support per queue manager to provide simple application setup Delivered through CD, initial focus on RHEL v7 on x86-64 only QM1 QM1 App 1 Pacemaker Monitoring QM1 QM1 IBM MQ Advanced Node 1 Node 2 Node 3 QM1 QM1 DRBD DRBD DRBD V9.0.4 CD

27 27 Replicated Data Queue Managers A Linux only HA solution with no external dependency No requirement for a network filesystem or for an HA cluster Leverages the proven data replication and monitoring capabilities of DRBD and Pacemaker MQ configures the underlying resources to make setup and operations natural to an MQ user Three-way replication for quorum support Synchronous data replication for once and once only delivery of messages Active/passive queue managers with automatic takeover Per queue manager control to support active/active utilisation of nodes Floating IP support per queue manager to provide simple application setup Delivered through CD, initial focus on RHEL v7 on x86-64 only Node 1 Node 2 Node 3 QM1 QM1 App 1 Pacemaker Monitoring QM1 QM1 IBM MQ Advanced QM1 QM1 DRBD DRBD DRBD V9.0.4 CD

28 Replicated Data Queue Managers IBM MQ Advanced App 1 App 2 App 3 V9.0.4 CD Recommended deployment pattern: Spread the workload across multiple queue managers and distribute them across all three nodes Node 1 Node 2 Node 3 Pacemaker Monitoring QM1 QM1 QM1 No one node runs at total capacity, allowing for failed over queue managers when required QM2 QM3 QM2 QM3 QM2 QM3 MQ licensing is aligned to minimise costs One full IBM MQ Advanced license and two High Availability Replica licenses (previously named Idle Standby) QM1 QM2 QM3 QM1 QM2 QM3 QM1 QM2 QM3 DRBD DRBD DRBD 28

29 Supporting your applications 29

30 Multiple APIs and Protocols APIs and Protocols IBM MQ supports multiple APIs and multiple client protocols. Both proprietary and open. APIs: MQI, JMS, MQ Light, REST Protocols: MQ, AMQP, MQTT, HTTP These support a wide range of application styles, from the simplest of messaging needs through to the richest MQ is the transport, messages produced from any API or protocol can be received by any other API or protocol. MQI Exposes the full set of MQ capabilities Uses the MQ protocol JMS 2.0 Supports the full JMS API for use in many JSE or JEE environments Uses the MQ protocol MQ Light A simple pub/sub messaging API Uses the AMQP 1.0 protocol MQ MQ s highly reliable and performant messaging protocol MQTT Supports the open standard MQTT API and protocol Open source Eclipse Paho clients AMQP Support for AMQP 1.0 enables support for open source clients such as Qpid Proton REST A very simple but secure messaging API over REST 30

31 REST Messaging Many users just want the simplest way to get messages in and out of an MQ system App App HTTPS POST /qmgr/qm1/queue/q1/message DELETE /qmgr/qm1/queue/q1/message A RESTful API gives you just that. Easily enabling messaging from just about any environment with no need for an MQ client or platform/language limitations App server REST API MQ QM1 MQ originally exposed its messaging capabilities using the IBM MQ bridge for HTTP. This requires your own HTTP server and setup. This feature has been deprecated since MQ V8. MQ support for z/os Connect now provides a similar mechanism on z/os. Q1 Existing channels MQ App App MQPUT(Q1) MQGET(Q1) Neither provide all the components, ready configured, inside MQ. 31

32 REST Messaging The new HTTP server support in MQ 9.0.x provides the platform for a properly integrated REST API solution App App HTTPS V9.0.4 CD POST /qmgr/qm1/queue/q1/message DELETE /qmgr/qm1/queue/q1/message CD releases start to see a messaging REST API evolve, first with simple synchronous point-to-point support. REST API QM1 Q1 Existing channels MQ App App MQPUT(Q1) MQGET(Q1) 32

33 Connectivity 33

34 Bridging MQ with external systems As well as connecting a wide array of applications directly to an MQ system, there are a growing set of bridges between MQ and external systems, many of them cloud services Message Hub Connect MQ to the Bluemix messaging service Salesforce Integrate MQ s publish/subscribe with Salesforce events Blockchain Use MQ messages to query and update the blockchain Kafka Use open source Kafka connectors to join MQ with Kafka IBM Integration Bus IBM s integration solution for everything else! 34

35 MQ Bridge for Salesforce V9.0.2 CD V9.0.4 CD Safesforce s cloud-based CRM platform enables events to be emitted when changes are made to data, or when applications run. You can inject these Salesforce events into your own systems using the new MQ Bridge for Salesforce with no need for your backend applications to connect to Salesforce. Similarly, events originating in MQ can be injected into Salesforce. PushTopic Platform Event IBM MQ Bridge for Salesforce MQ Publication The bridge runs on Linux, but connects to any queue manager and is enabled for monitoring with system topic metrics. IBM MQ 35

36 Blockchain Bridge MQ is a natural fit to connect existing business transaction systems to remote intra-business ledger services Asynchronous request reply MQ message flow for applications to request and update information from Blockchain over MQ messages Originally provided with MQ Advanced for z/os VUE V9.0.3, more recently with MQ Linux server and client images Connects to Hyperledger Fabric networks in IBM Bluemix and locally Supported for use with 9.0.x MQ Advanced queue managers IBM MQ Advanced IBM MQ Bridge for Blockchain MQ Adv z/os VUE V9.0.3 CD V9.0.4 CD 36

37 Kafka Connectors for MQ There is an increasing need to bridge between your MQ messaging platform and your Kafka stream data platform, unlocking your enterprise and your stream data Kafka Connect is a framework included in Apache Kafka that enables this IBM MQ sink and source connectors are currently being openly developed by IBM 37

38 Managed File Transfer Simplified licensing MFT Agents are now no longer separately and individually licensed but are free to deploy and use when connected to MQ Advanced entitled queue managers Redistributable MFT Agent The MFT agent now available from FixCentral, users simply download and unzip IBM MQ Advanced V9.0.1 CD V9.0.2 CD V9.0.3 CD Improved diagnostics Comprehensive fine grain coverage of FTP errors Enhanced logging of FTP communications for post diagnosis File transfer recovery timeout control, new option to automatically cancel failing transfers MFT agent status reporting, aids problem diagnosis by reporting last contact time 38

39 Cloud 39

40 MQ Runs Exactly How and Where You Need It Cloud Azure Message Hub IBM Bluemix AWS Google On-premises Linux Windows Private cloud HPUX Solaris Distributed platforms IBM MQ Appliance 40

41 Help and advice with MQ in the Cloud It has always been critical for MQ to run where it is needed and integrated into the tools of your choice For many that means clouds We have investigated running MQ in various public and private clouds. Using a variety of tooling for provisioning, configuration and monitoring And we ve been sharing that information to everyone to use 41

42 Getting started quickly Getting up and running with MQ quickly is important, but running it in the best possible way is critical MQ is publishing quick starts and preconfigured environments to show best practice IBM MQ is the first IBM product to work with Amazon to publish an AWS Quick Start, deploying a highly available MQ setup in AWS in less than thirty minutes IBM MQ s developer container is preconfigured with essential MQ resources to get going as quickly as possible 42

43 MQ in Containers IBM MQ is supported to run inside Docker containers, bringing the benefits of containers to MQ Lightweight containers for running MQ Predictable and standardized units for deploying MQ Process, resource and dependency isolation Best practice guidance IBM provided sample Docker files for customizing and building your own Docker images on GitHub IBM MQ Advanced for Developers V9 available direct from Docker Hub MQ has recently started to publish Helm Charts, showing best practice for running MQ in Kubernetes Providing MQ Advanced in IBM Cloud Private IBM Cloud Private is IBM s Kubernetes-based container platform that provides a catalogue of IBM middleware including MQ Advanced. It includes an integrated DevOps toolchain and core operations services. 43

44 Run MQ, exactly how and where you need it Cloud Message Hub (Based on Apache Kafka) AWS AWS IBM Bluemix (including Softlayer) AWS On-prem Private cloud Distributed platforms IBM MQ Appliance 44

45 or Let IBM host MQ for you and use it from where you need it Cloud Message Hub (Based on Apache Kafka) AWS AWS On-prem IBM Bluemix (including Softlayer) Future MQ AWS hosted SaaS AWS Private cloud Distributed platforms IBM MQ Appliance 45

46 Platforms 46

47 z/os 47

48 MLC Quick Reminder: MQ for z/os Portfolio IBM MQ for z/os Provides MQ capability, connectivity on and off the z/os platform & excellent integration with z/os s/w like CICS, IMS, WAS, DB2 48

49 Quick Reminder: MQ for z/os Portfolio OTC Offerings MLC IBM MQ for z/os Provides MQ capability, connectivity on and off the z/os platform & excellent integration with z/os s/w like CICS, IMS, WAS, DB2 IBM MQ for z/os Value Unit Edition (VUE) Same functionality as MQ for z/os, but intended for new workloads running within a znalc LPAR only. Can coexist and interact with MQ MLC offerings in other LPARs IBM MQ Managed File Transfer (MFT) for z/os Equivalent function as MFT on distributed platforms though more closely integrated with base MQ offering as of V8. Must be locally bound to a z/os Queue Manager, no client/agent support hosted on z/os IBM MQ Advanced Message Security (AMS) for z/os Provides end to end encryption of messages throughout the MQ network Data is encrypted at rest, not just on the wire IBM MQ Advanced for z/os Soft bundling of MQ MFT for z/os and MQ AMS for z/os only (i.e. no MQ z/os included) Can be deployed with MQ MLC or VUE offerings IBM MQ Advanced for z/os Value Unit Edition Soft bundling of IBM MQ for z/os VUE + IBM MQ MFT for z/os + IBM MQ AMS for z/os 49

50 MQ Advanced for z/os VUE unique function IBM MQ Advanced V9.0.3 CD The Richest Set of z/os Messaging Capabilities in a Single, Simple to Deploy Offering IBM Cloud Product Insights support Provides registration and usage information to the Cloud Product Insights Service to offer insight into the usage of the entire MQ estate across z/os and distributed systems MQ Blockchain connector Enables application integration with the IBM Blockchain service running in Bluemix, mediated via MQ. Managed File Transfer Agent Connectivity Allows a z/os Managed File Transfer Agent to remotely connect to a z/os Queue Manager to simplify the deployment of MFT on z/os The same MFT workload may require fewer z/os queue managers 50

51 For the Developers on z/os z/os Connect z/os Connect is about exposing mainframe resources to applications via a modern REST and JSON interface. The MQ Service Provider allows existing services that are fronted by MQ to be accessed via a RESTful front end. Client s applications need have no knowledge of MQ. MQ JMS in CICS Liberty Profile Allow CICS Liberty to use the MQ resource adapter Just like normal Liberty, allowing you to run Liberty messaging apps such as MDBs inside CICS. Connections to MQ supported using either client or bindings mode. CICS Liberty JVM JMS MQ Resource adaptor External MQ CHIN MSTR 51

52 Distributed 52

53 Distributed recovery logs V9.0.2 CD logfile MQ always logs all the data you need to recover from a queue manager failure in a recovery log logfile logfile Restart recovery (circular and linear logging) Enough information held in the log files to rebuild MQ resources to the level that they were at prior to the queue manager stopping Media recovery (linear logging only) Enough information held in the log files to rebuild MQ resources in the event of losing or corrupting MQ data logfile logfile Circular logging logfile logfile logfile logfile Linear logging 53

54 Significantly reducing the overheads of linear logging V9.0.2 CD Automatic Media Imaging Media images allow logs to be archived or reused, reducing space and speeding up recovery time Previously this was up to the administrator New ability for the queue manager to automatically schedule the recording of media images The frequency of imaging can be controlled based on time interval or data written New ability to exclude certain objects from media recovery logging to reduce overheads Queue manager controlled imaging reduces the impact on other workload during recording Manually controlled recording Queue manager controlled recording 54

55 Significantly reducing the overheads of linear logging V9.0.2 CD Automatic log reuse Linear log extents must be removed from the system to free up space You may choose to archive them before deleting You can now create a queue manager where log extents are automatically reused as soon as they are no longer required for media recovery Alternatively they can be automatically reused when no longer required and have been marked as archived Significant performance improvements over user managed linear logging, comparable to circular logging Persistent Message Throughput And in general Extra information added to queue manager status on log utilization to aid better log configurations Circular Logging Manual Linear Logging Automatic Linear Logging 55

56 MQ Appliance 56

57 MQ Appliance Hardware Update better capacity and performance Increased performance and capacity with new 3.2TB SSDs. Increased connectivity with additional 10GB network ports Floating IP support simpler application failovers Application V9.0.1 CD V9.0.2 CD V9.0.3 CD V9.0.4 CD Added floating IP support. Each HA queue manager can be associated with an IP address that automatically moves with the queue manager Administration better admin experience Added a host of administrative improvements, including SNMP support, administrative REST support for the appliance, new authentication and role based security and backup and restore capabilities SAN Support integration to existing systems IP1 QM1 IP1 QM1 57 Added support for external SAN storage

58 MQ for HPE NonStop 58

59 IBM MQ for HPE NonStop MQ V8 functionality is now offered on the HPE Integrity NonStop platform for both the NonStop X servers and NonStop i servers Released in June 2017 Provides many of the MQ V8 capabilities plus some of the unique HP NonStop capabilities from the previous version, MQ V5.3 Will follow the Continuous Delivery model of incrementally extending the capabilities to bring functional parity with the HP NonStop V5.3 features 59

60 AIX 60

61 AIX V9.0.4 CD To streamline continuous delivery the MQ CD 9.0.x releases are focused on the most active platforms. This was initially restricted to the following platforms Linux, z/os, Windows, MQ Appliance High customer demand has meant we have added AIX to the list of CD platforms from

62 Help shape the future of I MQ ibm.biz/mq-customer-survey 62

63 Session feedback Please submit your feedback online at: Paper feedback forms also available from the chair person This session is JE