Introduction to MQ. Sam Goulden IBM MQ L3 Service. MQ Technical Conference v

Transcription

1 Introduction to MQ Sam Goulden IBM MQ L3 Service

2 Agenda Messaging What is messaging and why use it? What does MQ give you? Fundamentals of IBM MQ Messaging models Key components Messaging applications MQ Environments Security Reliability and availability Administration MQ Advanced

3 What is messaging? It connects your applications! From the simplest pairs of applications and breaks the tight coupling...to the most complex business processes.

4 Why use it? Extended reach Reliability Scalability Flexibility Provides for simplification of application development Ubiquity Easy to change and scale Focus on the business logic Important regardless of the initial scale of deployment

5 Direct communication between applications App A Request an action Wait for a response App B Issues with this synchronous approach Both applications A and B must always be available for A to continue A cannot do anything whilst B is processing A s request What is B fails whilst A is waiting for it to complete? What is B needs to handle a high workload of different priority requests?

6 Fragility of tight coupling As systems become more tightly coupled, their reliance on each other increases The cost of a failure of a process increases Maximum number of connections goes up with the square of the number of systems Scaling systems independently to respond to requirements becomes unmanageable A process was originally designed for one, well-defined, purpose...it then needed to change to meet new requirements Being able to respond rapidly to internal and external challenges by rapidly modifying existing services gives a competitive advantage

7 Adding flexibility with Messaging Send request Perform action App A App B App B A queue is placed between the two applications Allows A to continue immediately, without waiting for B Allows B is efficiently process a queue of work Overcomes availability of B versus A store and forward of messages

8 What if I NEED a response? Send request Perform action App B App A Process response Send response Using messaging still adds value! Process the response whenever it becomes available No need for A to be idle whilst the request is performed Application B processes its workload efficiently and can handle spikes in load Application, network and infrastructure failures are handled

9 The power of events App Not all information is distributed on a one-to-one basis Think about streams of information Regularly updated information - such as stock prices or sensor data Business events - such as new customer or purchase Publish / Subscribe messaging is the solution! The owner of the information simply publishes it on a topic Anybody who is interested simply subscribes to the topic

10 Messaging Models

11 Point-to-point Message consumer Message producer Message consumer Message consumer

12 Point-to-point Message consumer Message producer Message consumer Message consumer Publish/subscribe Subscription Message consumer Topic Message producer Subscription Message consumer Subscription Message consumer

13 Durable publish/subscribe in action Durable subscriptions result in published messages being retained when the subscriber is not connected to the messaging provider. Subscriber 1 Message Publisher Topic Durable Non-durable Subscriber 2

14 IBM MQ

15 IBM MQ timeline MQI Assured delivery Multiplatform IBM MQSeries MQSeries WebSphere MQ IBM MQ Pub/sub Mobile XML V1.1 V2 V5 V2.1 V5.1 JMS SSL SOAP V5.2 IBM MQ Everyplace.NET V5.3 Common criteria Eclipse HTTP AJAX REST IBM MQTT Web 2.0 JMS 1.1 RCMS IBM MQ Low Latency 1990s 2000s 2010s V6 Multi-instance QMGR Managed File Transfer V7 V7.0.1 IBM MQ Managed File Transfer Hypervisors Multiple cluster XMIT queue Integrated Messaging V7.1 V7.5 IBM MQ Advanced Message Security System Pattern Application Pattern IBM MQ Advanced IBM MQ Appliance IBM MQ Advanced for Developers IBM MessageSight MQ Light developer toolkit and Bluemix service JMS 2.0 V8 MQ Light channels V9 MQ on IBM Cloud First LTS RELEASE: V9.1 New LTS Release: Aug 2018 June 2016 Two delivery models: 1) Long Term Service 2) Continuous Delivery

16 What MQ adds to messaging Enterprise Messaging Reliability Scalability Ubiquity Security Assured message delivery Once and once only Resiliency and high availability of the infrastructure Continued support and interoperability of systems for over twenty years High performance solution Incremental growth of applications and infrastructure Breadth of support for platforms and environments Multiple application environments and APIs to suit many styles Data encryption and integrity End use authentication and authorisation Audit trails for configuration and data flows

17 Anatomy of an MQ system Applications Applications use MQ clients to connect to an MQ queue manager Applications can connect to queue managers either on the same system (BINDINGS mode) or remotely over a network (CLIENT mode) System 1 App A System 2 Queue Managers A queue manager is a runtime that hosts messaging resources such as queues and their messages A queue manager manages the flow and storage of messages Each queue manager runs on a single system Multiple queue managers can be connected together using channels and messages routed between them QMgr Channel QMgr Queues Queues are a named resource where messages sent to by applications, stored by the queue manager and retrieved by applications App B Messages Are just chunks of data Applications build messages to send and receive System 3 Channels Channels define a way for one queue manager to connect to another queue manager Channels can be manually configured or dynamically created as and when needed using MQ Clusters

18 IBM MQ IBM MQ Architecture

19 Distributed Architectures Used for connectivity of heterogeneous systems MQ 9.1 MQ QMgr Store and Forward system to account for network outages MQ 7.5 QMgr This is the original deployment pattern for MQ QMgr MQ 7.1 MQ QMgr Queue managers will interoperate with other queue managers and clients at any other version of MQ QMgr

20 MQ hub architecture A hub of systems running queue managers on a standard deployment QMgr Applications connect as clients from remote systems QMgr This pattern has gained popularity as networks improve and administration costs go up QMgr

21 Connecting queue managers together Channels connect queue managers together, allowing messages to flow between them QMgr QMgr Two options: Manual configuration of channels Each channel relationship must be defined on both ends Additional resource also need to be defined (transmission queues and remote queues) QMgr QMgr QMgr MQ clusters Once queue managers join a cluster (a pair of special channels must be defined) they can route messages to any other clustered resource in the cluster without requiring further, per queue manager, configuration. As queue manager networks grow, clusters become a benefit Clusters also enable workload balancing and availability routing of messages QMgr QMgr QMgr QMgr QMgr

22 IBM MQ Applications

23 MQ APIs How do I connect my apps to my queue manager? MQI JMS MQ Light API MQTT REST API Messaging (point to point only!)

24 MQ APIs - MQI (MQ Interface) C, COBOL, Java MQ s proprietary API offering full access to MQ s capabilities QM1 APP.Q MQCONN(QM) MQOPEN(Q) MQPUT(Q) MQGET(Q) MQCLOSE(Q) MQDISC(QM)

25 MQ APIs - Java Message Service (JMS/XMS) JMS is part of the JEE specification. Fully supported in application servers such as WSAS, Liberty, WebLogic and more Simplifies programming for Java developers No MQ coding knowledge needed! XMS syntactically the same as JMS V1.1 but for C, C++ and C# QM1 APP.Q // Lookup the MQ specific objects in JNDI Context jndicontext = new InitialContext(); ConnectionFactory cf = (ConnectionFactory) jndicontext.lookup("jms/qm1"); Destination dest = (Destination) jndicontext.lookup("jms/app.q"); // Establish a connection with the queue manager & create JMS objects JMSContext context = cf.createcontext(); JMSConsumer consumer = context.createconsumer(dest); // Get a message Message msg = consumer.receive();

26 MQ APIs - MQ Light AMQP based API Node.js, Java, Ruby Connects cloud applications to MQ! # Receive: var mqlight = require('mqlight'); var recvclient = mqlight.createclient({service: 'amqp://localhost'}); recvclient.on('started', function() { recvclient.subscribe('news/technology'); recvclient.on('message', function(data, delivery) { console.log(data); }); });

27 MQ APIs - MQ Telemetry (MQTT) Product extension supports connectivity for smart devices to the enterprise Utilises the open standard MQTT protocol Vehicle Oil rig Enterprise a lightweight, public, low bandwidth messaging protocol for scenarios where enterprise messaging clients are too big or bandwidth intensive. Java, C and JavaScript libraries provided, but you can roll your own that implement the MQTT v3 spec Sensor e.g. RFID Retail Store Pervasive Device Petrol Forecourt Medical Smartphones

28 REST Messaging The new HTTP server support in MQ 9.0.x provides the platform for a properly integrated REST API solution App App POST /qmgr/qm1/queue/q1/message DELETE /qmgr/qm1/queue/q1/message Allowing applications to put and get messages from a queue without installing any MQ software locally HTTPS Ideal for environments with native REST support, such as common JavaScript libraries including NodeJS, and AngularJS Can only be used for point to point messaging QM1 REST API Q1 Existing channels MQ App App MQPUT(Q1) MQGET(Q1) For full functionality and resiliency an MQ client should still be used

29 Messaging APIs All interoperate with each other! Any application can receive messages from any other application.net MQTT QMgr QMgr JMS QMgr COBOL C

30 LearnMQ Totally new to MQ? Learn the basics Step-by-step guide to getting up and running with MQ Tutorials on building your applications Finding it hard to get developers started with MQ? Point them to: developer.ibm.com/ messaging/learnmq

31 Once and once only delivery Message persistence Persistent messages Stored to disk Queued messages are recovered following a server failure No matter what the failure, as long as the disks are intact, so will your messages be Non-persistent messages Kept in memory as much as possible (better performance) Queued messages are lost in the event of a server failure or restart Transactions Multiple messaging operations can be coordinated as a transaction Messaging applications are often updating other resources based on messages E.g. Receive a message, insert the data to a database MQ applications can coordinate messaging operations with other transactional resources A queue manager can be an XA transaction coordinator Or coordinated externally, for example a JEE application server such as WebSphere Application Server Available in MQI, JMS and XMS APIs Combining persistent messages with transactions gives you once and once only delivery of messages from an application s point of view

32 Transactional Messaging - Non Persistent - Persistent Request Queue Message producer Message consumer Reply Queue Persistent Queues

33 Transactional Messaging Request Queue Message producer Message consumer Reply Queue Persistent Queues Combining persistent messages with transactions gives you once and once only delivery of messages from an application s point of view Transaction

34 IBM MQ Environments On-premise & Cloud

35 Run IBM MQ in any location or cloud exactly as you need it On-premise, software and the MQ Appliance Run it yourself in any cloud, public or private Let IBM host it for you with its new managed MQ service in IBM Cloud AWS AWSAzure NEW IBM Z Linux AIX Windows Solaris HPE IBMi Appliance AWS IBM Cloud Private Private cloud

36 MQ in Containers MQ has been supporting Docker containers since 2015 with images on Docker Hub and Docker Store and sample setups on Github More recently it has been demonstrating how to get the most from containers using Kubernetes And now MQ Advanced is available as a fully supported product with IBM Cloud Private, a Kubernetes-based solution from IBM

37 MQ on IBM Cloud Provision queue managers directly into IBM Cloud IBM owns the infrastructure and the responsibility to keep the systems up to date and running The customer owns the configuration and the monitoring of the messaging Try the service for free at: console.bluemix.net/catalog/services/mq Hosted on

38 IBM MQ Security

39 Security provided on Client to Queue Manager connections Channel Authentication (BLOCKADDR) SSL/TLS Channel Authentication (ADDR/USER/SSL Map) Security Exit MQRC_NONE Or MQRC_NOT_AUTHORIZED Connection Authentication Channel Authentication (BLOCKUSER) Authorization

40 IBM MQ High Availibility

41 IBM MQ HA capabilities Come to my other talks! Benefitting from the MQ Appliance Room: Zebrawood Time: Monday, 14:30PM or Wednesday 09:45AM Support for HA clusters and network storage Multi-instance queue managers (Windows, Linux, UNIX) IBM MQ Appliance Client connectivity Automatic reconnection CCDTs Pre-connect exit Replicated Data Queue Managers (Linux) Queue-sharing groups (z/os) Support for cloud orchestration frameworks e.g. Kubernetes, Docker Swarm, Apache Mesos

42 IBM MQ Administration

43 Administration and monitoring Command line Scripting Come to my other talks! Programmatic APIs Administering MQ, The MQ Console and the MQ REST API! REST API Room: Zebrawood Time: Tuesday, 8:30AM or Wednesday 13:00PM Tivoli and thirdparty tooling Web console GUI tooling GSE Nordic Conference / 2018 IBM Corporation

44 IBM MQ MQ Advanced

45 MQ Advanced Message Security Secures application data even before it is passed to MQ Upgrade from base MQ No changes to existing applications or network required MQ standard security: Industry standard TLS channels (256-bit) Certified for Common Criteria Authentication is based on Operating System identifier of local process Message data can be encrypted in transport but not when it resides in the queues App B AMS Interceptor App A AMS Client Interceptor MQ Advanced Message Security adds: Authentication policies are based on certificates associated with each application Message data is protected end-to-end including when it resides in queues Much finer granularity in security policies No changes needed to applications or queues MQ Queue Manager

46 MQ Managed File Transfer Coordination Queue Manager MQMFT Eclipse Tooling MQMFT MQ MQMFT MQ Servers MQMFT MQ MQMFT MQMFT MQMFT MQMFT MQMFT Eclipse Tooling Clients

47 Where to go now?

48 LearnMQ Totally new to MQ? Learn the basics Step-by-step guide to getting up and running with MQ Tutorials on building your applications Finding it hard to get developers started with MQ? Point them to: developer.ibm.com/ messaging/learnmq

49 Where do I get more information? IBM MQ Knowledge Center IBM Messaging developerworks developer.ibm.com/messaging Youtube

50 Copyright and Trademarks IBM Corporation All Rights Reserved. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at

51 Thanks for listening Questions? Sam Goulden IBM UK, IBM MQ L3 Service