Using z/osmf to Expose MQ for z/os as a Service for the Cloud

Transcription

1 Using z/osmf to Expose MQ for z/os as a Service for the Cloud Gwydion Tudur IBM MQ for z/os Development 02/11/2016 Session JG

2 Important Disclaimer THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

3 Using z/osmf to Provision MQ for z/os This session will show how z/osmf can be used to create workflows to provision and de-provision MQ for z/os Queue Managers and resources such as queues. Learn how these workflows can be driven via a service portal using REST APIs. The session also describes how the task of customizing a Queue Manager can be automated and potentially simplified so that users can rapidly stand up or down MQ for z/os Queue Managers and resources for use in Continuous Delivery, Continuous Integration and old fashioned production scenarios!

4 Agenda zcloud Services Requirements z/osmf V2.1 Architecture Approach to writing workflows for MQ V9 for z/os Running the sample workflows for MQ zcloud POC Architecture z/osmf V2.2 POC code and Sample workflows Patterns Future thoughts

5 zcloud Services

6 zcloud Services requirements POC for Development/Test environments Provide Services to help customers rapidly provision/de-provision traditional z/os middleware: MQ, DB2, CICS, IMS, WAS (Classic + Liberty Profile) MWaaS (MQaaS, DB2aaS, CICSaaS, etc.) Create an environment to encourage new z/os applications/business Must be Automated: Self-Service/Single-click of a button Consumers should require little or no z/os skills Implement using z/os Management Facility (z/osmf) Workflows

7 zcloud Services requirements (continued) Private, on premise cloud environments (maybe Hybrid Clouds in the future) Service Providers (System Administrators) to have overall control Pre-allocate infrastructure Control usage via pre-set properties and customizable scripts/workflows Allow for corporate/site resource naming standards zcloud Beta program For selected zblc customers For more information please see:

8 Use Cases for MQaaS In a private, on-premise cloud environment Rapidly provision queue managers and queues at a click of a button In development/test (beta) environments Application developers/testers can stand up/down queue managers and queues on demand! No need to wait for system administrators to action System administrators can still maintain overall control Can enforce resource naming conventions Permit selected users to run, customized, pre-canned workflows For Automation Can automate queue manager configuration tasks/processes Reduces need for z/os skills

9 zcloud Services Following discussions with numerous z/os customers, a proof of concept (POC) was kicked off in 1Q 2015 to provide services to provision and de-provision MQ, CICS, DB2, IMS, WAS and Liberty Profile Server subsystems. In this presentation, we focus on the provisioning/de-provisioning of MQ for z/os subsystems (this includes both the queue manager and channel initiator address spaces) and resources. Provision/de-provision should be: Self-service, on demand and automated. It should be possible for application developers and testers (i.e. the service consumers) to provision/de-provision subsystems on demand at a click of a button from a graphical user interface. Rapid and with ease, with little or no z/os skill required. It should be possible to invoke services at the push of a button from the user interface with no/minimum parameters. Services should drive automated and customizable scripts/workflows to provision/de-provision subsystems. Subsystems are for DEV/TEST use but maybe also for QA/PROD. But, typically expect to provision in DEV/TEST and promote through to QA and PROD. System programmers/administrators (i.e. the service providers) should be able to maintain overall control. They should be able to define an infrastructure onto which subsystems can be provisioned via automated scripts. By using scripts, system programmers/administrators can pre-set values for properties and define corporate naming standards.

10 zosmf V2.1 Architecture

11 z/osmf V2.1 Architecture z/os Operating System

12 Approach for writing sample workflows for MQ V9 for z/os

13 Queue Manager Configuration Tasks Task Task 1: Identify the z/os system parameters Task 2: APF authorize the WebSphere MQ load libraries Task 3: Update the z/os link list and LPA Task 4: Update the z/os program properties table Task 5: Define the WebSphere MQ subsystem to z/os Task 6: Create procedures for the WebSphere MQ queue manager Task 7: Create procedures for the channel initiator Note: Assume SMPE install of MQ Tasks 1 4 already done

14 Queue Manager Configuration Tasks Task Task 8: Define the WebSphere MQ subsystem to a z/os WLM service class Task 9: Select and set up your coupling facility offload storage environment Task 10: Set up the coupling facility Task 11: Implement your ESM security controls (TODO!) Task 12: Update SYS1.PARMLIB members Note: Stand alone QMGR only (QSG in future)

15 Queue Manager Configuration Tasks Task Task 13: Customize the initialization input data sets Task 14: Create the bootstrap and log data sets Task 15: Define your page sets Task 16: Add the WebSphere MQ entries to the DB2 data-sharing group Task 17: Tailor your system parameter module Task 18: Tailor the channel initiator parameters Note: Stand alone QMGR only (QSG in future)

16 Queue Manager Configuration Tasks Task Task 19: Set up Batch, TSO, and RRS adapters Task 20: Set up the operations and control panels Task 21: Include the WebSphere MQ dump formatting member Task 22: Suppress information messages Task 23: Update your system DIAG member for AMS Task 24: Create procedures for Advanced Message Security Task 25: Set up the started task user Advanced Message Security Task 26: Grant RACDCERT permissions to the security administrator for Advanced Message Security Task 27: Grant users resource permissions for WebSphere MQ Advanced Message Security Note: To be considered in the future

17 Steps to provision a z/os Queue Manager Step Description JCL 1 Specify Queue Manager Criteria - 2 Define APF Load Library CSQ4ZNAP.jcl 3 APF authorise the Load Library CSQ4ZAAU.jcl 4 Generate MQ System Parms (ZPARms) CSQ4ZZPM.jcl 5 Define BSDS and Logs CSQ4ZNBL.jcl 6 Define Pagesets CSQ4ZNPG.jcl 7 Define INYS (for additional storage classes) CSQ4ZINY.jcl 8 Construct QMGR startup JCL CSQ4ZMST.jcl 9 Use SETSSI to define subsystem to z/os CSQ4ZSSI.jcl 10 Construct CHINIT Startup JCL CSQ4ZCHN.jcl 11 Construct DISX (CHINIT startup command) CSQ4ZDSX.jcl 12 Disable subsystem security CSQ4ZNSE.jcl 13 Build CHINIT init. dataset to start LSTR CSQ4ZIPX.jcl 14 Start QMGR CSQ4ZSQM.jcl 15 Run Installation Verification Procedure CSQ4ZIVP.jcl Further potential to merge and reduce steps or add installation specific steps (e.g. perform backups, monitoring, etc.)

18 Steps to de-provision a z/os Queue Manager De-provision everything that was provisioned (but undo of SETSSI not supported yet) Step Description JCL 1 Request CMD Prefix, SSID and Environment - 2 Stop Channel Initiator (listener will also stop) CSQ4ZTCN.jcl 3 Stop Queue Manager CSQ4ZTQM.jcl 4 Wait for Subsystems to stop CSQ4ZWSS.rex 5 Delete subsystem security permissions CSQ4ZDSE.jcl 6 Delete APF Authorisation for Load Library CSQ4ZDAU.jcl 7 Delete MQ specific Datasets CSQ4ZDDS.jcl

19 MQaaS - Building Blocks (included) qmgr_variables.xml + common_variables.xml + tcpip_variables.xml provision.xml (a z/osmf workflow) Step Step Step Step.... Step (referenced) JCL JCL.... JCL Provision z/os subsystems MQ DB2 CICS IMS WAS (associated) (referenced) workflow_variables.properties (a Text file) (an important file!) Note: Steps in a workflow can just gather data, submit JCL, invoke a REXX EXEC/shell script, call another workflow.

20 Sample z/osmf Workflows for MQ V9 for z/os Workflow Name Description Notes provision.xml Provision a z/os Queue Manager (and Channel Initiator) Takes minutes to provision! deprovison.xml De-provision a z/os Queue Manager (and Channel Initiator) Takes minutes to de-provision! startqmgr.xml stopqmgr.xml definequeue.xml displayqueue.xml deletequeue.xml putqueue.xml getqueue.xml loadqueue.xml offloadqueue.xml clearqueue.xml Start a z/os Queue Manager (and Channel Initiator) Stop a z/os Queue Manager (and Channel Initiator) Define a local queue Display a subset of a local queue s attributes Delete a local queue Put one or more messages to a local queue Get one or more messages from a local queue Load messages from a dataset to a local queue Offload messages from a local queue to a dataset Clear all messages on a local queue For more information, search for z/osmf in the MQ V9 Knowledge Center New in V9

21 Running the sample workflows for MQ

22 One-click provisioning of MQ for z/os

23 One-click provisioning of MQ for z/os resources MQ for z/os V9, available on 17th June 2016, introduces sample z/osmf workflows that allow you to rapidly provision (create) or de-provision (destroy) z/os Queue Managers, Channel Initiators and Local Queues. Samples are also provided to perform actions against provisioned resources. Once created and run in z/osmf, the sample workflows essentially submit, in automated succession, JCL and/or REXX execs to provision or de-provision MQ resources. Samples can be customized and default properties can be set, by the Service Provider (see diagram below), to meet local conventions and to minimise data entry, and skills required of the Service Consumer when provisioning or deprovisioning resources. Workflows can be driven from the z/osmf Web User Interface (WUI) or via REST APIs. One-click operations can be achieved by invoking REST APIs from userwritten service portals (as shown in the diagram). Expected scenarios for using the workflows are to allow: Application developers with little z/os experience to rapidly provision MQ for z/os resources MQ administrators or system programmers to rapidly deploy site standardized queue managers in minutes rather than hours New in V9

24 Driving the Provision Workflow via REST API First issue a POST request to create a workflow instance

25 Driving the Provision Workflow via REST API Output from POST request

26 Driving the Provision Workflow via REST API Next issue a PUT request to run the workflow instance

27 Driving the Provision Workflow via REST API Output from PUT request

28 Can use http step in UCD process to drive z/osmf workflow But, a z/osmf plugin is currently being developed by the Urban Code Deploy (UCD) team

29 HTTP Step to drive z/osmf workflow

30 CICS Provisioning Tool (cicspt) A simple USS utility Brings a Docker like experience to z/os To rapidly provision and de-provision CICS environments Uses z/osmf workflows to manage lifecycle of CICS cicspt run command provisions and starts CICS cicspt rm command shuts down and de-provisions CICS Released as part of CICS TS V5.4 Open Beta program For more info, see:

31 Using cicspt to run MQ workflows First download cicspt and unpack it in a USS directory Replace the CICS workflows with the MQ workflows (unpacked from MQ V9) Modify the workflow_variables.properties file to set the: Command Prefix (e.g.!zcd2) Environment (e.g. Dev) Subsystem ID (e.g. ZCD2) Issue the cicspt run ZCD command to drive the MQ provision.xml workflow where: ZCD2 is the name used for the workflow instance (I just set it to the SSID) are dummy parameters that are not required when provisioning MQ Issue the cicspt rm ZCD2 command to drive the MQ deprovision.xml workflow

32 Automating Queue Manager and Channel Initiator Provision/Deprovision

33 provision.xml steps to define datasets Note: Step to Define BSDS and LOGS. Submits JCL job CSQ4ZNBL. Note: Step to Define PAGESETs. Submits JCL job CSQ4ZNPG.

34 provision.xml runasuser Can specify a user ID with higher authority

35 qmgractions.xml actions against a queue manager startqmgr stopqmgr deprovision

36 workflow_variables.xml Note: Variables to control properties for DEV environment: - CSQ_DEV_DUAL_BSDS - CSQ_DEV_DUAL_ACTV_LOGS - etc.

37 workflow_variables.properties DEV TEST Note: Can define different characteristics for Queue Managers based on the environment they are being created for! QA

38 workflow_variables.properties Default Subsystem ID High level qualifiers for MQ target library

39 Loops/Conditional/Automatic variable substitution Note: ${instance-*} variables are automatically substituted at runtime. No more ++var++ values in JCL! Note: - ${*} local variables are also automatically resolved at runtime. - Looping + Conditional logic in JCL

40 How to Provision a Queue Manager and Channel Initiator

41 Defining a TCP/IP Port System plus TCP/IP stack

42 Defining a TCP/IP Port workload deployment group

43 Defining a TCP/IP Port Port Allocation Range

44 Creating and Running Workflows

45 1. Add Queue Manager template provision.template ProvisionQMgr provision.xml qmgractions.xml workflow_variables_properties

46 2. Test Run and Publish the template Publish Test Run Note: Once published, the template becomes your Cookie-cutter!

47 3. Run the Published template Run published

48 4. Provide Subsysid and Command Prefix, and Run Subsystem id (ZCT1) Command Prefix (!) Note: Command Prefix is actually!zct1, we do construct this internally but this needs to be sorted.

49 5. Queue manager instance created in the registry QMgr_6 has been created in the registryzcd8

50 6. Queue Manager Instance in the registry QMgr_6

51 7. Workflow instance

52 8. Workflow execution results

53 Queue Manager Actions

54 Add Define Queue Template

55 Define Queue

56 Define Small, Medium or Large Queues Define different values for MAXDEPTH and MAXMSGL in workflow_variables.properties

57 Queue Actions

58 zcloud POC Architecture

59 zcloud Architectural Overview

60

61 zcloud POC sample workflows for MQ for z/os

62 MQaaS - zcloud POC samples (to date) Provision/De-provision MQ for z/os Queue Manager and Channel Initiator Standalone Queue Manager (no Queue Sharing Group support yet) Dynamically allocate a TCP/IP port Start a Listener against it Provision/De-provision MQ Local Queues Display Queue Put Message to Queue Load/Off-load queue Clear Queue Some resource (TCP/IP ports) pooling work Note: - Requires z/osmf V2.2 (in development) Note: - Display Queue needs more work - Get Message from Queue not there yet

63 Configure, Publish and Run workflows WUI 1. Use Configuration Assistant to reserve a range of ports Angel Server 1) Configure z/osmf z/os z/os Communications Use Software Configuration to Publish workflows 2. Publish 4. Dynamically Allocate a port from a reserved range Services Catalog 3. Run MQ Workflows 5. Store State Registry 3. Use z/osmf Catalog REST APIs to run the workflows 4. Provision Resources Service Portal Servlets MQ Queue Manager Queues Channel Initiator LSTR (20000)

64 Steps to provision a z/os Queue Manager Step Description JCL 1 Specify Queue Manager Criteria - 2 Define APF Load Library CSQ4ZNAP.jcl 3 APF authorise the Load Library CSQ4ZAAU.jcl 4 Generate MQ Initialization Parms (ZPARms) CSQ4ZZPM.jcl 5 Define BSDS and Logs CSQ4ZNBL.jcl 6 Define Pagesets CSQ4ZNPG.jcl 7 Define INYS (for additional storage classes) CSQ4ZINY.jcl 8 Construct QMGR startup JCL CSQ4ZMST.jcl 9 Use SETSSI to define subsystem to z/os CSQ4ZSSI.jcl 10 Construct CHINIT Startup JCL CSQ4ZCHN.jcl 11 Construct DISX (CHINIT startup command) CSQ4ZDSX.jcl 12 Allocate TCP/IP port from a given range REST API call 13 Display TCP/IP port id - 14 Disable subsystem security CSQ4ZNSE.jcl 15 Build CHINIT init. dataset to start LSTR CSQ4ZIPX.jcl 16 Start QMGR CSQ4ZSQM.jcl 17 Run Installation Verification Procedure CSQ4ZIVP.jcl Further potential to merge and reduce steps Or add installation specific steps (e.g. perform IVP, backups, monitoring, etc.)

65 Steps to de-provision a z/os Queue Manager Step Description JCL 1 Request CMD Prefix, SSID and Environment - 2 Stop Channel Initiator CSQ4ZTCN.jcl 3 Stop Queue Manager CSQ4ZTQM.jcl 4 Wait for Subsystems to stop CSQ4ZWSS.rex 5 Delete subsystem security permissions CSQ4ZDSE.jcl 6 Delete APF Authorisation for Load Library CSQ4ZDAU.jcl 7 Deallocate TCP/IP Port REST API call 8 Delete MQ specific Datasets CSQ4ZDDS.jcl De-provision everything that was provisioned (but undo of SETSSI not supported yet)

66 MQaaS - Building Blocks qmgractions.xml qmgr_variables.xml + common_variables.xml + (included) provision.xml (a z/osmf workflow) Step Step Step Step.... Step (referenced) JCL JCL.... JCL Provision z/os subsystems MQ DB2 CICS IMS WAS tcpip_variables.xml (associated) (referenced) workflow_variables.properties (a Text file) (an important file!) Note: Steps in a workflow can just gather data, submit JCL, invoke a REXX EXEC/shell script, call another workflow.

67 Customise/Create Workflows Eclipse Workbence ftp USS directory e.g. /u/mayur/mqaas/workflows/mq Develop Workflows in your favourite XML editor WUI (graphical XML editor being developed)

68 Availability of z/osmf V2.2 POC code + sample workflows z/osmf V2.2 POC code + Sample workflows Currently available to zcloud beta customers Plan is to GA these in 4Q 2016 or 1Q 2017

69 Patterns

70 CICS and MQ Bind (in progress)

71 CICS and MQ Bind (in progress) Steps: 1. The System Programmer writes the MQ provisioning workflow (starting with any sample workflows that we ship of course) and uses the z/osmf Catalog Services to publish the worlkflow to the Catalog. 2. The MQ workflow has actions associated with it - a) Provision, b) Start, c) Stop, d) De-provision. 3. The Provision action is invoked to provision an MQ subsystem (i.e. a Queue Manager). The subsystem is then started (currently as part of provisioning but could also be started via the Start action). 4. The MQ workflow will use the z/osmf registry REST APIs to store details of the MQ Load Libraries and the Queue Manager name in the registry (the registry API allows you to store name value pairs). We could define an Initiation Queue (a special type of MQ queue used to trigger start CICS transactions when messages are placed on application queues) but then MQ would decide on the Queue Name and CICS would not be able to select the name. Today the default name for this queue is CICS01.INITQ but I CICS systems programmers can change this name if they want to so I think it would be better if CICS were to define this queue (see later step for this). 5. The System Programmer writes the CICS provisioning workflow (again, starting with any sample workflows that we ship) and uses the z/osmf Catalog Services to publish the workflow to the Catalog. 6. The CICS workflow has actions associated with it - - a) Provision, b) Start, c) Stop, d) De-provision. 7. The CICS workflow has a REST step to read the MQ data from the registry at provision time. The load libraries are set in the DFHRPL concatenation in the CICS start up job. 8. The CICS workflow has an MQ utility step to define the Initiation Queue to the provisioned Queue Manager (the name of the Initiation could also be stored in the registry if required). 9. The Provision action is invoked to provision a CICS region. The MQCONN CICS attribute is set up during provision to define the connection properties to the MQ Queue Manager (these properties can be set as a SIT (System Initialization Table) override during CICS startup or they can be set up as an MQCONN object using RDO (Resource Definition O-line) and installing the MQ object into the CSD (CICS System Definition). These connection properties include the name of the Queue Manager and the name of the initiation Queue. The CICS region is then started (currently via the provision workflow cut could also be started via the Catalog start Action). 10. The CICS region connects to the MQ Queue Manager (i.e. MQCONN). This could be automatic during CICS start up (if using SIT) or via a separate step if using RDO). 11. The CKTI (CICS Task Initiator transaction) transaction is started (this could be automatic during CICS startup or via a separate step.

72 Middleware Provisioning patterns (future thoughts) Appls Deploy WAS MQ DB2 Pattern 1 WAS MQ CICS DB2 Pattern 2 Appls Deploy WAS MQ DB2 Pattern 3 Queue Table WAS MQ CICS DB2 Pattern 4 Loaded Queue Table with Data

73 Middleware Provisioning (future thoughts) Provision middleware using z/osmf plugin (being developed) UCD Workflow (use z/osmf Plugin to provision middleware) Provision MQ (z/os) MQConn CICS (z/os) Bind DB2 (z/os)

74 Multi-platform Provisioning (future thoughts) Provision via UCD workflow Existing plugin for MQ on distributed (uses runmqsc) Enhance existing plugin (or write new one) to drive z/osmf for MQ on z/os Enhance existing plugin to use MQ MQSC REST APIs instead of runmqsc UCD Workflow Provision WAS (Dist) MQRA MQ (Dist) MQ Channel MQ (z/os) MQConn CICS (z/os) Bind DB2 (z/os)

75 MQaaS (future thoughts) Provision System and Application resources UCD Workflow (use new MQ plugin and z/osmf or MQ REST APIs (future)) UCD Workflow (use new MQ plugin and MQ Rest APIs (future)) Provision system resources (infrastructure) UCD Workflow (use z/osmf Plugin to provision MQ resources) UCD Workflow (use z/osmf Plugin to provision MQ resources) Provision application resources z/os MQ Queue Manager Channels Remote Queues Transmission Queues Local and/or Alias Queues Channel Initiator LSTR (20000)

76 Provisioning MQ and System Automation (some prototyping only) z/osmf Services Catalog Current Design z/osmf WUI z/osmf Workflows Registry Future Design exploit registry z/osmf Services Catalog z/osmf WUI z/osmf Workflows Registry Workflow Chain SA Workflow Provision/ de-provision middleware Un/Define Automation Policy Start/Stop Subsystem Provision/ de-provision middleware Un/Define Automation Policy Start/Stop Subsystem z/os z/os MQ Queue Manager Queues MQ Queue Manager Queues

77 MQaaS - zcloud Beta (future thoughts) Use z/osmf Console Command Support Use REST APIs to issue z/os Console Commands Dynamic allocation of a specified port Provision MQ and CICS, and bind together (complete this work) Discover existing MQ subsystems and define resources on them MQ Console (Web User Interface) Use MQ REST APIs instead of batch utilities Some material on MQ REST APIs is available via the MQ vnext beta program MQ Security (secure Queue Managers and resources through workflow) Define Channels QSGs Dynamic allocation of DVIPA addresses

78 MQaaS - zcloud Beta (future thoughts) Security RACF REST APIs (maybe beyond z/osmf V2.2) Format of data in the z/osmf registry Federated registries? Multi-tenancy Scalability Monitoring Auditing Lifecycle Management of resources z/osmf Workflow Version Control Patterns Orchestration Standards/schemas

79 Summary zcloud Services Requirements z/osmf V2.1 Architecture Approach to writing workflows for MQ V9 for z/os Running the sample workflows for MQ Demo using CICS Provisioning tool (cicspt) zcloud POC Architecture z/osmf V2.2 POC code and Sample workflows Patterns Future thoughts

80 Where do I get more information? IBM MQ Knowledge Center IBM Messaging developerworks developer.ibm.com/messaging IBM Messaging Youtube LinkedIn ibm.biz/ibmmessaging IBM MQ Facebook facebook.com/ibm-mq /

81 Session feedback Please submit your feedback at Session is JG

82 Copyright and Trademarks IBM Corporation All Rights Reserved. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at