Unit 1a - Overview. WebSphere Application Server Liberty Profile z/os. WebSphere Application Unit Server 3 Liberty z/os Profile Connect z/os

Transcription

1 Unit 1a - Overview IBM Advanced Technical Skills ZCONN1 WebSphere Application Server Liberty Profile z/os WebSphere Application Unit Server 3 Liberty z/os Profile Connect z/os z/os Connect Unit 1a - 1

2 This page intentionally left blank 2 Unit 3-2

3 z/os Connect at a High Level This is the picure we showed back in Unit 1: Liberty Profile z/os Anything that supports REST/JSON Configuration XML file z/os Connect CICS IMS Batch* This can be IBM MobileFirst Platform, some other mid-tier device, or even other mainframe programs Data Conversion Audit (SMF) Discovery Access Control Green starts indicate topics covered in this unit This topic we'll leave for the unit on security * By batch we mean a long-running job that uses the WOLA host a service API to listen for calls coming over from z/os Connect Extensible 3 Now we come to a high-level description of what z/os Connect is. We've hinted at it several times in the earlier discussion, but here we'll start the more direct discussion of it. At its heart z/os Connect is an application that runs inside a Liberty Profile z/os server instance. The z/os Connect application (it's a servlet) provides the framework to handle REST and JSON sent by clients, and then to invoke the backend program or service associated with the REST request. The relationship between REST request and backend service is defined in the Liberty Profile configuration XML file. That's where you indicate what URI patterns are to be recognized, and then what should be done when that URI pattern is received. The backend programs or services that z/os Connect is capable of communicating with is CICS, IMS and Batch. By batch we mean a long running COBOL (or C/C++, PLI, or High Level Assembler) program that is listening for a call from z/os Connect. In addition, z/os Connect provides four functions within its framework the ability to perform data conversion from JSON to whatever the backend data format is; auditing using SMF records; a set of REST APIs to provide information and details on the configured services; and the ability to configure granular access control so users have defined limited access. There's quite a bit of detail we've left unsaid here, of course. We have a whole unit on this where we'll get into that discussion. Unit 3-3

4 z/os Connect is Extensible InfoCenter search string: twlp_admin_extend A service provider is the mechanism used to reach the backend. The built-in sevice provider is WOLA. You may add your own. A data transformer is code that transforms JSON to the format required by the backend. The built-in transformer is based on the CICS utility to transform JSON to COBOL COPYBOOK or other language data structure. An interceptor is function called for each request. The built-in interceptors are audit (SMF) and authorization. You may add your own to do custom function. 4 HA When looking at z/os Connect it's important to realize that it is an extensible framework. As delivered by IBM it has certain functions coded, but if you want to extend z/os Connect to have custom functionality, you can do it. This is built on the capability of Liberty Profile itself to be extended with user-functions. For z/os Connect there are three areas that can be extended: Service Provider this is the function that is used by z/os Connect and Liberty to talk to the backend system. The built-in service provider that comes with z/os Connect is WOLA. That can be used to communicate with CICS and long-running COBOL (or C/C++, PL/I or High Level Assembler). The IMS support for z/os Connect doesn't use WOLA; it uses a custom service provider the IMS people wrote to talk to IMS Connect. That's an example of someone (IBM in this case) extending z/os Connect with an additional service provider. Interceptor this is function that is called (if configured; this is optional) every time a service is invoked. The built-in interceptors are for audit (SMF ) and authorization (more granular control over who can use the REST APIs vs. simply invoke services). If you want to have you own custom inteceptor, you can code it and configure it with a service. (Note: interceptors can get a copy of the request data, but they can't change the request payload.) Data Transformer this is function that transforms the request JSON to the data format required by the backend, and transform back to JSON when a response is returned. The built-in data transformation function is based on the utility CICS provided for its web services function. That utility takes as input COBOL COPYBOOK (or other programming language data structure) and provides the bind files and JSON schema needed to do the transformation. If you have a different data transformation requirement, you can extend z/os Connect to do your own custom data transformation. Information on how to do this is provided in the Knowledge Center under the search string shown at the top of the chart. Unit 3-4

5 z/os Connect Can Be Made Highly Available Liberty Profile can be made highly available through duplicated server instances. z/os Connect makes this relatively simple because it's stateless by nature: HTTP Distribution Liberty Profile z/os z/os Connect Liberty Profile z/os z/os Connect CICS IMS Batch server.xml CICS IMS Batch Instances of Liberty Profile on different LPARs can share a server.xml and be mirror copies of each other Or have separate copies that are maintained to be very consistent with one another REST is a stateless mechanism, so no need to affinity route If backend systems are present on each LPAR then environment is duplicated and HA possible Setup 5 When thinking about z/os Connect as a 'gateway' into the mainframe for services, it's natural to start thinking about how it might be made highly available. The first step in high availability is duplication of resources. Liberty Profile and z/os Connect can be duplicated fairly easily, and the nature of REST keeps it easy. Two (or more) instances of Liberty Profile z/os can share a copy of the server.xml using the Parallel Sysplex shared file system facility. If you do that, then the two instances of Liberty Profile (and z/os Connect running inside) can be identical. They're separate physical instances running on two separate logical partitions, but in their configuration and behavior they are the same. Note: it is not necessary to use Liberty collectives for this. You may if you wish, but it is not required. The nature of REST helps with this because REST is stateless. That means there is no requirement a client go back to a particular instance of Liberty Profile for its calls. The first call from a client can go to one instance of Liberty Profile, the second call to the other, and so on. Each REST call stands on its own, without a requirement to maintain any state information about the previous call. It's important to think about the backend systems being called by z/os Connect, because ultimately that's what the client is going after. If you configure up a duplicated instance of Liberty Profile with z/os Connect, you need to make sure the backend systems connected to by z/os Connect are also duplicated. z/os Parallel Sysplex helps with this since instances of CICS or IMS on each LPAR can be made Sysplex aware and use shared data. The key message is this if having a highly available REST/JSON interface to your mainframe environment is important, it is possible to duplicate instances of Liberty Profile and z/os Connect to achieve this. Unit 3-5

6 Setup How to do initial enablement of z/os Connect 6 Unit 3-6

7 z/os Connect and WOLA are Features of Liberty And as such, they need to be enabled. This is done with the the featuremanager shell script, which is provided with Liberty Profile z/os: Install File System (Mounted WRITE) /<mount> /bin featuremanager Installs the z/os Connect feature into Liberty Installs the WOLA feature into Liberty./featureManager install zosconnect when-file-exists=ignore./featuremanager install zoslocaladapters when-file-exists=ignore./featuremanager featurelist /<path>/features.txt Lists the features currently installed in Liberty image <feature name="zosconnect-1.0"> <feature name="zoslocaladapters-1.0"> 7 Note: you must have Internet access to run these commands Configuring services After installing Liberty Profile z/os V or higher using IBM Installation Manager (IM), you will have a file system containing the Liberty Profile product files. z/os Connect is supplied as a feature to Liberty Profile z/os, but it's a feature you have to install. This is done using the featuremanager shell script, which is found under the /bin directory of the install location. Note: to perform the install the file system must be mounted write, and the LPAR on which this is being done must have internet connectivity. The install process will access files from the IBM repository to complete the install. There are two features to install the z/os Connect feature and the WOLA feature. If you were accessing just IMS using their supplied JCA resource adapter and IMS Connect the WOLA feature would not be necessary. But for accessing CICS or a long-running COBOL or other batch program, then WOLA is needed. Use of the featuremanager shell script requires that your UNIX environment has JAVA_HOME resolving to a valid 64-bit JVM on the system. Once that's in place, then the command to install the z/os Connect feature is as shown in the chart. You will be prompted to accept the license, and once that's done it takes a minute or so and the feature will be installed. Repeat the same thing for the WOLA support. You can verify the features are present by using the featuremanager again, but this time with the featurelist action. This will produce an XML file with the installed features listed out. Find the two features (as shown) and that verifies the features are present in the Liberty Profile z/os install. Unit 3-7

8 Configuring Services Configuring the z/os Connect to host services to backend programs 8 Unit 3-8

9 It's All About the server.xml File Always keep in mind that z/os Connect is only what the contents of server.xml tell it to be. The key to understanding z/os Connect is understanding the XML structure. The server.xml configuration file Key Points: Liberty Profile is, fundamentally, defined by its configuration (server.xml) z/os Connect relies on what's in server.xml to understand how it is configured and how it is to behave There's no magic to this there is a lot of detail, but no magic 9 Abstract flow We will start with a reminder about how the server.xml file the primary configuration file for a Liberty Profile server is the focus for most of the discussion of z/os Connect. A given instance of Liberty Profile will be only what's defined in the server.xml. Further, z/os Connect is in large part an application that runs inside Liberty Profile, and it too relies on the configuration XML found in server.xml. We start with this because we're about to show a fair amount of detail related to XML that gets updated in server.xml. Going through some of the detail is unavoidable in any discussion of z/os Connect. It is unavoidable because to Liberty Profile and z/os Connect, it really is all about what's in the server.xml file. Unit 3-9

10 Abstract Flow of Processing of XML Definitions The following flowchart is intended to offer an abstract view of XML definition processing, which will help put the XML examples that follow in context: Client 4 Interceptor Maps Returns Interceptor List 1 Service Definition Call/Return Interceptor 2 Calls Invokes 3 Service Provider Backend Returns Returns 5 Data Transformation Data Bind Files and JSON Schema Files This represents one service; you may have many services defined. Each would have similar structure. Essential XML 10 Before we launch into examples of XML, we need to set the stage with a high-level picture of what's going on with the configuration XML. The flow-chart shown here is intended to represent a kind of flow that takes place within the server.xml file definitions for z/os Connect. This is not meant to represent the exact flow processing within the z/os Connect function; rather, just how the XML definitions relate at an abstract level. 1. The service definition is what identifies the receives URI and starts the process of mapping it to other elements in the XML. If the received URI does not map to any service definition, then an error is thrown. But if it maps to one of the service definitions then z/os Connect has something to work with. 2. The XML will point to a service provider, which is a definition of how to access the backend system and program. 3. The backend is invoked and the response is returned. 4. As part of the service definition, it's possible to define a reference to an interceptor list, which will invoke the named interceptors (think of these as call-out programs invoked for each request received). This is optional; if configured, interceptors are invoked if not configured, then no interceptors are involved. Two interceptors are supplied one for audit (SMF ), and another for role-based access authorization. The SMF interceptor we'll briefly cover at the end of this unit; the authorization interceptor will be covered in the security unit. 5. Another optional function is data transformation, which will turn the received JSON into the data format expected by the backend program. This involves generating bind files which define how JSON is to be mapped to the data structure the program expects, and how the response is mapped back to JSON. We have a few charts on this later in the unit. That's the flow. At its simplest it flows from service definition to provider to backend, then back to the client. But you may also have optional definitions in the configuration that makes the flow for sophisticated. Unit 3-10

11 Example of XML Relationships Here's an example of what the previous flowchart was illustrating the service definition maps to both a service provider and a data transformation definition: Service Definition <zosconnectservice id="cics" URI servicename="cics-backend" dataxformref="xformjson2byte" serviceref="wolacics" /> Service Provider <localadaptersconnectservice id="wolacics" registername="cicsreg" servicename="olacb01" connectionfactoryref="wolacf" /> Data Transformation <zosconnectdataxform id="xformjson2byte" bindfileloc="/u/user1/bindfiles" bindfilesuffix=".wsbind" requestschemaloc="/u/user1/json" responseschemaloc="/u/user1/json" requestschemasuffix=".json" responseschemasuffix=".json" /> Backend System That is the essential nature of understanding the server.xml for z/os Connect service definition to provider definition and including any optional interceptor inforamation 11 Essential XML Here's an example of the relationship in the XML for a service definition. This is not the complete server.xml, but rather some key snippets to illustrate the flow and relationship of one section to another. The starting point is the service definition. This defines the URI pattern for this service. In this example you have a URI pattern of CICS-backend (how exactly this maps to a real URI we'll see in a bit). The service definition points to a service provider. That defines the basic WOLA definitions to the backend CICS region and the program to be invoked. Because the service definition has a data transformation reference, that points to XML that defines where the transform files are located. And that's the flow service definition maps to service provider, and references in the service definition point to optional functions, such as a data transformation information. At first this will be new and you'll have to become accustomed to it, but with practice it will become second nature and you'll be copying and pasting sections and modifying the key elements of the XML. Unit 3-11

12 Essential server.xml Configuration Before we get into any service and provider definition details, let's take a look at a few essential things that must be in place first: <featuremanager> <feature>jsp-2.2</feature> <feature>appsecurity-2.0</feature> <feature>zosconnect-1.0</feature> <feature>zoslocaladapters-1.0</feature> </featuremanager> <keystore id="defaultkeystore" password="liberty"/> <webappsecurity allowfailovertobasicauth="true" /> <basicregistry id="basic1" realm="zosconnect"> <user name="fred" password="fredpwd" /> </basicregistry> <authorization-roles id="zos.connect.access.roles"> <security-role name="zosconnectaccess"> <user name="fred"/> </security-role> </authorization-roles> These two <feature> definitions enable z/os Connect and WOLA in the instance of Liberty Profile z/os These are required security definitions. This shows the simplest way to achieve the minimum security requirements: Internal (non-saf) key and trust store Basic auth rather than default client certificate authentication Internal (non-saf) user registry Internal (non-saf) mapping of user to role 12 The first bit of XML we'll explore is some essential housekeeping things: Structure of service definition Update of <featuremanager> section to include both z/os Connect and WOLA this tells Liberty Profile to load the features installed earlier. This is a relatively easy two element update of server.xml. Some security updates to accommodate the requirements of the z/os Connect application. Those are discussed below. The z/os Connect application is marked protected, which means any user accessing it must be authenticated, and the connection must be SSL. Our objective at this point in the workshop is to keep security things as simple as possible. Later, in the security unit, we'll go into more detail on how to secure z/os Connect more properly. The definitions shown on the chart provide for the minimum security requirements defined and managed entirely within Liberty Profile, and not requiring SAF. Those include: The keystore for the SSL certificates. That's the <keystore> element. XML to tell z/os Connect to use basic authentication rather than the default client certificates. Basic authentication is simpler than client certs. Later we'll show using client certificates, which is what we'd expect is much more common for z/os Connect. The definition of a userid for authentication. In this case it's Fred. Finally, the definition of the access role that allows Fred to use the z/os Connect application. Without this Fred would be able to log on, but would then get an error from z/os Connect. With this z/os Connect knows Fred is authorized to use the application. Again since z/os Connect is a protected application, we have to put some security definitions in place. This allows us to provide the minimum required in a way that is entirely based on XML updates. This isn't productionlevel security, but it's good enough for early validation and testing. Unit 3-12

13 Structure of a Service Definition This may have more XML elements than shown here, but this gives a sense for the key elements and what the service definition provides: <zosconnectservice id="cics_create" 1 servicename="cics-create" 2 dataxformref="xformjson2byte" interceptorsref="interceptorlist_1" serviceref="wolacics" /> 4 3 The servicename= value determines the URI that will invoke this service: The dataxformref= value points to another section of server.xml where the data transform definition for this service can be found. (More on data transform coming up.) The interceptorsref= value points to another section of server.xml where the optional interceptor list for this service can be found. (More on interceptors coming up.) The serviceref= value points to another section of server.xml where the service provider definition can be found. That is what defines how to access the backend system for this service. 13 The URIs used Here's our first detailed example, and this shows a service definition. This is for a CICS program that does some form of create function, as the name of the service implies. Let's walk through the numbered blocks: 1. The servicename= element is used to indicate the URI pattern that can be used for this service. In this example we're showing an invoke URI pattern as well (numbered block 2), but you could have just the servicename= value, and if that were the case then the URI would be based on the context of /zosconnect/services followed by the servername= value, followed by?action=invoke. Note: some do not like exposing the /zosconnect/services part of that, which is why the next thing we show the invoke URI pattern is supplied. It hides that level a detail. That's on the next chart. 2. The dataxformref= element points to another portion of XML where the data transformation definition is maintained. This is optional; if you don't wish to do data transformation, you can omit this. We'll show data transformation a bit later in this unit. 3. The interceptorsref= element points to another portion of the XML where interceptors are specified. This too is optional; if you don't wish to use any interceptors, then this may be omitted. 4. Finally, the serviceref= element points to another portion of XML where the service provider is defined. This is what defines the connectivity to the backend system and program. We'll see that in an upcoming chart. At a high level, that's it a service definition provides the URI pattern for the service, and then provides information on how to connect to the backend and whether optional data transformation and inteceptors will be called as part of this. Unit 3-13

14 The URI Clients Would Use for the Defined Service Using the XML example on the previous chart, let's reinforce the understanding of the URI clients would use to invoke the service: <zosconnectservice id="cics_create" servicename="cics-create" invokeuri="/mycicscreate" dataxformref="xformjson2byte" interceptorsref="interceptorlist_1" serviceref="wolacics" /> This is required This is optional Client HTTP PUT or POST with JSON z/os z/os Connect 14 Both achieve same thing the invokeuri is simply a way to make the URI simpler and shield clients from undertanding it's z/os Connect Service provider This chart is introducing an additional element and adding it to the information on the previous chart. It is the invokeuri= pattern. There are two URI patterns that would work for this service. One is the raw format that is, with /zosconnect/services and?action=invoke after the service name. The other is a kind of alias that is, a simpler pattern is used and the details and?action=invoke are hidden. Both end up calling the same service. Which you use is really up to you. The invokeuri pattern is a way to hide any details of the actual backend service name behind an alias. We anticipate the invokeuri= pattern will be commonly used for just that reason. Unit 3-14

15 The Service Provider Access to Backend The next piece of the puzzle is the service provider, which defines how z/os Connect is going to access the backend resource: <zosconnectservice id="cics_create" servicename="cics-create" invokeuri="/mycicscreate" dataxformref="xformjson2byte" interceptorsref="interceptorlist_1" serviceref="wolacics" /> Service definition from the previous chart <localadaptersconnectservice id="wolacics" RegisterName="CICSREG" ServiceName="OLACB01" connectionfactoryref="wolacf" /> Two pieces in place service used to map URI; then provider maps connection to the backend The WOLA registration name used by the external address space The backend program name to be called over WOLA The WOLA JCA resource adapter ID to be used for the WOLA connection 15 Multiple services The service provider definition indicates to z/os Connect how to access the backend service. There are two choices WOLA for CICS and long-running programs such as COBOL, or the IMS JCA resource adapter if connecting to IMS via IMS Connect. Note: in this workshop our focus is on CICS and long-running COBOL. The principles defined here are identical for batch, but the syntax might be slightly different. For WOLA connectivity the key things that need to be define are the WOLA registration name to communicate across, the service name (the target program name) to invoke on the other side of the WOLA registration, and the WOLA JCA resource adapter connect factory to use. (We're not showing the connection factory definition here it's fairly generic and is shown in detail in the WP Techdoc at ibm.com/support/techdocs.) The service definition points to the provider definition with the serviceref= element. That's how the connection is made between the service definition and the provider definition. Unit 3-15

16 Multiple Services, Multiple Providers You are not limited to a single service, of course you may configure many services with many backend provider connections: CICS1REG CICS2REG z/os Connect BATCHREG CICS Region 1 A CICS Region 2 B C Batch Service There's no practical limit to this one, several, hundreds your choice D <zosconnectservice id="programa" servicename="programa" serviceref="cics1a" /> <zosconnectservice id="programb" servicename="programb" serviceref="cics2b" /> <zosconnectservice id="programc" servicename="programc" serviceref="cics2c" /> <zosconnectservice id="programd" servicename="programd" serviceref="batch" /> <localadaptersconnectservice id="cics1a" RegisterName="CICS1REG" ServiceName="A /> <localadaptersconnectservice id="cics2b" RegisterName="CICS2REG" ServiceName="B" /> <localadaptersconnectservice id="cics2c" RegisterName="CICS2REG" ServiceName="C" /> <localadaptersconnectservice id="batch" RegisterName="BATCHREG" ServiceName="D" /> 16 IMS The previous charts illustrated a single service, with a single provider. You are not limited to that you can configure multiple services and providers in a single instance of z/os Connect. This chart illustrates this in a somewhat high-level way. Consider the diagram in the upper-left of the chart. A single instance of Liberty Profile and z/os Connect is connecting to two CICS regions and a long-running program. Three WOLA registrations are in place with the names shown on the chart. Target backend programs A, B, C and D are deployed in the regions as indicated. The service and provider definitions are as shown four service definition, four provider definitions, and the specifics of the service names highlighted in blue. There's not really much new news here. This is really the same story we showed in the previous few charts, but replicated so four services and four providers are represented. Why four providers when there are only three WOLA registrations? Because the provider definition carries the specifics of the target program to be called. In the case of the CICS Region 2 WOLA registration (CICS2REG) there are two unique target programs to call (programs B and C). Therefore, two provider definitions to call out each program (B and C). The same WOLA registration is used for both. Note: keep in mind these are just definitions in the XML. This does not mean the z/os Connect code is having to load up four separate sets of connector code. The code is fairly efficient about such things; no heap is wasted duplicating things just because the definition has separate provider sections as shown. Unit 3-16

17 IMS and z/os Connect IMS has implemented z/os Connect in a slightly different way. The same principles apply, but the service provider is another feature to Liberty Profile, not WOLA: Liberty Profile z/os Client z/os Connect IMS Mobile Feature Service Provider IMS Connect IMS IMS Explorer ims-services.xml server.xml Administration and Configuration of the z/os Connect and IMS Mobile Feature environment Part of IMS Enterprise Suite IMS Mobile Feature is a z/os Connect Service Provider Everything else the same discovery, data conversion, authorization interceptors, audit interceptors Data conversion 17 z/os Connect works with IMS as well, but the way it operates against IMS is different from what we've seen to this point with CICS. With CICS the service provider was WOLA. For IMS, the IMS development lab extended the z/os Connect framework and provided their own service provider. This service provider then communicates with an instance of IMS Connect, which in turn communicates with the IMS region. Note: IMS can serve as both a transaction manager and a database manager. The z/os Connect support for IMS includes interacting with transactions only. z/os Connect can not at present be used to access data through the IMS database function. This is an example of z/os Connect being extensible. The developers implemented the service provider interface (SPI) of z/os Connect to create this IMS-specific service provider. Everything else about z/os Connect we've discussed up to this point (and what we'll discuss after) is the same. That's because his IMS service provider operates within the z/os Connect framework and that illustrates the value of z/os Connect being extensible. Anyone can write service providers for z/os Connect. In this case it turns out another development group in IBM did it. This custom service provider is supported by a second configuration XML file called ims-services.xml. This provides definitions related specifically to the IMS Mobile Feature. Further, when an instance of Liberty Profile z/os is running this IMS Mobile Feature, it can be accessed by a copy of IMS Explorer (which runs in Eclipse and is a workstation-based graphical interface tool) to configure and manage the Liberty Profile server in which all this runs. Note: You will hear some speak of this as the IMS Gateway Server. When they do, they are referring to the combination of the Liberty Profile z/os server, z/os Connect, and the IMS Mobile feature. Also, it is possible to have this IMS Mobile feature and WOLA at work simultaneously in the same Libety Profile server instance. That would provide access to IMS, CICS and long-running programs. This functionality is part of the IMS Enterprise Suite. The URL second from the bottom on the chart provides an overview of the IMS Enterprise Suite offering. The URL at the bottom provides a location where the IMS feature can be downloaded. Unit 3-17

18 Data Conversion Converting JSON to the data format of the backend program 18 Unit 3-18

19 Data Conversion at a High Level This subject focuses on converting JSON to the data format expected by the backend system: JSON COBOL { } "account": " ", "firstname": "Smith", "lastname": "John", "action": "Balance", 01 account PIC 9(8). 01 lastname PIC X(16). 01 firstname PIC X(12). 01 action PIC X(8). Transform this into this and back again To do this, z/os Connect needs to understand the relationship between the two formats so it can do the transformation. That's the role of the data tranform utility, which produces bind files and JSON schema files 19 Passthrough or data convert The basic concept of data conversion is fairly straight-forward if the backend program has a data format it expects, and z/os Connect is receiving JSON from the client, then z/os Connect needs to understand both JSON and the backend so its conversion utility can transform between the two. The data conversion utility provided with z/os Connect is based on the CICS web services conversion utility, which transforms COBOL COPYBOOK into bind files and JSON schema. Notes: The data conversion function of z/os Connect is one of the extensible interfaces, so if the supplied conversion utility does not meet your needs, you can write your own data conversion utility and plug it into the z/os Connect framework. Your backend may not require data conversion that is, it may accept JSON, in which case z/os Connect can be configured to simply pass-through the JSON to the backend. If z/os Connect exposes an API interface for JSON conversion to a backend program, it would be helpful to have a way to discover (or query) for the JSON schema the interface requires. z/os Connect provides that as one of its REST APIs for management. We'll see that later in the unit. The conversion utility for z/os Connect is Java-based, which means it's ziip-offloadable work. Unit 3-19

20 With No Data Conversion, JSON in and JSON out Data conversion is optional; but if you don't use it, then understand that z/os Connect will pass JSON unchanged to backend and expect JSON on return: Client Request JSON Request JSON If a service has no data conversion definition, then z/os Connect will simply pass the received JSON to the backend program unchanged This will work only if the backend program understands JSON and the JSON format passed back If z/os Connect passes JSON to the backend program, then it expects to get JSON in return Response JSON If something other than JSON is returned, then an error is thrown by z/os Connect Other Format You as administrator of z/os Connect must have some knowledge of the data requirements High-level schematic 20 z/os Connect takes in JSON data from clients. It is not designed to handle anything but JSON. JSON is what the clients must send. But the backend program may not understand JSON. Or it may. That's something you as administrator of the z/os Connect configuration must understand. z/os Connect provides a data conversion function to map JSON to whatever data structure the backend program expects, and map the response back to JSON for the return. It's an optional function you don't have to to use it. But if you don't use it, then you should understand what to expect. If you choose to omit the data conversion function, z/os Connect will pass the received JSON unchanged to the backend program. What the backend program does with the JSON is up to the backend program. If it doesn't understand JSON it'll throw some kind of error. However, if the backend program understands JSON and no data conversion is done by z/os Connect, then that backend program must return JSON. z/os Connect expects JSON to be returned when JSON is passed back. If JSON isn't returned, then z/os Connect will throw an error. That's the key message of this chart you may forego use of the data conversion function of z/os Connect if you know the backend program accepts JSON, but you must make sure the program returns JSON as well. If so, then z/os Connect will be happy and return the JSON to the original requesting client. Unit 3-20

21 High-Level Schematic of Data Conversion Process z/os Connect uses a utility based on the CICS data conversion utility. It involves creating bind files and making those files available to z/os Connect: Supplied Data Conversion Shell Script Data structure (example: COPYBOOK) Supplied JCL to run the shell scripts and define input and output locations Generated Bind and JSON Schema Files This tells z/os Connect where files can be found and the suffix values <zosconnectservice id="cics" invokeuri="/mycicsbackend" servicename="cics-backend" dataxformref="xformjson2byte" serviceref="wolacics" /> <zosconnectdataxform id="xformjson2byte" bindfileloc="/u/user1/bindfiles" bindfilesuffix=".wsbind" requestschemaloc="/u/user1/json" responseschemaloc="/u/user1/json" requestschemasuffix=".json" responseschemasuffix=".json"> </zosconnectdataxform> 21 More detailed This chart is showing the essence of this function. It comes in two parts (1) the generation of bind and schema files from the data structure definitions of your target program; and (2) definitions in the server.xml to tell z/os Connect where the files are located. The generation of the bind and schema files is done using a supplied utility, which is based on a similar utility supplied with CICS for its web service function. The utility is invoked with JCL and takes as input the data structure of your program (for example, the COPYBOOK structure) and outputs bind files (binary files that map data structure fields to JSON) and the associated JSON schema files. For each target program in the backend you would generate the bind and schema files. Then you tell z/os Connect where the files are located. Define a zosconnectdataxform element as shown, and you point to it from the service definition. That zosconnectdataxform element provides information about the location of the bind and schema files. You may have multiple locations for such files. If so, then provide different zosconnectdataxform sections in the server.xml, and point to whatever one you wish to use for a given service. z/os Connect will go looking in that location for the requested files. Unit 3-21

22 More Detailed Schematic With Notes This is from the WP Quick Start Guide. See speaker notes for explanation of numbered blocks: WP at ibm.com/support/techdocs 22 REST APIs This is a more detailed layout of the process of creating and using the data conversion utility to create the bind and JSON schema files. This come out of the WP Quick Start Guide. The numbered blocks relate to the following notes, also from the Quick Start Guide: 1. The starting point is the request and response data structure for the program. These must provided in a PDS member. Just the data structure elements are provided here, not any of the other program source code. 2. A JCL job is used to call the BBGLS2JS JCL procedure. It's role is to read the inline parameters you supply (block 3) and call the BBGLS2JS procedure (block 4) 3. The inline parameters provide the utility knowledge of where the data structure PDS (1) is located and what member names to read in, and various other things to control the generation of the data conversion bind files. 4. The BBGLS2JS (LS2JS = language structure to JSON structure ) procedure is supplied with z/os Connect. This is called by the job (2). No changes are needed to this procedure. 5. The utility writes bind files and JSON schema files out to the UNIX Systems Services location you specify in the inline parameters (3). 6. Updates to the server.xml of the Liberty Profile server provide z/os Connect with knowledge of where the bind and schema files are located. Unit 3-22

23 The REST APIs Using the supplied REST APIs to monitor and manage the environment 23 Unit 3-23

24 REST APIs in Context The REST APIs we now refer to are supplied with z/os Connect. They are used to discover services and manage the z/os Connect instance: Client The service APIs you define in server.xml for your backend programs Liberty Profile z/os CICS z/os Connect IMS Role ID The supplied REST APIs these will not appear in server.xml they are built-in Batch This Role ID is a very loose term at the moment initially all IDs will have access to all REST APIs, but later we'll discuss authorization interceptors which provide a more granular access model to the APIs 24 APIs z/os Connect supplies a series of REST URI patterns that can be used to discover configured services and manage them. The REST URI pattersn or APIs are built into the z/os Connect function. You will not see reference to them in the server.xml file. On the next chart we will see a listing of them and what they provide. On this chart we take a moment to keep clear in our minds the difference between the URI patterns for your exposed services and the APIs exposed by z/os Connect itself. For the purposes of keeping them straight, we'll call the supplied interfaces REST APIs, and we'll call the interfaces you configure configured services. In this chart we show the REST APIs being accessed by a Role ID. That's a very generic term at the moment right now with the security model we currently have in place that Role ID is the ID used to access z/os Connect. If an ID has that access, they have access to use any of the REST APIs. But in the security unit we'll show how z/os Connect also provides a way to make access more granular. There are three access roles administrator, operator and invoke that you can assign IDs to. But we won't go into that right now because it clouds the APIs story a bit. We'll save that for the security unit. Unit 3-24

25 The Supplied REST APIs for Discovery / Management Here's a listing of the APIs and a brief description of what each provides: 25 Returns a JSON object with the configured services the user is authorized to access Returns a JSON object with the details for the named service Returns a JSON object with the status of the service (started or stopped) Returns a JSON object with the request schema for the named service Returns a JSON object with the response schema for the named service Returns a JSON object with request statistics for the named service Stops the named service and returns a JSON object with the status Starts the named service and returns a JSON object with the status These can be used by developers to understand the APIs exposed by z/os Connect and by administrators to control services Example These are the REST APIs exposed by z/os Connect. The blue service_name in the URI patterns is where you'd put the service name as configured on the servicename= element in the service definition. Note: if your service has the invokeuri= pattern to create an alias for invoking the service, the REST APIs will still work, but you have to use the servicename= value, not the invokeuri= value. The first API on the chart simply lists back all the configured services as seen in the server.xml file. That's a way to discover the configured services in a summary format. The next API is a way to get details on a specific service. The action=status API will return the status of the service started or stopped. At the bottom of the chart you see APIs that can be used to start or stop a service. A stopped service will throw an HTTP 404 error, which is effectively the same thing as the service not existing. The schema APIs will return the request and response schemas if data conversion is in use. The getstatistics API will return some basic numbers on usage. It's not as robust a source of usage statistics as the audit interceptor (the SMF writer), which we'll see next. Unit 3-25

26 REST API Example Get List of Configured Services Here's an example of the REST APIs in action: REST API to get a list of configured services In this example two configured services are returned 26 Example Here's an example of a REST client issuing the API to get a list of the configured services for an instance of Liberty and z/os Connect. In this example, it returns a JSON object with two configured services. If you had 100 configured services, the returned JSON object would show all 100 services. Unit 3-26

27 REST API Example Get Request Schema Here's an example of the action=getrequestschema REST API: getresponseschema is also available This allows a developer to not only know what services are available, but what the invoke pattern is and what the JSON request/response patterns are. They may then write their REST/JSON client to the API. API Management utilities can query this interaface and catalog the results, providing a repository of API information for developers to use. 27 Audit interceptor Knowing that a service exists is not enough a REST client also needs to know what JSON to send in and what JSON to expect. The action=getrequestschema and action=getresponseschema APIs will return the configured data transformation JSON schema patterns for the service. With that, a developer can create the REST client to use against this service. Unit 3-27

28 Interceptors Callout points for each request received 28 Unit 3-28

29 High-Level of Interceptor Framework Think of interceptors as code that is called for each request for a service: Request Received Interceptor Configured? Interceptor List Interceptor 1 Interceptor 2 : Interceptor n If a service has a reference to an interceptor list, the interceptors defined in that list are invoked for each each request that is received by z/os Connect Call Call Backend Notes about interceptors 29 This chart shows a very high-level representation of the concept of interceptors. An interceptor is simply code that is called for each request that is received by z/os Connect for a service. Rather than restrict interceptors to a single bit of code that is called, z/os Connect invokes a list of interceptors. That list may contain from one to many references to interceptor code to call for each request. On the next page we'll provide some more detail on interceptors... Unit 3-29

30 Overview of Interceptor Framework Interceptors represent optional code that is called upon receive of each request that comes into z/os Connect: Optional You may configure them if you wish, or not, depending on your needs. May be configured at global level (all) or service level (some) This allows you to apply interceptors to those services you wish, without necessarily having to crawl through every one updating XML. It is also possible to opt out of globally defined interceptors. Quite a bit of flexibility. Uses interceptors lists A interceptor list is simply a bit of XML that specifies which interceptors to invoke and the order in which to invoke them. You can define multiple lists, with some services calling one, and some services calling another. Extensible z/os Connect comes with a Service Provider Interface (SPI) for interceptors so you can write your own if you wish. Two IBM-supplied interceptors IBM supplies two: one writes SMF records, the other is provides a way to limit what the authenticated user can do in z/os Connect Audit Interceptor 30 The topic of interceptors can be a little slippery if you jump straight into the syntax of the server.xml. To start with we'll approach it from a higher level first, then later get into how they are configured. Interceptors are designed to invoke code on each request received. This provides a way for z/os Connect to temporarily pass control to other code for some processing for the request. Note: it's important to understand that the interceptor framework does not allow interceptors to change the JSON payload coming in or going out. Interceptors can get a copy of the JSON, but interceptors can not modify the payload. Interceptors are optional it is not required you use them at all, or if you do use them, it is not required you use them on all services. They are configurable either at the global level so they apply to all the services defined, or at the service level so they apply only to specific services, or a mixture of both. This is where the XML syntax can get confusing the combination of global, and service-level, and how the two relate can be tricky to understand at first. Interceptors are configured as part of a list. A service is not limited to invoking just one interceptor. It can invoke a chain of interceptors if you want. That chain is defined in the XML as a list of interceptors, and if a service calls the list then z/os Connect will work through the interceptors in the list and call them, one after another, in the order you specify. The z/os Connect interceptor framework is designed to be extensible, meaning you may write your own interceptor if you wish. The interceptor framework comes with a Service Provider Interface (SPI) and a class library for development of interceptors. You develop your own, register it to Liberty Profile and call it through the configuration in the XML. Finally, IBM supplies two out of the box interceptors for you to use one is called the audit interceptor and it writes SMF records; the other is called the authorization interceptor and it provides a way to limit the scope of activities a given user of z/os Connect may perform. We'll briefly touch on each next. Unit 3-30

31 Brief Review of Audit Interceptor The audit interceptor is a supplied piece of code that will write SMF records. Those records contain information on z/os Connect usage: System Name Sysplex Name Jobname Job Prefix Address Space Stoken Arrival Time Completion Time Target URI Input JSON Length Response JSON Length Method Name Service Name Userid Server Identification Section z/os Connect User Data Section Notes: Liberty Profle z/os itself does not cut SMF records; this is a function of z/os Connect SMF is relatively new format, so your analysis tool may or may not understand that format yet No CPU numbers in Can capture CPU at started task level for Liberty z/os Connect is a pass-through function; it will use some CPU (mostly ziip). Multiple services same backend work 31 The audit interceptor is really a function of z/os Connect that writes SMF records for those services configured to have them written. SMF is a facility of z/os that provides a highly efficient means of writing records capturing information about activity on the system. There are many different SMF record types, each specific to the program or subsystem that's writing the record. For example, CICS writes 110 records. The full-function WAS z/os product writes 120 records. The record we'll referring to for z/os Connect is a subtype of 120 the 120 subtype 11, or records. Note: it's worth nothing that Liberty Profile itself does not write SMF records. This SMF capability of z/os Connect is a function of the z/os Connect feature of Liberty. If you want to capture statistics for the Liberty Profile server itself, you'd use the SMF Type 30 record for the address space. The chart shows the record layout for the SMF record. A record is cut for each request received by z/os Connect for the service configured to write SMF records. A common question is whether the SMF record captures CPU statistics; specifically CPU on GP and speciality engine usage. The answer is it does not. Those numbers can be captured with the address space SMF Type 30 records. That would not provide CPU statistics at the individual service level, but it would provide numbers for the Liberty Profile server acting as the z/os Connect gateway. z/os Connect uses some CPU, but keep in mind that (a) z/os Connect is a pass-through function for the most part, with the data conversion routines being perhaps the user of CPU, and (b) it's written in Java, so it would be dispatched to the speciality engines of the LPAR. Unit 3-31

32 Two Services Invoking Same Backend Work Having two defined services invoke the same backend work is doable. But why? Liberty Profile z/os CICS z/os Connect CICSREG TRANXYZ <zosconnectservice id="mobile Traffic" servicename="mobile" invokeuri="/mobile" serviceref="cicswola" /> <zosconnectservice id="internal Traffic" servicename="internal" invokeuri="/internal" serviceref="cicswola" /> <localadaptersconnectservice id="cicswola" RegisterName="CICSREG" ServiceName="TRANXYZ" /> SMF Mobile <data> SMF Mobile <data> SMF Internal <data> SMF Mobile <data> SMF Mobile <data> SMF Internal <data> SMF Mobile <data> : SMF Mobile <data> 32 Authorization interceptor A question that comes up is how to differentiate two (or more) channels of input to work on the LPAR. One potential means of accomplishing this using z/os Connect is to configure multiple services that map to the same work in the back-end. With the SMF capability of z/os Connect, you can then capture the number of requests that are coming in through each service. Note: this does not capture the CPU for the work accomplished in the backend systems. But it does provide a way to identify and quantify the requests. If those requests can be equated with work in the backend that you can measure separately, then it gives you a way to arrive at an estimate of backend work attributable to the input channel. Unit 3-32

33 Brief Review of Authorization Interceptor The authorization interceptor is a supplied piece of interceptor code that will check to see if the user has the authority to perform the action requested: This is what the interceptor provides Fred This is controlled by a defined role Allowed to Enter? No Go Away Yes Administrator Full authority Operator Start, Stop, etc. The mechanics of how this is done and how it is configured is a topic for the unit on security Invoke Invoke service only 33 Security The other interceptor supplied with z/os Connect is the authorization interceptor, which if configured provides a way to differentiate users to certain levels of authority. The authorization interceptor provides three levels of authority administrator (full authority), operator (the ability to use the REST APIs to control the configured services, such as start and stop, or get the JSON schema), and invoke (the ability to invoke the configured services, but not use the REST APIs). The authorization intereceptor, like any interceptor, is optional. If you do not configure it then a user who has access to the z/os Connect servlet (via the application role) will have administrator authority. When the interceptor is configured, then you may control at the user level. This interceptor can be configured at the global level (all configured services), at the service level, or a combination of the two. This is where the behavior can become a bit perplexing at first. To help you understand, a guide has been published at the WP Techdoc (ibm.com/support/techdocs) called Interceptor Scenarios. This provides a series of five different scenarios of the authorization interceptor showing combinations of global and service-level and how things would work. That's all we're going to say about this interceptor at this point in time. The security unit, which comes next, will go into more detail on security overall for z/os Connect, as well as the authorization interceptor itself. Unit 3-33