Understanding Java Batch and WebSphere XD Compute Grid

Size: px

Start display at page:

Download "Understanding Java Batch and WebSphere XD Compute Grid"

Sydney Morgan Allen
5 years ago
Views:

1 Understanding Java Batch and WebSphere XD Compute Grid Gregg Lippa Senior Technical Analyst Themis Education Themis, Inc. Visit us at: Also:

2 This Webinar Addresses These Questions Why should you use Java-based applications in z/os batch environments? What alternatives exist for executing and controlling Java batch jobs? What are the advantages of using the Java EE framework for batch jobs? What are the advantages of using WebSphere XD Compute Grid? What are the capabilities of WebSphere XD Compute Grid? How do batch workloads operate in Compute Grid? What are the responsibilities of the batch container? What are the Compute Grid components and the purpose of each? What useful design patterns exist to support modern batch jobs? What is the role of JZOS in Compute Grid? We will not have time for questions until the end Please submit questions via to 2

3 Legacy System Lifecycle Application lifecycle Continues beyond initial development and implementation Includes maintenance, modernization and replacement Eventually, ongoing maintenance is no longer acceptable Obsolete user interfaces, incompatible operating environments, etc. One simple approach is to implement a wrapping layer Provides newer interfaces using techniques such as screen scraping Advantages of screen scraping Tools exists to facilitate transformation of existing applications Protects the investments in existing applications Much less risky than replacement with a newer system However, underlying applications are not modernized Makes these solutions difficult to maintain 3

4 Legacy System Replacement Replacement Last phase of legacy system modernization process Legacy system is completely replaced with a new system Involves much greater risk No guarantee that new system will provide better service to users in spite of good intentions and design of the new application The legacy systems must not be dismissed lightly They have been supporting the business successfully for many years New systems may need to access resources based on z/os Such systems may also benefit from Java batch components Opportunities to modernize application portfolio exist here as well 4

5 Considerations for Replacement Factors to consider when replacing a legacy application: 1. Availability of skills, especially looking toward the future Fewer COBOL developers available all the time More Java developers every day More tools to increase developer productivity in the Java realm 2. Integration with other systems over the network Java and eclipse-based IDEs offer a wide range of APIs E.g. JDBC, JMS, XML parsing, Web Services support 3. Performance as measured by application throughput Processing a large volume of data within a certain time frame Java has been improving with regard to this important factor 4. Portability allows an application to run in other operating environments Applications may move off z/os: write once, run anywhere 5. Cost! ZIIP processor can be used to run Java code 5

6 Java Stand-Alone Application Scenarios Some scenarios that lend themselves to Java: Data processing jobs Large volumes of data processed in batch Flat files, VSAM data sets or DB2 data Messaging to communicate with other environments Can use JMS to feed a remote queue Allows an application to integrate with other environments Generating and printing reports Large amounts of print work to be completed in batch May utilize print servers, terminal servers and HTTP servers 6

7 Batch Applications Features of batch applications Start and stop time are not critical Response times measured in seconds are not expected No terminal or browser is waiting for output from the job Batch jobs typical process a large workload Repeat a task thousands or even millions of times More efficient processing can be achieved using this approach Throughput can be increased via batch execution Can make use of advanced workload management techniques 7

8 Overview of Stand-Alone Java Batch Figure 7-1 Overview of stand-alone Java batch Source: Batch Modernization on z/os (sg247779) p.77 8

9 Job Management Run-time environment of a Java z/os batch program Compiled Java program, often in a jar file JCL and the job execution environment Java program executed via JCL as a batch job Execution via BPXBATCH, BPXBATSL or JZOS Integrating legacy and modern applications using JCL Job execution management using JCL conditions 9

10 Using JCL for Java Batch Execution IBM recommends using JCL to run Java batch Can be done by using BPXBATCH, BPXBATSL or JZOS Why use JCL? Behaves like regular batch work, with JES2 and job scheduling support Supports batch security environment Can organize the job flow Combining Java program execution with other JCL steps to complete a business process Simplifies getting guaranteed CPU resources via WLM and getting memory for the JVM Can direct output to the spool for further handling or archiving 10

BPXBATCH, BPXBATSL and JZOS BPXBATCH has limitations: Unable to handle standard I/O streams when run via JCL Executing programs cannot access DDs defined in enclosing JCL BPXBATSL is an alternative

11 BPXBATCH, BPXBATSL and JZOS BPXBATCH has limitations: Unable to handle standard I/O streams when run via JCL Executing programs cannot access DDs defined in enclosing JCL BPXBATSL is an alternative utility program Addresses the limitations of BPXBATCH DD cards specified in JCL available to invoked program JZOS has full support for DD statements Supports temporary data set to pass output from one step to a later step Directs stdout and stderr streams to standard MVS data sets and JES SYSOUT Easy integration of Java programs with other job steps Tivoli Workload Scheduler Source: Batch Modernization on z/os (sg247779) p

12 Java Batch and Compute Grid Today s z/os batch workloads use COBOL or PL/I to perform business logic and process large amounts of input and output data As we have seen, Java batch applications can now run on z/os However, there can be disadvantages: JVM is created for every batch job More CPU usage; additional resource consumption; greater overall cost Managing JCL via a workload scheduler can help to mitigate these costs Traditional WebSphere Application Server for z/os containers Allow Java execution to utilize the On Line Batch Processing paradigm WebSphere Extended Deployment (XD) Compute Grid for z/os Provides a batch container for Java EE applications Extends the existing WebSphere core functions 12

13 WebSphere XD Compute Grid Java EE platform provides important advantages: Supports concurrent Java batch and OLTP workloads effectively Simplifies and improves management of complex system operations Application versioning support Reduced cost and complexity of managing WebSphere resources Allows greater focus on the WebSphere to configure resources Helps increase enterprise s adaptability to business change Optimizes the performance of business-critical applications Maximizes scalability for high volume transaction processing Takes advantage of existing Java skills and resources 13

14 Java Enterprise Edition (Java EE) WebSphere has become increasingly important Popular environment for modern, enterprise-class business applications Java EE skill base is rapidly growing in the industry The Java language and Java EE servers offer many advantages Java can reduce application development cost Because it is object oriented, Java can also reduce development time Powerful Java applications and frameworks are available Further improves the cost and time requirements for application development JVM performance is continually improving 14

The Java EE Environment on z/os WAS supports enterprise-class Java-based business applications Typical OLTP workload in WAS includes various types of clients: - Web clients - MQ messages - EJB

15 The Java EE Environment on z/os WAS supports enterprise-class Java-based business applications Typical OLTP workload in WAS includes various types of clients: - Web clients - MQ messages - EJB clients - Web Service clients WAS-based applications are often transaction-oriented Business logic written in Java is invoked to create a response for the client WAS is able to manage all of the transactional processing Source: Batch Modernization on z/os (sg247779) p.95 15

16 WAS System Services WebSphere provides many system services (container services) Container services include Security management Transaction support Web service support Session and connection pooling Caching and Messaging The container services provide these benefits Increases application security, availability, performance, and stability Simplifies Java application design and application management Accelerates development time But WAS-based business logic is not accessible from traditional batch Integrating traditional runtime environments and WAS is based on connectors and messaging WAS-based business logic is not accessible from traditional batch 16

17 Batch Workloads for Java EE WebSphere XD Compute Grid supports batch workloads Able to run concurrently as is done in traditional batch on z/os Expands on existing z/os support for WebSphere Able to run both Java EE-based transactional and Java EE-based batch workloads WebSphere XD Compute Grid provides a batch container Existing WAS-based business logic now available to traditional batch Can be integrated into the traditional batch execution environment Can be shared across batch and OLTP applications Batch workloads require computing resources for long periods Traditionally, had to run either in a separate, isolated environment, or execute during a specific time period (such as off hours), or both Causes duplication of resources and limits ability to share resources Supports continued use of existing batch programs written in COBOL or PL/I As well as JCL and workload schedulers 17

18 Traditional Batch and OLTP Workloads in WebSphere Today Source: Batch Modernization on z/os (sg247779) p.96 18

19 Batch and OLTP Workload in WebSphere XD Compute Grid on z/os Source: Batch Modernization on z/os (sg247779) p.97 19

20 WebSphere XD Compute Grid Overview WebSphere XD Compute Grid for z/os builds on existing WebSphere Java EE programming model and container services Provides a job scheduler and an execution environment Plus additional features designed for long-running batch applications Traditional WebSphere Application Server Provides an environment for hosting Java enterprise OLTP style applications WebSphere XD Compute Grid extends that capability Provides an environment for deploying Java batch applications Modeled after the z/os Job Entry Subsystem (JES) using 1. Job Control Language (JCL) 2. A job dispatcher 3. Job executors (JES initiators) 20

21 Batch and OLTP Workload in WebSphere Source: Batch Modernization on z/os (sg247779) p.98 21

22 Compute Grid (CG) Applications What are WebSphere Compute Grid components? Answer: Java EE applications An XML-based job description metadata called xjcl A job dispatcher called the Job Scheduler (JS) Multi-threaded job executors (batch container) Called Grid Execution Endpoints (GEE) A Parallel Job Manager (PJM) component Manages execution of parallel batch jobs Components can be placed in one JVM or in a clustered environment Support WAS security, scalability, high availability, and life cycle management 22

23 Role of JZOS in Container Managed Batch Source: Batch Modernization on z/os (sg247779) p

24 Compute Grid vs. JZOS JZOS delivers both the JZOS Launcher and the Java z/os APIs JZOS allows a stand-alone Java program to be invoked using JCL Its role in a Compute Grid environment is to integrate Java and traditional z/os The JZOS launcher has several drawbacks making it inefficient for the high volume of batch jobs that run within a batch window It lacks these capabilities: Security, transaction, or connection management facilities Checkpoint / restart facility for batch jobs High availability and other quality of service (QoS) provided by WAS A persistent, reusable JVM each Java-based job step requires reloading the JVM Compute Grid offers all necessary QoS and services including: Integration with z/os WLM to enhance job execution and management Persistent and reusable JVM and execution container Security, transaction, and connection management Prioritization and high availability Thread pooling 24

25 Basic Flow of a Batch Application Source: Batch Modernization on z/os (sg247779) p

26 Programming with Compute Grid WebSphere XD Compute Grid batch container Runs as an extension to a standard WebSphere Application Server (WAS) Contains and controls a set of POJOs comprising the batch application Batch container controls and runs batch job Uses a container-managed thread that defines the unit of execution Carries out the life cycle of the job running in the batch container Batch application components you provide: Batch job step (exeuted by the batch container) Provides the business logic to execute as a step in a batch job Batch data stream Provides the batch job step with access to data Relational databases File systems Message queues J2C connectors of other types of data One or more batch data streams may be accessed by a batch application 26

27 Application Components Relationship to the Batch Container Source: Introduction to batch programming using WebSphereExtended Deployment Compute Grid p.3 27

28 Batch Container Responsibilities During the life cycle of a job step, the batch container is responsible for open, close, and checkpoint-related callbacks on a batch data stream Methods on the batch data stream to get and put data are called by the batch job step Batch application may include either or both of these user-provided components: Checkpoint algorithm Results algorithm 28

29 Checkpoint Algorithm Container provides a mechanism to support job restart from a known-point of consistency Interrupting and then restarting a job might be necessary After a planned or unplanned outage Batch container periodically calls checkpoint algorithm To determine whether to take a checkpoint Compute Grid provides two pre-built checkpoint algorithms One that supports a time-based checkpoint interval One that supports a checkpoint interval based on record-count 29

30 Results Algorithm A return code is supplied upon completion of each job step Return codes from all steps in a batch job are visible to the results algorithm The overall return code for the job as a whole is returned A provided results algorithm returns the highest step return code as the overall job return code 30

31 Job Life Cycle Source: WebSphere XD Compute Grid Parallel Job Manager System Design & Architecture Document p.8 31

32 Modern Batch Patterns Modern Batch implementation in RAD* includes Batch step patterns (below) Input/output stream patterns Patterns for checkpoint algorithms and results algorithms Support restart processing (next page) Batch Step Patterns Batch Step Pattern Generic Batch Step Error Tolerant Batch Step Description A simple step that uses one input and one output stream A simple step that uses one input, one output, and one error stream * Rational Application Developer 32

33 Threshold Patterns Supports restart processing Applicable if using the threshold batch step of ThresholdBatchStep Threshold Pattern RecordBasedThresholdPolicy PercentageBasedThresholdPolicy Description Counts the number of error records processed. If the result is greater than the threshold, it forces the job to go into a restartable state. Calculates the percentage of the number of error records processed to the total number processed. If the result is greater than the threshold, it forces the job to go into a restartable state. 33

34 Compute Grid (CG) Interfaces CG batch programming model utilizes several interfaces The first two below are essential to building a batch application The other two below are optional (intended for advanced scenarios) Required interfaces BatchJobStepInterface Defines the interaction between the batch container and the batch application BatchDataStream Abstracts a particular input source or output destination for a batch application Defines interaction between Compute Grid and a BatchDataStream implementation Optional interfaces CheckpointPolicyAlgorithm Defines custom checkpoint policy implementation for Compute Grid to use Used to determine when Compute Grid will checkpoint a running batch job ResultsAlgorithm Defines a custom results algorithm for Compute Grid to use Used to provide the overall return code for a job Has visibility to the return codes from each of the job steps 34

35 Simplified Batch Programming Model Source: Batch Modernization on z/os (sg247779) p

36 Compute Grid Unit Test Environment Provided local Compute Grid unit test environment Batch simulation inside the Eclipse workspace or on z/os Executes in a standalone WAS instance Batch Simulator is a testing aid Included with Compute Grid Allows initial testing of batch POJOs inside Eclipse Runs in a J2SE environment rather than the Java EE environment supported by Compute Grid (which runs in WAS) Useful for testing business logic of POJOs in the application 36

37 MVS Data Sets Enterprise data is stored in files and relational databases Java DataBase Connectivity (JDBC) supports access to relational data JDBC APIs can be used in a stand-alone environment Libraries with classes and beans are available for file access A Java application running in the UNIX environment may require access to MVS data sets No problem using Java to access files stored in UNIX format in HFS How does Java access these MVS data sets? The JZOS toolkit library provides the answer 37

38 JZOS Toolkit Library JZOS toolkit library supports mainframe file systems Not available via the standard Java I/O library JZOS provides functions to access MVS data sets Allows Java programs to access any MVS data sets: Sequential files Partitioned Data Set (PDS) Partitioned Data Set Extended (PDSE) Virtual Sequential Access Method (VSAM) KSDS, RRDS, and ESDS 38

39 ZFile Class Constructed with two arguments: 1. The filename refers to a dataset name or a DD name 2. The mode string is used, among other things to determine if the dataset is to be read in record or stream mode Open the DD name MYDD for reading: ZFile dd = new ZFile("//DD:MYDD", "r"); Open ACCOUNT member in SYS1.HELP for reading text ZFile dsn = new ZFile("//'SYS1.HELP(ACCOUNT)'", "rt"); Open {MVS_USER}.SEQ for sequential binary writing ZFile dsn = new ZFile("//SEQ", "wb,recfm=fb,lrecl=80"); Sample ZFile instance methods: getrecfm() returns record format of dataset (F, V, U, FB, or VBS) getlrecl() returns logical record length of dataset 39

40 Working With VSAM Data Sets Types of VSAM data sets: Key-Sequenced Data Set (KSDS) - Entry-Sequenced Data Set (ESDS) Relative Record Data Set (RRDS) - Linear Data Set (LDS) JZOS provides support for all VSAM data sets except LDS VSAM terminology: Logical record: unit of data stored in a VSAM data set Contains fields of data and a key, used to retrieve a specific record Cluster: set of related components comprising a VSAM data set VSAM data sets are referred to by their cluster names Contains a data component and an index component IDCAMS: utility program used to define VSAM data sets Data sets must be allocated before Java applications can work with them JZOS does not support allocation of VSAM data sets 40

41 Accessing VSAM Data Set with JZOS JZOS provides several methods to manipulate VSAM data sets ZFile has methods for adding, deleting and updating records in a VSAM data set Also methods to get information such as logical record length (LRECL) ZFile supports reading a portion of a record (useful for large records) Read a record from a VSAM file: This code opens a file; locates a record by matching its key using the offset to the start of key buffer; and reads the record into byte array, recbuf ZFile zfile = new ZFile(filename, options); zfile.locate(key, offset, keylen, ZFile.LOCATE_KEY_EQ)); int nread = zfile.read(recbuf); 41

42 ZFile Methods for VSAM Data Sets Commonly used methods of a ZFile instance: getlrecl Get the native file's logical record length getvsamkeylength Get length of key of the VSAM data set getvsamtype locate read update write delrec Return the VSAM type Locate a record given a key, record number or RBA Read from the native file into the supplied buffer Updates record at current position of cursor Writes the buffer to current position of cursor Deletes record at current position of cursor read( ), write( ) and update( ) receive an array of bytes Array size should match size of record read, written, or updated JZOS methods can also take as input a byte array of any size 42

43 Creating Java Record Classes z/os datasets records are often described by COBOL copybooks Contain fields that conform to the set of valid COBOL types Alpha, alpha-numeric, unscaled numeric, packed decimal, binary, etc. A Java program may need to process this kind of dataset May be difficult to convert from COBOL to Java datatypes Can build Java record class programatically from a copybook Automated process uses input from the COBOL copybook JZOS includes development time tools for doing this Offers a complete set of field converter types Convert from COBOL types to Java types A Java class can be hand coded to provide access to COBOL described record fields from a Java program Much more difficult to hand code these record structures 43

44 Summary WebSphere Extended Deployment Compute Grid Provides a simple abstraction of a batch job step and its inputs and outputs Concise, straightforward programming model that is easy to use Supports job scheduling capabilities and life cycle management Built-in checkpoint/rollback mechanism Makes it easy to build robust, restartable Java batch applications Supports compute-intensive processing as well as data processing type Automates parallel processing capabilities Supports integration with legacy host resources (e.g., COBOL and z/os files) Supports utilization of Java EE resources built for OLTP Is supported by a variety of eclipse-based tools to assist in development, packaging, deployment and testing 44

45 Learning More About Java, OO and Database Interaction Java 1 (Intro Java) and Java 2 (Intermediate Java) These courses spend two weeks in Java Standard Edition Both courses are necessary to get a complete background Java Enterprise Servlets and JSPs One week course on these critical components of Java EE Object Oriented Analysis and Design This course gives the budding OO developer a solid background in OO requirements specification and UML Many other courses depending on your environment Enterprise Java Beans (EJBs); Java Persistence API (JPA); Java Messaging Service (JMS); Java Server Faces (JSF); Web Services with Java; Design Patterns; Spring; XML; HTML; Hibernate; Struts; many more 45

Thank you for coming www.themisinc.com US 1-800-756-3000 Intl.

46 Thank you for coming US Intl On-site and Public Instructor-led Hands-on Training Hundreds of IT Courses Customization Available

WebSphere Java Batch WP at ibm.com/support/techdocs Version Date: September 11, 2012

WebSphere Java Batch WP at ibm.com/support/techdocs Version Date: September 11, 2012 WebSphere Java Batch Version Date: September 11, 2012 Agenda Business Pressures on Traditional Batch IBM WebSphere Java Batch Overview IBM WebSphere Java Batch Feature Focus IBM WebSphere Java Batch for