WebSphere Java Batch. WP at ibm.com/support/techdocs Version Date: September 11, 2012

Transcription

1 WebSphere Java Batch Version Date: September 11,

2 Agenda Business Pressures on Traditional Batch IBM WebSphere Java Batch Overview IBM WebSphere Java Batch Feature Focus IBM WebSphere Java Batch for z/os Focus IBM WebSphere Java Batch Deployment Scenarios Wrap-Up Summary 2 This presentation is divided into six sections as shown on this chart. The first section -- "Business Pressures" -- provides some explanation of the business pressures facing batch processing. This sets the context for the information on WebSphere Java Batch that follows. The next section -- "WebSphere Java Batch Overview" -- provides a framework of understanding for what "WebSphere Java Batch" is and how it operates. The third section -- "Feature Focus" -- takes a closer look at some key functions provided with WebSphere Java Batch. This provides a better sense for the capabilities of the function. The fourth section -- "z/os Focus" -- provides information on how WebSphere Java Batch for the z/os platform provides z/os-specific benefits to batch processing. The fifth section -- "Deployment Scenarios" -- paints a picture of several ways in which WebSphere Java Batch might be deployed and used. The final section -- "Summary" -- provides a high-level wrap-up of the information provided in the earlier parts of the presentation. 2

3 Business Pressures on Traditional Batch Or, "Hey, who closed my batch window?" 3

4 Concept of "Dedicated Batch" Window Going Away Windows of time which used to be dedicated to batch processing are shrinking. The demands of online processing require more and more x 7 x 365 Access Users of your online systems expect availability at all hours Users from other parts of the world means availability is expected around the clock In the past... Today... Online Online Batch Batch Mobile Users Users are no longer tied to a desk and a computer. Today users have access to mobile computing devices that are with the user wherever they may be. Day or night, home or office. The need to process batch work has not gone away. The need to perform the work concurrent with OLTP has emerged. 4 There used to be a time when "batch" and "online" processing were quite separate from one another. It used to be the case that online processing was stopped altogether so batch processing would have access to the system resources to complete its work. But those days are well behind us. Online processing is becoming more and more a 24 x 7 operation. This is because client access is increasingly global, with people from different time zones seeking access at all times during the day. There may be times during the day when online processing is greatly reduced, but in most cases it is never stopped altogether. That means batch processing much work at the same time as online. To the degree it has dedicated time, that window of time is shrinking. The advent of mobile devices means client access is now even more frequent than before. The transaction work that flows back to information processing systems because of mobile device activity is increasing. And mobile device activity may occur at any time of night or day. It's truly a 24 x 7 world for online processing. But batch processing has not gone away. There are still requirements to do batch work. So it's evident batch and online must be done at the same time... and in a manner that implies both work cooperatively. 4

5 The Value of Shared Services It's not just that the window is shrinking... it's also the cost pressures on maintaining the batch and OLTP environments: Batch Support Staff OLTP Support Staff Common Support Staff Batch Infrastructure OLTP Infrastructure Batch + OLTP Common Infrastructure Batch Applications OLTP Applications Batch Applications Shared Java Assets OLTP Applications Batch Development Tools Homegrown Middleware Infrastructure OLTP Development Tools Batch + OLTP Common Tooling Homegrown Middleware Infrastructure Efficiencies through consolidation around common assets 5 In addition to the batch window shrinking, there are pressures to contain and reduce costs related to information processing as well. Having separate infrastructures for online and batch... separate computers, separate tools, separate support staff... is an idea that is increasingly being drawn into question. Further, custom-built mechanisms for running batch -- particularly Java batch -- represent middleware solutions that require a considerable amount of effort to maintain and support. The trend is definitely in the direction of convergence. Online and batch processing are both information processing, and as such can and should be handled in a cooperate, converged manner. This allows for efficiencies in infrastructure, staff, development tooling, and potentially Java assets shared between OLTP and batch. Homegrown middleware solutions are not favored and will eventually be phased out. 5

6 Java for Batch Processing? Yes... for many very good reasons: Availability of Skills Java is a programming language with wide adoption in the industry. Skills for Java programming are common and affordable. Tooling Support Development tooling for Java has advanced to the point where some tools (IBM Rational Application Developer) are very powerful and sophisticated. This also provides an opportunity to consolidate to a common tooling environment for both OLTP and batch development. z/os Specialty Engines Pressures on cost containment often dictate greater use of z/os specialty engines. Java offloads to zaap. Java batch does as well. Processing in OLTP Runtime Running Java batch in the same execution runtime as Java OLTP provides an opportunity to mix and manage the two processing types together under the same management model. Java is a registered trademark of Oracle 6 Which brings us to Java as a programming environment for batch processing. Some may ask, "Why use Java when other languages -- COBOL, for instance -- work well?" First, the question is not Java or COBOL, but not both. There is a very practical realization in the world that batch assets written in COBOL are still very useful. To the extent Java enters the discussion, it is as a complement to COBOL, not a total replacement. Second, there are business pressures that make Java an attractive batch solution: Skills -- Java skills are simply more plentiful than COBOL skills. That's a fact. Many organizations have very good Java development staffs. It makes sense to leverage those Java staffs for batch work as well. Tooling -- today's development tooling products are powerful and quite sophisticated. Acquiring them also represents an investment in technology and an investment in skills. It makes good sense to leverage that investment across multiple information processing disciplines. Specialty Engines -- COBOL runs on general processors, Java runs on specialty engines. Specialty engines (zaaps) provide a way to lower overall System z costs... both in acquisition costs as well as ongoing licensing costs. Specialty engines are an attractive solution and leveraging them for batch work is desirable. Cooperative Processing -- online processing runtime infrastructures are very often designed around Java. For example, IBM's WebSphere Application Server is a powerful Java processing runtime. There is an investment in maintaining that online infrastructure. So to the extent that same investment can be leveraged for batch work as well, the better. The bottom line is that Java for batch processing is very much an area of growing interest, and in fact is in use in many very large processing operations. 6

7 The Objective -- OLTP and Batch Mixed and Managed: OLTP and Batch do not need to be "either / or"... it can be "both": 11:00pm Midnight 1:00a 2:00am 3:00am Batch Processing OLTP Processing Batch Processing Compute Processing Resources OLTP Processing Batch Batch OLTP Batch OLTP Batch Batch Processing OLTP Processing OLTP Batch OLTP Processing Batch OLTP With IBM WebSphere Batch this is possible. OLTP and Batch processing within a common execution runtime (WebSphere Application Server) allows the WAS platform to mix and manage the two workload types. 7 The picture on this chart represents an objective of cooperative processing of online and batch within same execution runtime -- the managed intermixing of online work with batch work. This is possible when the execution runtime is a multi-threaded design (which IBM WebSphere Application Server is), and the execution runtime is designed to manage the many processes simultaneously (which IBM WebSphere Application Server does). IBM WebSphere Java Batch is a functional solution that is built on top the proven runtime platform of IBM WebSphere Application Server (WAS for short). WAS is today very much capable of handling concurrent online processes simultaneous. To process batch work all WAS needs is an understanding that batch is present, and a container design for handling batch work. With that we're ready to go into what IBM WebSphere Java Batch is. 7

8 Overview A high-level look at the IBM WebSphere Java Batch model 8

9 IBM Compute Grid V8 and IBM WAS V8.5 The IBM WebSphere Java Batch function is provided in two ways today: Java Batch Function IBM WebSphere Compute Grid Version 8 Compute Grid V8 function incorporated into WAS V8.5 Add the function ("Augment") Java Execution Runtime IBM WebSphere Application Server Version 7 or 8 IBM WebSphere Application Server Version 8.5 Operating Systems Supported: AIX, IBM i, Linux, Windows, HP-UX, Solaris, Linux for System z, z/os Operating Systems Supported: AIX, IBM i, Linux, Windows, HP-UX, Solaris, Linux for System z, z/os Function is identical between the two environments Compute Grid V8 available for those who have not yet migrated their execution runtimes to WAS V8.5 9 We're going to start by getting on the table a key bit of information about how IBM WebSphere Java Batch is delivered... even before we explain what exactly it is. IBM WebSphere Java Batch is delivered as part of two different products at this time: IBM WebSphere Compute Grid V8 IBM WebSphere Application Server Version 8.5 Key Point: the Java batch function provided in each is identical. Here's a quick background on the history of the function to set some context as to why it is presently offered in two different products: Originally the Java Batch function was provided as part of a product called "WebSphere Extended Deployment," or XD. XD consisted of Java Batch, another function called "Virtual Enterprise," and a third function called "extreme Scale." The three functions were then split out into separate products so customers could better choose which function they wished to employ. That was the start of WebSphere Compute Grid, which was the Java batch function. That exists today in the form of WebSphere Compute Grid V8. Compute Grid has always relied upon an existing WebSphere Application Server runtime to operate. Compute Grid is really a set of Java batch function that is added to a WAS runtime foundation. That act of adding the function is called "augmenting," which is a fancy word for incorporating the Compute Grid function into the configuration of the WAS runtime. When the WAS V8.5 product was in the planning stages, the decision was made to roll the Compute Grid (and Virtual Enterprise) function into the base WAS foundation. So that's where we are today -- Compute Grid V8, which requires an existing WAS V7 or V8 runtime onto which it is "augmented," or WAS V8.5 which has the Compute Grid function rolled into the WAS base. 9

10 Batch Container Added to the WAS Runtime At a very high-level, you may think the IBM WebSphere Java Batch function as a "batch container" operating alongside the other containers of WAS itself: Application Web Modules Web Container Container-managed Services Application EJB Modules EJB Container Container-managed Services Batch Applications Batch Container Container-managed Services WebSphere Application Server Runtime Environment Batch job dispatching and management system Data record read and write support services Checkpoint and job restart services Job resiliency services (skip record, step retry) Parallel job management and execution services COBOL module call services 10 The WebSphere Application Server product provides what's known as a "Java EE" (or "Enterprise Edition") runtime. Java EE is an industry standard specification for an application server that provides a long list of standard application specifications. Part of the design of this "Java EE" runtime is the idea of a "container." Containers are simply runtime functions that provide managed services to the programs that run in the containers. Container-managed services mean the application programs themselves may focus on their core business logic and not have to implement common functions. WAS by itself has two containers -- Web and EJB. The addition of the WebSphere Java Batch function adds a third -- a batch container. Like other containers, the batch container provides function services to the batch applications that run in the container. The chart shows a summary listing of some of those services. The purpose of this presentation is to provide more information about those services. For the purposes of this chart, just scan through the list of services and think about a Java batch program having access to those services. Then think about what it might take if you had to write those services yourself. But of course you don't have to write them yourself -- they're provided with the batch container. 10

11 Overview of the Management and Execution Model This picture illustrates some of the key components of the WebSphere Java Batch model as provided in Compute Grid V8 and WAS V8.5: 1 Job Properties Declaration File 2 Job Management Console Job Dispatching Function 1. Job Management Console (JMC) provides a view into the batch environment and allows you to submit and manage jobs 2. Job declaration file (xjcl) provides information about the job to be run, such as the steps, the data input and output streams and the batch class files to invoke 3. The Job Dispatching function interprets the xjcl, dispatches the job to the endpoint where the batch application resides, and provides ability to stop and restart jobs 4. The Execution Endpoint is a WAS server in which the deployed batch applications run 5. The development libraries and tooling assist in the creation of the batch applications 3 4 Job Execution Endpoint Batch Applications Job Execution Endpoint Batch Applications Development Libraries and Tooling Support A comprehensive Java batch execution platform Built on the proven Java runtime environment of WebSphere Application Server 5 11 Now we'll start to go a bit deeper into the technical specifics of WebSphere Java Batch. What this chart is saying -- at the very highest level -- is that WebSphere Java Batch provides a separation of several key elements of batch processing: Job submission and dispatching -- the act of having a Java batch job begin the process of execution. This is done through a defined interface. Job description -- to avoid hard-coding job properties into the application code, it's always best to externalize that information. An external job properties file is used to tell the job submission function what the job is and how to run it. Job execution -- the execution of the job takes place in an "endpoint," which is really a WAS application server where the Java batch application is deployed. The Job Dispatching function signals to the endpoint to begin execution of the batch code named in the job declaration file. Job development and deployment -- Java batch applications implement the batch logic, and they are deployed into the endpoint servers like any WAS application is deployed. Those Java batch application class files are invoked when the dispatching function requests it, based on the submission of a job declaration file. The numbers on the chart correspond to the notes on the chart. If you're familiar with batch processing on z/os with things like JCL and JES, then this should look familiar. The concepts are the same, because the concepts are still very much applicable even though it's Java rather than compiled COBOL. Key Point: this function separation of the "job" from the "batch application," and the separation of "dispatching" from "execution," provides considerable flexibility as you'll see. 11

12 Batch Job and Batch Job Steps A batch job consists of one or more steps executed in order specified in xjcl: xjcl Job Properties of the overall job Job Step 1 Java class Input and output declarations Other properties of the step Job Step 2 Java class Input and output declarations Other properties of the step Job Step n Java class Input and output declarations Other properties of the step The xjcl is submitted through the Job Management Console Interfaces provided: HTTP browser, command Line, Web Services, RMI The Job Dispatching function interprets xjcl and determines which endpoint has batch application class files deployed Dispatching Function invokes job and passes to the endpoint an object containing all the properties in xjcl Steps are executed in order, with conditional step processing if declared Dispatching Function maintains awareness of job state When job ends, job output file accessible through Job Management Console 12 A Java batch "job" is a logical collection of one or more "steps" that are executed in the order specified in the job declaration file (xjcl). The Java code that does the batch processing is named in the xjcl as part of the step declaration. Along with the Java class file that implements the Java batch step, the xjcl may also specify job step properties, such as where to get data and where to write data to. The xjcl is passed into the Job Dispatching Function as part of job submission. The dispatching function reads the xjcl to determine what the job involves, then it signals to the batch endpoints where the Java batch application is deployed to begin execution. Note: this does not involve launching a JVM. The JVM is already active as part of the WAS runtime. This is a signal for the JVM to begin execution of a deployed class file. As part of the dispatching to the endpoint, the dispatching function passes over an object that contains all the properties that were specified in the xjcl. Those properties are available to Java batch job as it executes. For example, the properties that specify the JDBC data source to use to access DB2 to get data. The Java batch code may abstract all of that and rely on the actual data being part of the properties object passed to the batch container at time of job execution. The steps are processed in the order specified in the xjcl. And yes, conditional step processing is supported, so if the first step fails it's quite possible to have the second step not execute. During job execution the dispatching function is well aware of the state of the job. The batch endpoints are maintaining state information in a relational table that is accessible by the dispatching function. When the job ends, the job output log is accessible through the Job Management Console, where it may be viewed or downloaded. 12

13 Job Execution "State" The following picture illustrates a simplified view of the job states... it helps illustrate a key point: executing jobs can be acted upon; failed jobs restarted. Submitted Executing Problem Stop or Cancel Restartable The Job Management Console provides you ability to act upon an executing job The Batch Container is maintaining checkpoint status and will restart at the last checkpoint interval Restart Ended This is possible because of the Java batch runtime services that are part of the batch container model If you were to write this yourself then just what's shown here would require a significant amount of custom batch middleware code. IBM WebSphere Java Batch provides that as part of the product. 13 In a perfect world, a job would go through three states -- submitted, executing, and successful end. But we don't have a perfect world -- things sometimes go wrong. Or, you may wish to act upon a batch job during execution. So to that end WebSphere Java Batch provides the ability -- through the Job Management function -- to see the state of a job and to act upon it if you choose. For example, imagine a batch job that is submitted and dispatched to an endpoint. That job enters step 1 and something goes wrong. WebSphere Java Batch will put that job into a "restartable" state. You may then inspect the job log, determine what went wrong, correct that problem, and restart the job. WebSphere Java Batch will then start the job from where it left off. Note: or, more precisely, at the last good checkpoint interval. We'll talk about checkpoint processing in a bit. Aside from cases where something goes wrong, you may wish to stop, suspend or cancel a job during execution. You may do that as well through the Job Management function. The container will cease execution at the next checkpoint and put the job into a restartable state. This is an execution model considerably more sophisticated than a simple process of launching a JVM and having it run some Java "main" type program. The function we've described here -- which is only a small set of the total WebSphere Java Batch function -- would take a great deal of time and effort for you to code yourself. And if you did code it yourself you'd have a custom middleware function you'd then have to maintain and support. But thankfully you don't have to create and maintain custom middleware because IBM provides it as a supported middleware function in WebSphere Java Batch. 13

14 Batch Data Stream Framework (BDSF) This is a key function service provided by the batch container - it abstracts data read and write operations so your code may focus on the business logic: Your Java class that implements the supplied framework and provides the specific data access logic Example: SQL query for JDBC Batch Data Stream Framework Data object passed based on your mapping in BDSF class Your job step Java class, which implements the business logic required for the batch processing Supplied "patterns" for data access: JDBC read or write operations JPA read or write operations File read or write operations z/os Data Set read or write operations Batch Data Stream retrieves result set from data persistence store (DB, file, etc.) Batch Data Stream maps data fields to data object For each record in result set, BDSF invokes your job step, passing a data object mapped to your specifications Your job step code stays focused on business logic, not Java stream handling and data object formatting 14 The "Batch Data Stream Framework" (BDSF) is a significant piece of the provided services for batch programming. Its purpose is to provide a container function for data access so your Java batch logic does not have to deal with all the lower-level stream handling. The BDSF provides a set of popular "patterns" for data access -- JDBC, JPA, UNIX file and z/os data set read and write operations. You implement a BDS (batch data stream) by extending the Java class files provided with WebSphere Java Batch. This is where you provide such customization such as the SQL query strings to use to fetch the result set from the JDBC database. You name your customized instance of a BDS in the xjcl for a job step. A job step may have both an input and output function, so that job step would have two BDS declarations -- one for input, and one for output. The batch data stream will perform the read operation and then iterate through the result set. For each iteration it will map the data record to a data object you define, then invoke your batch step logic and pass in the data record object. Imagine a SQL query that returns 1000 customer records. The BDS will then take the first record, map the customer data into a data object you define, then call your batch step, which will process the record (and likely call an output BDS). The input BDS will then get control back, map the second record to a data object, and call your batch step. The process repeats until the result set is completed. This takes the data handling away from your batch processing logic. Your batch processing logic does what it needs to do (calculate tax, determine interest earned, calculate penalties, etc.), but it does not have to handle getting the records and mapping the data. That's all handled by the BDS that you implement from the supplied class file with the necessary customization to fit your environment. We bring up the BDSF because (a) it's a significant functional service of WebSphere Java Batch, and (b) many of the other services are defined and operate at the BDSF level. We'll see that coming up. 14

15 Integration with Enterprise Scheduler Functions The Job Dispatching Function has a Message Driven Bean (MDB) interface. IBM supplies a program that integrates schedulers with WebSphere Java Batch: Enterprise Scheduler Example: IBM Tivoli Workload Scheduler, CA Workload Automation CA 7, or BMC Control-M WebSphere MQ or the integrated Default Messaging of WAS Input Queue Output Queue WSGRID Program Shell script, BAT file or JCL job Message Driven Bean Interface WSGRID is seen by Scheduler as any other batch job it starts and monitors WSGRID interacts with Job Dispatching, submitting the job and processing Java batch job output back to STDOUT or JES Spool if z/os WSGRID program stays up for life of job in WebSphere Java Batch To the Scheduler, WGRID is the Java Batch job... but behind WSGRID is all the WebSphere Java Batch function we'll discuss 15 Enterprise scheduling functions, such as IBM's Tivoli Workload Scheduler, may integrate with the WebSphere Java Batch function by using a supplied program called "WSGRID." The scheduler submits the JCL (or invokes the shell script or BAT file; all forms of submission are supported) that starts the WSGRID program. WSGRID then turns and submits the Java batch job to WebSphere Batch by using the Message Driven Bean (MDB) interface of the Job Dispatching function. Key Point: WSGRID stays active for the duration of the Java batch job execution in WebSphere Java Batch. The enterprise scheduler knows nothing of the Java batch execution. It sees WSGRID. WSGRID receives back from WebSphere Java batch the output from executing job. When the Java batch job completes, that information feeds back to WSGRID, which then completes. In summary, WSGRID uses message queuing to interact with WebSphere Java Batch. A message flows from WSGRID to WebSphere Java Batch to submit the job; many messages flow back to WSGRID so the Java batch output can be spooled back to STDOUT of the WSGRID job. On z/os that means the Java batch output goes to JES spool, which means normal archiving mechanisms may archive the Java batch output. The enterprise scheduler sees WSGRID, which is like a mirror of the Java batch job executing in WebSphere. When WSGRID ends, it means the Java batch job has ended. 15

16 Feature Focus A closer look at some of the features and functions of the IBM WebSphere Java Batch model 16

17 Transactional Checkpoint Processing The batch container provides the ability to checkpoint at intervals based on either record count or time. The container keeps track of last checkpoint. Batch Container Commit Processing Java Batch Application Last good checkpoint persisted xjcl says: Checkpoint = 5 Checkpoint interval (record or time) specified in the xjcl This is a function of the batch container, not your application code As checkpoint intervals are reached, container commits and records the checkpoint attained In the event of a failure, job may be restarted at the last good checkpoint Set the checkpoint interval based on your knowledge of balance between recoverability and efficiency 17 Another service of the batch container is checkpoint processing. This allows long transactional batch jobs to have a point to fall back to in the event of an error (or you stopping the job), and it gives the container a point to restart from when the job is restarted. It's important to understand your batch step code does not do any of this. This is handled by the container based on definitions in the xjcl. In the xjcl you specify the type of checkpoint policy (record-based, or time-based), and the value to use (for example, checkpoint every 100 records). The chart shows an example where the checkpoint interval is 5. That means every 5 records the container will process commit and store the checkpoint value for the job step. If the job does fail and is restarted, then the container looks in the database table and restarts from that point forward. Setting the checkpoint interval is a balance between granular recoverability (a lower checkpoint interval) and efficiency (a higher checkpoint interval). 17

18 Skip-Record Processing Provides a container-managed way of tolerating data read or write errors so the job itself may continue on. Information about data errors may be logged. Java Batch Application Batch Data Stream Framework xjcl tells BDSF: How many data read or write exceptions to consume What exceptions to consider for skiprecord processing Alternatively, what exceptions to exclude from skip-record processing Objective: allow job to continue if a data read or write exception occurs in BDSF Why fail a million-record job just because of one or two read or write exceptions? Better to complete the job and allow auditors to go back and investigate the few exceptions. Skip-Record processing allows BDSF to keep exception and not surface it to your application This takes burden off your application code to explicitly handle data read or write exceptions that may occur A "skip-record listener" may be called so your code may lot information about skipped record More on "batch listeners" coming up xjcl properties allow you to specify how many records may be skipped and what exceptions to include or exclude from consideration When skip limit is reached, further exceptions are surfaced to application. That may result in job failing and going into a restartable state Normal restart-at-checkpoint would occur 18 Skip-record processing is a function added to Compute Grid V8 and WAS V8.5. It provides a means of defining container behavior when a read or write operation by the Batch Data Stream Framework (BDSF) experiences an exception. Without this function, exceptions thrown by the BDSF will bubble up to the batch application. If the exception is not explicitly handled, the job will go into a restartable state. The job can then be restarted, but that implies manual intervention and the loss of time. The objective of this function is to allow transaction batch jobs to survive the occasional error. With this function enabled, an exception that occurs on a BDSF read or write operation is "consumed" by the BSDF... that is, not surfaced up to the application. The job survives the exception. Properties specified in the xjcl indicate how many records may be skipped. Beyond that limit no skipping takes place... exceptions after that limit is reached are surfaced up to the application. Other properties in the xjcl allow you to specify what exceptions to include for skip-record consideration or, alternatively, which records to exclude from consideration. It's important to point out that there is a mechanism to log information about records being skipped. It's called a "skip-record listener," which is a mechanism where custom code you provide is called each time a record is skipped. Your code may then log specific information about the exception and the record skipped. Your auditors may then review the skipped records after the job completes and take appropriate action to complete the skipped records. 18

19 Retry-Step Processing Provides a means of retrying a job step in the event of an exception thrown. If successful on retry then the job continues and your processing completes. xjcl tells Container: On exception, retry up to n times How many step retries may be attempted What exceptions to consider for retry-step processing Alternatively, what exceptions to exclude from retry-step processing Whether to process a delay before attempting a retry of the step Objective: retry step in attempt to allow overall job to continue and complete when an unanticipated exception is thrown This is at level higher than skip-record... this is if an unhandled exception is thrown when the job step function is called Batch container falls back to last good checkpoint and restarts from there A "retry-step listener" may be called so you can perform custom action upon retry-step processing More on "batch listeners" coming up xjcl properties allow you to specify how many retry attempts will be performed and what exceptions to include or exclude from consideration When retry limit is reached, job will go into restartable state Normal restart-at-checkpoint would occur 19 Retry-step processing is a little like "skip-record" in that the objective is to provide the batch job some resiliency to survive exceptions that pop up during execution. But in this case the focus is exceptions that occur at a higher level -- that is, exceptions that come up during execution of the batch step itself (as opposed to exceptions on read or write operations, which occur down in the Batch Data Stream Framework code). Here again, properties in the xjcl allow you to specify how many times a batch step may be retried, and which exceptions to include or exclude from consideration. You may also code how long to delay before retrying a batch step. Imagine a batch step is executing and during one of the invocations of processjobstep() an exception is thrown. If you have a greater-than-zero retry-step property coded in the xjcl then the container will, upon exception, roll back to the most recent checkpoint and then retry the step starting from the checkpoint. As was the case with "skip-record," there is a "listener" that gets invoked with retry-step as well. This allows some code you write to be called at two points --onerror() and onretry(). Here you may log information about the exception and take whatever other actions you feel are appropriate. 19

20 Batch "Listeners" These are callout points where your customer "listener" code will be called when key events occur. The callouts are managed by the batch container: Job Listener Retry-Step Listener Skip-Record Listener Listeners provide ability to have your code called at key points during batch job execution Job Listener Callouts occur: Start of the job; Start of each step; End of each step; End of job Register your code to container with property in xjcl Use this to perform any special setup or cleanup actions at those points in the lifecycle of a batch job Retry-Step Listener Callouts occur: When the exception is thrown; When the retry is attempted Register your listener with code in application createjobstep() method Use this to take action at these points, such as logging information about the exception and the point in the processing where it occurred Skip-Record Listener Callouts occur: On skipped read or skipped write operation Register your listener with code in application createjobstep() method Use this to take action at these points, such as logging information about the exception and the record skipped 20 Several times now we've mentioned "listeners" that get invoked at certain points of batch processing. Listeners are implemented with code you write which you "register" to the batch environment so it knows what to call when those call-out events take place. There are three listeners we'll focus on -- Job listeners, Retry-Step listeners, and Skip-Record listeners. Job listeners -- this is invoked at the start of the job, and the start and end of each batch step, and at the end of the job. You register your listener class file by coding a property in the xjcl. At those points during batch job execution the container will call the methods for that lifecycle point. Your code then does what you determined was appropriate for that point in the job lifecycle... setup work, cleanup work, logging... whatever. Retry-Step listeners -- this is invoked whenever a job step is retried using the new Retry- Step processing function. The listener is called when an exception is thrown that triggers Retry-Step processing, and again just as the step is being retried. You register your listener with a simple line of code in the batch application itself. Skip-Record listeners -- this is invoked whenever a record is skipped using the new Skip- Record processing function. The listener is called when an exception is thrown that triggers Skip-Record processing. Your code may then log information about the record that has been skipped and the exception that took place. You register your listener with a simple line of code in the batch application itself. Listeners provide you a way to insert custom processing logic into the batch job lifecycle at key points during execution. 20

21 Parallel Job Manager The Parallel Job Manager (PJM) provides a way to "parameterize" logic so parallel sub-jobs may act on a slice of the overall batch job data: Time = 0 Time = 1 Time = 10 One job processing 1M customer records or Sub-job Sub-job Sub-job 1-100K 100K - 200K 900K - 1M Ten sub-jobs acting on a 1/10th slice of data each Objective is reduction in overall job completion time Which shortens overall batch window if other jobs are dependent on this job for completion xjcl specifies whether job is to be run in parallel, and if so how: One JVM, multiple threads Multiple JVMs Your "parameterizer" code is called at start so data range may be segmented into sub-job slices Job is submitted, then PJM dispatches "sub-jobs" to act on each data range "Parameterizer" code constructs data range query strings to be used by each sub-job PJM manages "top-job" and all subordinate "sub-jobs" to completion 21 The Parallel Job Manager (PJM) function provides a way to run batch jobs as parallel sub-jobs. If the data lends itself to parallel sub-jobs... meaning, the data can be divided in such a way the sub-jobs will not encounter access contention or locking issues... then running the jobs as parallel sub-jobs can shorten the total execution time of the overall job. The first part of this function is a property in the xjcl that determines if the job is to be run in parallel, and if so, then how many sub-jobs should be run, and if they should be run on separate threads in a single JVM or across multiple JVMs. The second part of this function is a call the container does to code registered in the xjcl as the "parameterizer." This is code you write that determines how the data is to be split up for each sub-job. For example, if the data involves customer numbers, then one way to split the data is by customer number ranges. The parameterizer then returns control back to the container, which then creates the specified sub-jobs and dispatches them. Each sub-job will operate on the data range determined by your parameterizer code. The PJM watches over each of the sub-jobs and collects up the results to form the return code of the overall job. If a sub-job goes into a restartable state, the overall job is restartable. You then restart the top-job, and the PJM manages the restart of the sub-job that was in a restartable state. 21

22 Java Batch on z/os A review of what IBM WebSphere Java Batch brings specific to z/os 22

23 The Value Statements of WebSphere Batch on z/os If we start from a high level, we see the following platform benefits that accrue up to Java batch running on the platform: Java Batch Applications WebSphere Java Batch WebSphere Application Server z/os Operating System z/os Parallel Sysplex System z Hardware Batch runtime services Batch development tooling Proven Java runtime environment WAS deployment and management model WAS Qualities of Service Decades of maturity, stability and reliability Consolidated operation and management model Rich set of system facilities: WLM, SMF, RMF, SAF z/os instance clustering with central data sharing Elimination of single points of failure for availability Near linear scalability up to 32 nodes Engineered from beginning for reliability and stability Engineered for high levels of I/O Extremely long mean time between failure Speciality engines for specific work offload Dynamic capacity expansion Logical partitioning using PR/SM hypervisor 23 As we turn to look at the specific ways in which WebSphere Java Batch takes advantage of the z/os platform, it's useful to step back and look at this from a broad perspective. The chart shows the System z and z/os environment as a layered architecture, starting from the hardware at the bottom working up through to the Java batch application on top. The bullets to the right correspond by color to the layer of the architecture in the illustration. Start at the bottom and work towards the top: Hardware -- the System z hardware has a long history of reliability. It is designed to have redundant and resilient components to provide for availability. It is also designed to handle high levels of data input/output, which is often the case with transactional batch processing. Parallel Sysplex -- this is a z/os operating system clustering mechanism that provides a shared data facility for the members of the cluster. The shared data mechanism, provided by the Coupling Facility, has a near-linear scalability up to the maximum 32 cluster members supported. z/os Operating System -- z/os, like the hardware on which it runs, has a long history of reliability. Its design permits many different processes to run simultaneously Further, it has a very rich set of management facilities built up around it. It is an extremely robust and reliable operating system platform. WebSphere Application Server -- this provides the Java runtime foundation for WebSphere Java batch. WAS provides the underlying services key to Java batch -- data access support such as JDBC, transactional support, and clustering. WebSphere Java Batch -- this is the focus of this presentation... this is what provides the runtime environment for Java batch programs. You Java batch applications run on top this layered stack of operational benefits. That's the picture in total. 23

24 Scaling Up the Java Batch Solution on z/os There are several ways in which a WebSphere Java batch solution can be scaled up to provider greater batch throughput and shorter execution windows: Controller Region 2 1 Servant Region Servant Region Sysplex-Enabled Data Subsystems DB2, CICS, IMS, MQ Parallel Sysplex Data Sharing WebSphere extreme Scale Caching Grid 5 PJM Controller Region 3 4 Servant Region Servant Region Sysplex-Enabled Data Subsystems DB2, CICS, IMS, MQ 1. Vertical WAS z/os servant regions provide a type of "vertical cluster," giving you additional batch compute resources 2. Capacity on Demand CPU processors may be dynamically added to a z/os LPAR, increasing the capacity for processing work 3. Horizontal WAS z/os clustering on top Parallel Sysplex provides near-linear scalability up to 32 nodes with a central data sharing model 4. Parallel Processing The Parallel Job Manager may be used to partition data into sub-jobs, which may then be run on multiple threads, different servants, or different servers on other LPARs. 5. Data Caching WebSphere extreme Scale provides a data caching grid from which Java batch may fetch and store data 24 WebSphere Java Batch applications running on System z and z/os have several means by which you may scale up the processing. The chart above illustrates several of the approaches: 1. With WAS z/os the ability to create "vertical clusters" of servant regions. This provides additional JVMs in which to run Java batch, which implies additional worker threads on which to dispatch batch processes. 2. The System z hardware and z/os operating system architecture allows CPU resources to be dynamically allocated to a logical partition, providing additional "capacity on demand" if needed. 3. Horizontal clustering for availability and scalability is provided by the WAS z/os foundation on which Java Batch is built. WAS z/os clustering may then take advantage of Sysplexenabled middleware functions such as DB2 or CICS. The shared data model of Parallel Sysplex means the data accessed by Java batch is the same on any logical partition. 4. The Parallel Job Manager function discussed earlier may take advantage of vertical and/or vertical clusters. 5. Java batch applications that have intense data requirements might benefit further from taking advantage of IBM's WebSphere extreme Scale, which is a distributed in-memory caching grid architecture. There are many paths to scalability provided by System z and z/os. 24

25 WLM Classification The submitted job can be tagged with a WLM "transaction class," which may then be used to map the job to a WLM Service Class or Reporting Class: Configurable rules map job submission to a "Transaction Class" (TC) name TC name sent to endpoint where batch job will run xjcl Job Dispatching Function Job Execution Endpoint Batch Applications z/os WLM WLM "CB" subsystem rules map TC name to Service Class and Reporting Class Batch job runs under that Service Class and data is gathered under the Reporting Class Classifying to a Service Class allows WAS z/os to place work into separate servant regions based on Service Class A somewhat sophisticated practice not widely used Classifying to a Reporting Class allows WLM to gather system information for all work running under that Class A much more common practice that is very useful for understanding usage patterns and for capacity planning 25 One of the key points of integration with the z/os operating system by WebSphere Application Server is with WLM. WAS z/os is able to classify work to WLM Service Classes and Reporting Classes. Classifying to specific Service Classes allows WLM to better manage work of differing goals; classifying to specific Reporting Classes allows RMF to report on work more granularly. The WebSphere Java Batch function on z/os has the same ability. Configurable rules in the Job Dispatching Function allow submitted batch jobs to be classified to individual Service Classes or Reporting Classes. With multiple Service Classes it becomes possible to have batch jobs of one service class goal be dispatched to one servant, and batch jobs of a different service class goal be dispatched to another. WLM then may manage system resources to the z/os address space (the servant) to help meet the assigned goals. Being able to classify batch jobs to different WLM Reporting Classes allows you to collect system resource usage information by batch job, even though multiple batch jobs may be running in the same server. 25

26 SMF Activity Recording WAS z/os supports the use of activity recording using the SMF record. WebSphere Java Batch extends the record with batch activity information: WebSphere Java Batch Compute Grid z/os V8 WAS z/os V8.5 SMF Buffers and Data Sets Job activity records allow you to understand how your system is being used and to provide chargeback data Activity recording available on all platforms, but only z/os uses SMF, which is an extremely efficient logging mechanism Provides historical records for usage analysis and batch capacity planning Information captured: Job submitter Date and time of submission Final job state Total CPU used for job General processor used for job zaap usage derived: Total - GP = zaap 26 For even more granular information about batch job resource usage you can turn on SMF recording of batch activity. This is an extension of the WAS z/os SMF record to include batch activity information. 26

27 Use of JZOS Services JZOS is a set of functions that make using Java on z/os much easier and useful. The JZOS class libraries may be used in batch application development: Job Execution Endpoint Batch Applications JZOS Libraries z/os JZOS is technology acquired by IBM from Dovetail Technologies* and incorporated into z/os** Examples of some z/os services available: DfSort - interface for invoking DFSORT MvsConsole - class with static methods to interface with the MVS console. MvsJobSubmitter - class for submitting batch jobs to JES2 or JES3 from a Java program PdsDirectory - class for opening a PDS directory and iterating over its members. WtoMessage - simple data object/bean for holding a WTO message and its parameters. ZUtil - static interface to various z/os native library calls other than I/O. WebSphere Java Batch and JZOS are not mutually exclusive... the JZOS class libraries may provide exactly what you need for your batch application to access z/os functions and services * ** 27 JZOS is a set of technology developed by Dovetail Technologies that makes running Java on z/os must more useful. JZOS provides a set of Java interfaces that give your Java program access to z/os services such as what's listed on the chart. Note: JZOS also has a JVM-launcher function to run Java "main" programs in a "batch" mode. That function is not compatible with WebSphere Java Batch because the JVMs in which batch runs with WebSphere Java Batch are always-up JVMs. No need to launch a JVM. What we're talking about here are Java class libraries with accompanying native libraries that provide access to z/os services. To use the JZOS Java class libraries you import them into your application and STEPLIB the batch server to the native libraries. 27

28 COBOL Container The COBOL Container provides a way to call and execute COBOL modules in the WAS z/os server address space... a very efficient way to call COBOL WebSphere Java Batch Container 5 IBM Rational Application Developer 1 Call Stub Generator 7 Call Stubs 6 4 COBOL Container Call Code 8 COBOL Container LE Enclave 2 COBOL Module WAS z/os Servant Region Address Space Separate LE Enclave from COBOL Container Compiled COBOL Library PDSE or USS Directory 3 1. Batch application runs in the WAS z/os servant region address space 2. The COBOL container is created as a separate LE enclave in the address space 3. COBOL DLLs are accessed using STEPLIB or LIBPATH 4. COBOL Container code provides the "glue" between the Java environment and the native COBOL 5. Java batch code uses supplied class methods to create the container and use it 6. Call stubs provide an easy way to call the COBOL DLL and marshal data back and forth 7. The call stubs are generated by a supplied utility that uses COBOL source to understand data bindings 8. JDBC Type 2 connections created in the Java batch program may be shared into the COBOL module in the COBOL Container Lines of code needed to invoke COBOL many times less than other means of calling COBOL from Java 28 The COBOL Container function provides a way to run compiled COBOL DLLs inside the WAS z/os servant address space where the Java batch application runs. This makes calling the COBOL module much more efficient than other ways to call COBOL from WAS. The COBOL Container is code written by IBM that provides the Java-to-native bridge. It creates a separate Language Environment (LE) enclave for the COBOL code. That insures the COBOL enclave does not "dirty" the WAS enclave, and vice-versa. In addition, a code-generation tool called the "Call Stub Generator" makes calling the COBOL DLL and passing and getting data much easier. The Call Stub Generator operates in IBM Rational Application Developer 7.5 or later. It reads the COBOL source to understand the data input and output requirements, and it generates the Java call stub and Java data bindings automatically. Your batch application developer then uses the generated call stubs rather than having to do all that coding themselves. This also provides the ability to create a JDBC Type 2 connection in the Java batch application and share that connection with COBOL running in the COBOL container. Key Point: the COBOL Container provides a highly-efficient way to use COBOL assets as part of the Java batch processing in the WebSphere Java Batch environment. 28

29 Deployment Scenarios A review of some potential ways to deploy the WebSphere Java Batch function 29

30 Co-Location on z/os With the WebSphere Java Batch function on z/os several advantages surface: Job Execution Endpoint Batch Applications Data SubSystems DB2, CICS, IMS, MQ z/os Use of cross-memory connectors for high-speed and low-latency access to data JDBC Type 2 connector for access to DB2 CICS Transaction Gateway (CTG) local EXCI WebSphere Optimized Local Adapters (WOLA) Much more secure -- cross memory data exchanges can not be 'sniffed' or intercepted Parallel Sysplex data sharing provides highly available clustered environment without reliance on a single instance of a data subsystem Use of COBOL Container technology for re-use of COBOL assets in very efficient calling pattern Use of WebSphere MQ Bindings Mode for integration with Enterprise Scheduler for very fast job submission and job output return Reduction of per-access latency is critical when dealing with large volumes of records where job completion time is important 30 One deployment scenario is to have WebSphere Java Batch deployed on z/os in the same LPAR as the data. WebSphere Java Batch on z/os provides several advantages based on the WAS z/os exploitation of z/os. It also provides an opportunity for reduced latency per transactional record which can help reduce the overall batch job execution time. 30

31 Linux for System z and Hipersocket Access to z/os Data Hipersockets is a technology that maps a TCP/IP network onto the memory backplane of a System z divided into multiple logical partitions (LPAR): z/os LPAR Data SubSystems DB2, CICS, IMS, MQ Linux for System z Job Execution Endpoint Batch Applications To programs and processes that use Hipersockets it looks like a routed TCP/IP network TCP/IP Network Memory Backplane of System z CEC Advantages of Hipersockets Secure -- does not go over adapters or external wires Efficient -- memory transfer speeds implies lower overall latency Advantages of Linux for System z Consolidation -- host many Linux images in a virtualized environment Virtualizing on the zvm hipervisor provides a means of quickly scaling up in Linux instances to meet requirements 31 Locating the WebSphere Java Batch function on Linux for System z still allows very good access to z/os data across Hipersockets, which is a TCP network mapped to the physical memory of the System z machine. Network traffic does not go out over network adapters or wires, so there is an inherent security about it. Because it is mapped to memory it is very efficient. Linux for System z is attractive because it provides a means of consolidating many Linux images from distributed servers onto zvm. The consolidation is often attractive from a financial perspective, and management of the Linux images may be done under the umbrella of a single platform. 31

32 zenterprise and zbx The zenterprise system is designed around principle of right-fit placement: z/os Job Execution Endpoint Batch Applications zbx Virtual Servers System z LPAR serves as the anchor for a zenterprise "node" A zbx blade extension rack hosts IBM p or IBM x blades capable of hosting AIX, Linux or Windows virtual servers Data SubSystems DB2, CICS, IMS, MQ 10Gb Intra-Ensemble Data Network A 10Gb network connects it all WebSphere Java Batch endpoints may be placed where the work they do makes best sense: Batch processes requiring a highly available and highly secure environment may operate on z/os Batch processes that use relatively more CPU may be offloaded to zbx blade servers WebSphere Java Batch Dispatching function would be able to "see" all the different endpoints and dispatch based on where batch applications were deployed 32 IBM zenterprise and the BladeCenter Extension (zbx) provide a means of locating WebSphere Java Batch onto a multi-os platform with a dedicated data network. The 10Gb network that connects the virtual servers of a zenterprise has routers or intermediate hops. Batch endpoints executing on servers in the zbx may access data on z/os across this network. 32

33 Wrap-Up and Summary 33

34 WebSphere Java Batch WebSphere Application Server v8.5 integrates capabilities from WebSphere Compute Grid and delivers a complete enterprise level Java batch processing solution Key Features: Java Batch programming model Java Batch container built on WAS QoS Development and deployment tooling Batch execution environment Concurrent OLTP and batch workloads Enterprise scheduler integration Parallel processing of batch jobs Container based checkpoint and restart Mixed batch workloads COBOL support on z/os Compute Grid capabilities integrated into WAS The first of three summary charts. 34