TPF Lab: z/tpf Development Environment Tools and SCMs

z/tpf EE V1.1 z/tpfdf V1.1 TPF Toolkit for WebSphere Studio V3 TPF Operations Server V1.2 IBM Software Group TPF Users Group Fall 2006 TPF Lab: z/tpf Development Environment Tools and SCMs Name : Brian Laferriere Venue : Hot Topic AIM Enterprise Platform Software IBM z/transaction Processing Facility Enterprise Edition 1.1.0 Any references to future plans are for planning purposes only. IBM reserves the right to change those plans at its discretion. Any reliance on such a disclosure is solely at your own risk. IBM makes no commitment to provide additional information in the future. TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 1

Topics Overview Components Workstation Linux for zseries z/os TPF SCM Versioning APAR Development Process TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 2

Overview TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 3

to a working directory on Linux IBM Software Group Lab Requirements We support system code development The closest thing we have to an application is a test driver - but even those follow the system code development model! We require concurrent development We do not have "file" ownership. Two or more developers must be able to have the same source file out on change at the same time. We require file-based versioning to support our APAR delivery model We support three system configurations Base only (BAS) Basic Subsystem (BSS) Subsystem (WP) We shadow (and serve) two levels of production code on Linux TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 4

Supported Production Code Levels CURRENT Code baseline changes with each APAR /ztpf/cur COMMITTED Code baseline changes once a month, rolling in all APARs since the previous baseline /ztpf/commit APARs applied CURRENT COMMITTED 05/06 06/06 07/06 08/06 09/06 10/06 11/06 Baseline change TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 5

to a working directory on Linux Components IBM Software Group DASD z/tpf ftp/xmit SMB NFS Network Client/Server z/os Linux for zseries RETAIN Lotus Notes APAR DB SCM z/tpf website TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 6

High-Level System Information z/tpf z/tpf Native VPARS on VM TPFMVSXA Native MVS z/os Unix z/os Linux for zseries Linuxtpf Guest on VM System z9 EC 2094-S28 S28 3 book machine 30 LPARs 8 CPs 384 GB SCM Workstation Windows XP Server AIX TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 7

Connectivity IBM Software Group SMB mounts and/or RSE provide r/w access to files on Linux and MVS from the workstation NFS mounts provide r/o access to files on Linux from MVS NFS respects "group" level access control z/os Unix Linux for zseries Mounts: SMB / RSE NFS text NFS binary Client/Server: SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 8

MakeTPF Build Tools Are Product Code The MakeTPF Build Tools MUST be maintained as part of the z/tpf source code. DO NOT separate the tpftools/ directory and try to use one copy of the MakeTPF Build Tools with different release levels (shadows) of z/tpf source code. TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 9

Components TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 10

Workstation - Tasks Edit Debug Remote assemble, compile, link Code comparison and merge Access the hfs shadow of the production source code levels Access host system Manage defects Interface with SCM (command line/gui) Manage APARs TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 11

Workstation - Tools Edit, debug, and remote assemble, compile, link TPF Toolkit for WebSphere Studio Access source code stored on Linux SMB RSE (for the TPF Toolkit) Manage defects and interface with SCM IBM internal use only SCM client Manage APARs Lab developed Lotus Notes APAR DB Access host systems Putty, Telnet, PCOMM Code comparison/merge TPF Toolkit compare Beyond Compare TPF Merge Tool TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 12

Linux for zseries - Tasks Generate z/tpf online and Linux offline binaries Perform development (old, tld) loads Serve shadows of the production source code Serve user home (/home) directories Linux for zseries Build and package APAR and PUT deliveries Manage APARs Edit Search, scan, compare Interface with SCM (command line) TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 13

Linux for zseries - Tools Generate z/tpf online and Linux offline binaries and perform development (old, tld) loads MakeTPF Build Tools (maketpf, bldtpf, loadtpf) Serve shadows of the production source code and user home directories NFS, both binary and text mounts, used to serve mounts to z/os (respects GROUP authorities) /ztpf/cur /ztpf/linux/cur, /ztpf/linuxbin/cur /home /home/linux, /home/linuxbin SMB, used to serve mounts to the workstation RSE server, used by TPF Toolkit Build and package APAR and PUT deliveries, manage APARs Lab developed APAR automation tools Edit vi(m), pico, (x)emacs Search, scan, compare find, grep, diff Interface with SCM (command line) IBM internal use only SCM client TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 14

Linux for zseries - Current Software Levels Require Software Current Level Linux for zseries SuSE SLES 9* Native GCC gcc version 3.3.3 (SuSE Linux) flex version 2.5.4 bison version 1.875** GCC TPF-cross compiler gcc version 4.0-tpf-05r1-3 IBM HLASM for Linux (PRPQ 5799-TCQ) (PTF 08Jan06) IBM DB2 Universal Database for Linux DB2 SDK 8.2.3 GNU make GNU Make 3.80 pdksh PD KSH v5.2.14 * The Linux distribution used for our system is managed by our IGS team. It must be a 64-bit Linux for zseries distribution, run in 32-bit compatibility mode. TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 15

Linux for zseries - Getting Started With Tooling Basic Unix shell commands File/directory management: ls, mkdir, rmdir, rm, cp, mv, chmod, chgrp File locate/scan: find, type, locate, which, grep File content display/modify: read, sed, awk, cat, echo, print, less, more Flow control: for, while, until Scripting/automation languages Korn Shell, Bash Shell, Perl Editors pico (or nano), vi(m), (x)emacs Links: Basic scripting: http://www.tldp.org/howto/bash-prog-intro-howto.html Advanced scripting: http://www.tldp.org/ldp/abs/html/index.html awk: http://www.vectorsite.net/tsawk.html sed: http://www.grymoire.com/unix/sed.html GNU Make: http://www.gnu.org/software/make/manual/make.html TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 16

z/os - Tasks IBM Software Group Generate z/os offline binaries Perform system and development (old, tld, ald) loads Serve user home (/u) directories Edit Search, scan, and compare Interface with the SCM (command line) TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 17

z/os - Current Software Levels Require Software Current Level IBM z/os (with z/os Unix) V1R6 HLASM Release 5, PTF UK10754 IBM Enterprise PL/I V3R3 IBM DB2 DB2V7 IBM Sub-Capacity Reporting Tool V12.1 GNU Make 3.79.1 Built for s390-ibm-os390 TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 18

z/os - Tools IBM Software Group Generate z/os offline binaries and perform system and development (old, tld, ald) loads MakeTPF Build Tools (maketpf, bldtpf, loadtpf) Serve user home directories SMB, used to serve mounts to the workstation RSE server, used by TPF Toolkit Edit vi(m), oedit Search, scan, and compare find, grep, diff Interface with the SCM (command line) IBM internal use only SCM client TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 19

z/os - Getting Started With Tooling Basic Unix shell commands Same as those listed for Linux for zseries Scripting/automation languages z/os Unix Shell (compatible with Korn Shell), rexx Tools submit tsocmd Links: IBM Tools and toys for z/os UNIX System Services http://www-03.ibm.com/servers/eserver/zseries/zos/unix/bpxa1toy.html TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 20

z/tpf - System Images Used In The Lab CUR Production image, refreshed with each APAR Each APAR is built and loaded independently throughout each PUT Full system build prior to the first APAR of a PUT COMMIT Production image, refreshed monthly, adding all APARs released since the last refresh Full system build at each refresh SVT (system verification test) Pre-production code Based on the COMMIT system level, updated with project SVT level code Varies between a full system build (all binaries built) or an SVT code only build TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 21

Source Control Manager (SCM) - Tasks Defect management (create, assign, update, close) Branch/workarea management (create, integrate, baseline) File management (create, check out, check in, extract, undo, delete) Collision management (identify, resolve) <- Specific to concurrent development TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 22

SCM - Requirements Requirements The SCM must be capable of supporting files with: Long file names (> 8 characters) File extensions (.asm,.c) Pathnames (base/rt/cl11.asm) The SCM must have a Windows command-line interface for integration with the IBM TPF Toolkit for WebSphere Studio. TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 23

SCM - Tools IBM Software Group IBM Internal Use Only SCM Managed and maintained by another development lab in Poughkeepsie Tasks performed through the GUI or command line Additional automation scripts written using command line interface TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 24

SCM - Available SCMs ClearCase http://www-306.ibm.com/software/awdtools/clearcase ClearCase LT http://www-306.ibm.com/software/awdtools/clearcase/cclt/index.html Concurrent Version System (CVS) http://www.nongnu.org/cvs GIT http://git.or.cz Subversion http://subversion.tigris.org Many others... TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 25

SCM - Selection While IBM can certainly help you select an SCM, the Lab cannot tell you what SCM to use: Most importantly, your selection must be based on your unique development environment, system requirements, and release process. In addition, the Lab: Does not have experience with the other SCMs listed. Does not have the resources to evaluate the other SCMs. To obtain assistance from IBM in your SCM selection, contact your CSR. TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 26

SCM - Selection: Questions To Consider Some (of the many) questions to consider when evaluating an SCM: What SCM interfaces will you need: GUI? Command line on Windows? Linux? z/os? Remember Windows command line is needed for integration with the TPF Toolkit and consider that if you do integrate with the Toolkit, you may not need a separate GUI. What development mode(s) do you need: serial? concurrent? Do you have file ownership? Do you need to allow more than one developer to be able to update a file at the same time? How many release levels of the source do you manage/shadow at the same time? Can they all be managed in one "instance" of an SCM through branching, levels, or some other means, or will you need multiple instances of the SCM? Do you want to keep z/tpf source and your application source in the same SCM? What file access controls does the SCM provide? Do you need to control who can see certain files in the SCM? How well does the SCM work for remote users? TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 27

SCM - Selection: Questions To Consider (continued) What SCM versioning model do you need? File-based: Each file has a version that only changes when the file itself is changed. Repository-based: The version of the entire repository changes when any file is changed. What is your source code update model? Do you update by file? By service pack? How are the source files organized? Is each application in its own hfs? Are applications merged in the same hfs? How many common (cross-application) files do you have? How many files are in any one directory? Note: We have received feedback that the z/tpf directory structure is less than optimal for repository based versioning systems (like Subversion), because "checkouts" are performed at the directory level, not on individual files. So for directories like base/rt that contain 3000+ files, this is a consideration. TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 28

Versioning TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 29

Source File Versioning Versions of source files are managed by the SCM Every "APAR" level is retrievable. File names do not contain version codes If file names contained versions, each version change would also require a corresponding change in the makefile for the program that contains the source file. The file name includes both the relative pathname and the base name of the file, for example: base/rt/cl11.asm Managing different versions of a source file on Linux requires each version to be extracted under a different root directory Root directories can be set up to shadow each desired baseline. For example: /ztpf/ga /ztpf/put01 /ztpf/put02 The shadows can be complete shadows of all of the source code, or the shadow can be deltas that only contain the source files changed in that baseline. MakeTPF supports layering root directories to define a search hierarchy. TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 30

Binary File Versioning Shared Object Files (TPF executables) All shared objects contain a zero-byte object that defines the APAR level and link date and time: Named: bld-pjxxxxx.o Contains a symbol named " TPF_TOD" set to: YYYYMMDDHHMMSS Automatically added by the maketpf link rule; it is not specified in the makefile. The PJxxxxx is defined by the "TPF_APARNUM" environment variable. By convention: PJxxxxx for individual APAR build. PUTxx for full PUT level builds. Object Files All object files are modified so that the ".tpfzdmap" section contains the value of TPF_APARNUM. Querying the version The bld-pjxxxxx object in each shared object can be displayed online using ZDMAP The.tpfzdmap information for each object within the shared object can be displayed online using the INFO-LPATH option of ZDMAP The.tpfzdmap information can also be displayed on Linux using a combination of the readelf and objdump utility TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 31

APAR Development Process TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 32

to a working directory on Linux IBM Software Group APAR Development - CSR 1. Send APAR report from RETAIN to the APAR DB, which automatically creates a defect in the SCM. RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 33

to a working directory on Linux IBM Software Group APAR Development - Developer 2. Accept the APAR in the APAR DB 3. Accept the defect in the SCM RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 34

to a working directory on Linux IBM Software Group APAR Development - Developer 4. Create a defect workarea in the SCM RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 35

to a working directory on Linux IBM Software Group APAR Development - Developer 5. Check-out the "current" version of the code on-change to a working directory on Linux. Linux for zseries /home/lafer/pj12345 RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 36

to a working directory on Linux IBM Software Group APAR Development - Developer 6. Update code (Toolkit editor or vi). 7. Build code (Toolkit remote build or maketpf) against the shadow of the production-level "current" filesystem. Linux for zseries /home/lafer/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 37

to a working directory on Linux IBM Software Group APAR Development - Developer 8. Load the code to the "current" image on the z/tpf test system (Toolkit remote load action or loadtpf). 9. Test. 10. Repeat steps 5-9 until the APAR coding changes are complete. z/tpf Linux for zseries /home/lafer/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 38

to a working directory on Linux IBM Software Group APAR Development - Developer 11. Check-in the updated version of the code. 12. Run an audit to detect problems and identify "collisions". 13. Resolve collisions and repeat steps 5-12, as needed. z/tpf Linux for zseries /home/lafer/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM Note: "collisions" occur when the baseline for a file in the APAR has changed since the file was checked out. TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 39

to a working directory on Linux IBM Software Group APAR Development - Developer 14. Generate the APEDIT documentation (including build instructions) and mark the APAR ready for lead-review in the APAR DB. z/tpf Linux for zseries /home/lafer/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 40

to a working directory on Linux IBM Software Group APAR Development - Lead Reviewer 15. Perform final review of the code. 16. Mark the APAR ready for build in the APAR DB. z/tpf Linux for zseries /home/lafer/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 41

to a working directory on Linux IBM Software Group APAR Development - Build Team 17. Perform a final check for "collisions" against other APARs that have closed between the time the APAR was sent for lead review and the time it was marked ready for build. If a collision is found, return the APAR to the developer at step 5. z/tpf Linux for zseries /ztpf/aparbld/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 42

to a working directory on Linux IBM Software Group APAR Development - Build Team 18. Extract the code to an APAR build directory. 19. Extract the maketpf build commands from the APEDIT content in the APAR DB to a shell script on Linux. 20. Run the maketpf commands to build the APAR on Linux. z/tpf Linux for zseries /ztpf/aparbld/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 43

to a working directory on Linux IBM Software Group APAR Development - Build Team 21. Load the executables to the "current" image of the system. DASD z/tpf z/os MVS Linux for zseries /ztpf/aparbld/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 44

to a working directory on Linux IBM Software Group APAR Development - Build Team 22. Create a new baseline in the SCM containing the updates. 23. Promote the updates to the "current" hfs shadow. MVS z/os DASD z/tpf Linux for zseries /ztpf/aparbld/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 45

to a working directory on Linux IBM Software Group APAR Development - Build Team 24. Generate an APAR package and post to the z/tpf maintenance website. z/os DASD z/tpf Linux for zseries /ztpf/aparbld/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM z/tpf website TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 46

to a working directory on Linux IBM Software Group APAR Development - Build Team 25. Update RETAIN with the APEDIT and close the APAR. 26. Close the APAR in the APAR DB. z/os DASD z/tpf Linux for zseries /ztpf/aparbld/pj12345 /ztpf/cur RETAIN Lotus Notes APAR DB SCM z/tpf website TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 47

to a working directory on Linux IBM Software Group APAR Development DASD z/tpf ftp/xmit SMB NFS Network Client/Server z/os Linux for zseries RETAIN Lotus Notes APAR DB SCM z/tpf website TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 48

Trademarks IBM, AIX, ClearCase, DB2, DB2 Universal Database, Lotus Notes, MVS, OS/390, RETAIN, S/390, WebSphere, zseries, z/os, z/vm, and z9 are trademarks of International Business Machines Corporation in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Notes Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-ibm products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography. This presentation and the claims outlined in it were reviewed for compliance with US law. Adaptations of these claims for use in other geographies must be reviewed by the local country counsel for compliance with local laws. TPFUG Oct 2006 HOTT DevEnv 10.PRZ 10/30/06 Pages 49