WebSphere MQ File Transfer Edition V7.0.2 PoT

Transcription

1 WebSphere MQ File Transfer Edition V7.0.2 PoT Product Introduction An IBM Proof of Technology

2 Does this sound familiar? Many approaches to file transfer File Transfer Protocol (FTP) Home grown Other (old) file transfer products Why is FTP use widespread? Lowest common denominator FTP is free, simple, intuitive and ubiquitous Quick fix repent at leisure Simple concepts Low technical skills to get started IBM WebSphere MQ File Transfer Edition - Product Introduction 2

3 Why use WebSphere MQ File Transfer Edition? You want to Management, control and governance Enterprise logging and management Service oriented architecture (SOA) need to move files Automate file movement Reduce complexity and cost Avoid human-errors Avoid home grown code Reuse existing infrastructure for files Reliable Secure Managed IBM Solution WebSphere MQ File Transfer Edition WebSphere MQ File Transfer Edition V7.0 IBM WebSphere MQ File Transfer Edition - Product Introduction 3

4 WebSphere MQ File Transfer Edition reduces Integration Costs You want to Reduce integration costs by a factor of 2 to 4 Cost Handwritten Code All Custom Development Basic/Free FTP Technology + Custom Development Handwritten extensions to FTP Value Added FTP Solutions + Custom Development Enterprise Application Middleware IBM connectivity software Size of Project IBM Solution WebSphere MQ File Transfer Edition V7.0 Single infrastructure to Manage Messaging traffic include file transfers Both operations and Users Common Log Enterprise wide Audit Trail More efficient use of existing resources IBM WebSphere MQ File Transfer Edition - Product Introduction 4

5 Centralized Monitoring and Management You want to Meet regulatory compliance SOX, Basel-II, HIPAA, etc Monitor all file transfers Server to Server Desktop Users Trading Partners / Vendors or other outside companies Security of file transfers Platforms Source file locations Target file locations File names or file patterns IBM Solution WebSphere MQ File Transfer Edition V7.0 provides Full Logging Full Transfer History Comprehensive access control list (ACL)- based security?????????? IBM WebSphere MQ File Transfer Edition - Product Introduction 5

6 File transfer Reliability, Integrity and Security You want Eliminate FTP pitfalls Assured delivery of files High performance Integrity and security Server to Server Desktop Users Trading Partners / Vendors or other outside companies IBM Solution WebSphere MQ File Transfer Edition V7.0 provides Secure communications Optionally PGP (Pretty Good Privacy) encryption Checkpoint / Restart Automatic network recovery Assured delivery IBM WebSphere MQ File Transfer Edition - Product Introduction 6

7 Pitfalls of unmanaged FTP Limited Reliability Lacks Checkpoint/Restart Transfers fail without notification Partial can cause data integrity problems Code Page Conversion often trivialized or incomplete ASCII ASCII ASCII EBCDIC EBCDIC EBCDIC Limited Flexibility All resources usually have to be available concurrently Each transfer requires a dedicated session (one at a time) No support for Priority Compression Encryption Limited security Lack of user/password protection (often in the clear) Lack of support for Nonrepudiation Lack of support for Authentication Encryption Limited visibility and traceability No centralized Management Logging capabilities are limited and may only record transfers between directly connected systems Lack of logging from source to target IBM WebSphere MQ File Transfer Edition - Product Introduction 7

8 Automation with WebSphere MQ File Transfer Edition reduces costs You want to Automate simple or complex File Transfers steps using a standards approach Integrate User Applications Operating System (OS) Commands Before or after transfers Event driven processing Specific time of day or week Interval Operational events IBM Solution WebSphere MQ File Transfer Edition V7.0 provides Integration with existing Schedulers IBM z/os or distributed platforms Integrated scheduling for repetitive tasks Integration with operations Comprehensive Ant based scripting Conditional step processing Transfer History includes step execution IBM WebSphere MQ File Transfer Edition - Product Introduction 8

9 WebSphere MQ File Transfer Edition Reduces Operational Complexity You want to Avoid parallel infrastructures Messaging Files Reduce administration burden Faster/Easier problem resolution Improve operational efficiency IBM Solution WebSphere MQ File Transfer Edition V7.0 provides the following qualities of service: Reuse messaging infrastructure Manage with a single tool Monitor with a single tool Route in a consistent manner Eliminate multiple skills Reduce personnel requirements File Transfers + = Consolidated Application Messaging Transport for messages & files IBM WebSphere MQ File Transfer Edition - Product Introduction 9

10 IBM s Vision SOA Messaging Backbone Addressing full spectrum of universal transport requirements Skills End-Points Qualities-of-Service Delivery Styles Languages COBOL, C/C++, RPC Java, JEE, JMS.NET, C#, VB, WCF AJAX, Perl, Python Orientations Service Batch Files Message Resource Mindsets WSDL, XML, WS-* REST, MEST, KISS Vendor Platforms JEE,.NET, etc Operating Systems Exploitation & Support Applications SAP, Siebel, etc Devices Mobile, Wireless, POS, Sensor, Actuator, RFID Web services SOAP, WSDL, WS-RM, WS-N Web 2.0 HTTP, AJAX, REST, Appliances Transactional Guaranteed Persistent At-Most-Once Replay At-least-once Best-Effort Fire-and-Forget Request-Reply Fastest speed Lowest Latency Client-Server Backbone Point-to-Point Peer-to-Peer Publish/Subscribe Grid Bus Multicast Unicast IBM WebSphere MQ File Transfer Edition - Product Introduction 10

11 What is WebSphere MQ File Transfer Edition? Flexible backbone for transfers not a single-hop solution like FTP Multi-purpose use for messages and files Loggable with logging subsystem that tracks transfer at source and at destination for Log purposes Massive files larger than WebSphere MQ messages Reliability leveraging the WebSphere MQ transport Integration with WebSphere MQ-enabled apps and enterprise service buses (ESBs) No need to program no need to use application programming interfaces (APIs) Simple graphical tooling enabling remote configuration Automatic file conversion and compression Security - of file payload using Secure Sockets Layer (SSL) Visual transfer status reporting Support for common WebSphere MQ environments configure track File Transfer Edition WebSphere MQ Log A B C X Y Z Announced October 7, 2008 GA December 5, 2008 (Distributed) GA December (z/os) IBM WebSphere MQ File Transfer Edition - Product Introduction 11

12 Key Themes WebSphere MQ File Transfer Edition Auditable Log logs of transfers at source and target Log data persisted to WebSphere MQ queues and/or relational database. Captures time-stamped log at source and target Ease-of-Use Remote console for transfer initiation, unattended operation, scripting, scheduling, restart policies, status display Integrated with WebSphere MQ Explorer configuration tooling Simplicity Small footprint, fast install No need to write code or use API to configure transfers Enabled via GUI Leverages WebSphere MQ no other technology pre-requisites Security Access to individual files subject to file system permissions Link level security (inheriting WebSphere MQ SSL security) Breadth Support WebSphere MQ V6 and V7 for transfers Core Platform support (z/os, Linux (32 Bit), Solaris, IBM AIX, HP, Windows ) Good file type support (ASCII/EBCDIC, CR/LF, Flat files, z/os) Automated Transfers Transfers can be scheduled to repeat at predetermined intervals Transfers can be triggered by range of file system events e.g. new files, updated file, etc. IBM WebSphere MQ File Transfer Edition - Product Introduction 12

13 IBM WebSphere MQ File Transfer Edition - Product Introduction 13

14 Example Topologies A A B A Q Q A Coord C A Coord A Hub and Spoke A Star A A A Q Q Q A A Coord Bus Agents QM Admin A A IBM WebSphere MQ File Transfer Edition - Product Introduction 14

15 WebSphere MQ File Transfer Edition Agent Architecture JVM File Sender threads File Receiver threads File Xfre Ack thread Thread Pool File Transfer Edition Agent Command Process threads File Chuck Buffer Pool Scheduler Triggers JVM File Sender threads File Receiver threads File Xfre Ack thread Thread Pool File Transfer Edition Agent Command Process threads File Chuck Buffer Pool Scheduler Triggers DATA Queue Manager STATE DATA Queue Manager (WebSphere MQ V6 +) (WebSphere MQ V6 +) STATE COMMAND REPLY EVENT COMMAND REPLY EVENT Multi-threaded Architecture for performance and scalability Highly efficient WebSphere MQ pacing based stream I/O with full in-flight checkpoint and restart Built in scheduler and trigger support Bindings (Server) and Client support Leverages WebSphere MQ Publish and Subscribe infrastructure for Transfer Logs and progress IBM WebSphere MQ File Transfer Edition - Product Introduction 15

16 WebSphere MQ File Transfer Edition Agent using Server Connection WebSphere MQ File Transfer Edition Agent z/os WebSphere MQ Queue Manager CSQ6 MQSeries Server Channels Sender/Receiver z/os WebSphere MQ Queue Manager CSQ1 WebSphere MQ File Transfer Edition Agent Sender/Receiver Sender Receiver Sender/Receiver Windows Linux UNIX WebSphere MQ File Transfer Edition Agent MQSeries Client Channel WebSphere MQ Queue Manager LINUX01 WebSphere MQ File Transfer Edition Agent IBM WebSphere MQ File Transfer Edition - Product Introduction 16

17 WebSphere MQ File Transfer Edition Agent using Server Connection Agents connects in Bindings Mode Cross memory data passing High performance Asynchronous Channel transfer data Network hiccups do not impact transfers Opportunity for very high performance network throughput File Transfer traffic flows in both directions at the same time IBM WebSphere MQ File Transfer Edition - Product Introduction 17

18 WebSphere MQ File Transfer Edition Agent using Client Channel Connection WebSphere MQ File Transfer Edition Agent z/os WebSphere MQ Queue Manager CSQ6 MQSeries Server Channels Sender/Receiver z/os WebSphere MQ Queue Manager CSQ1 WebSphere MQ File Transfer Edition Agent Sender/Receiver Sender Receiver Sender/Receiver Windows Linux UNIX WebSphere MQ File Transfer Edition Agent MQSeries Client Channel WebSphere MQ Queue Manager LINUX01 WebSphere MQ File Transfer Edition Agent IBM WebSphere MQ File Transfer Edition - Product Introduction 18

19 WebSphere MQ File Transfer Edition Agent using Client channel connection Agents connect over the network to a Queue Manager (QM) Susceptible to network failures Not always good over the wide area network (WAN) All traffic needs to go via the hosting QM Server Less efficient when transferring between clients Client Channels are more chatty Additional network traffic WebSphere MQ V6 clients use more sessions and resources Additional load on the Server resources WebSphere MQ V7 clients can share sessions and have options for improved performance over V6 clients IBM WebSphere MQ File Transfer Edition - Product Introduction 19

20 How to start a File Transfer Command Line Interface is consistent across all supported platforms Transfer commands can be invoked from the supported Operating Systems shell environment Commands can be invoked from anywhere across the file transfer i.e. Command could be invoked from a Windows machine for transfers taking place between z/os and UNIX machines Developers can use any native command line language on the OS that can invoke these commands (shell, bat, cmd, etc.) Application Programs can place a request using a messaging interface in XML Examples: ftecreatetransfer Starts a new file transfer from the command line ftestartagent Starts a File Transfer agent from the command line ftestopagent Stops a File Transfer agent in a controlled way fteshowagentdetails Displays the details of a particular File Transfer agent fteshowagents Displays the status of all known Transfer agents IBM WebSphere MQ File Transfer Edition - Product Introduction 20

21 Centralized Configuration & Administration Logically centralized configuration of remote, distributed backbone Remotely view & configure entire backbone including on z/os Create Delete Display Modify Deploy Test Visual display at a glance Eclipse-based environment Extensible and customizable Remote connection from Linux x86 and Windows SSL secured connections IBM WebSphere MQ File Transfer Edition - Product Introduction 21

22 Approaches on automating modified file transfer (MFT) processes Which possibilities / tools exists today? JCL REXX KSH, SH, CMD, BAT Scripts Vendor Software A new possibility: Apache Ant IBM WebSphere MQ File Transfer Edition - Product Introduction 22

23 What is Apache Ant Apache Ant is an Java-based Open-Source Build-Tool, similar to Make. Originally intended for automated build (compile) of Java code Ant provides Java-Classes (Tasks) for automating different things Build-Scripts are formulated in XML Web Page: IBM WebSphere MQ File Transfer Edition - Product Introduction 23

24 How does Ant work? Ant Ant Tasks - copy, zip, exec, sql, delete, mkdir, ftp, telnet, cvs, smtp, smnp4ant Ant provides different functions as so called Ant- Tasks Processes are described in XML Build-Scripts. The Ant-Tasks are called from within those scripts Ant steps can be invoked conditionally Each Ant-Task is built out of a Java-Class Buildfile (XML) Buildfile (XML) Buildfile (XML) User can easily develop custom Ant-Tasks. Ant is constantly being extended and enhanced IBM WebSphere MQ File Transfer Edition - Product Introduction 24

25 Standard Ant Tasks (Core & Optional Tasks) File operations Copy, Move, Delete, Rename, Mkdir Zip/Unzip Compression/expansion of files SQL Database queries (requires JDBC Driver) FTP File transfer to remote systems Mail Sending of s (including attachments) Telnet Remote control of a Telnet-Session Exec Executes a local system command or executable Echo Issues messages IBM WebSphere MQ File Transfer Edition - Product Introduction 25

26 Using Ant to automate MFT processes Uses WebSphere MQ Java-based Ant Extensions to invoke remote task at WebSphere MQ File Transfer Edition agents File Transfer Edition Agents extended to include special command launcher capabilities Ant-Tasks interoperate with File Transfer Edition Agents via WebSphere MQ Supports both Server and Client type Agents Use any standard Ant task in combination with File Transfer Edition Ant tasks WebSphere MQ File Transfer Edition Agent FTE Extensions Data Any Command Any FTE Platform Ant Runtime Ant Tasks - copy, zip, exec, sql, delete, mkdir, ftp, telnet, cvs, smtp File Transfer Edition Ant Extensions - ftecopy ftemove - fteinvoke, fteping. WebSphere MQ Network Any Ant supporting platform File transfer Edition Ant Job (Build file) (XML) Audit Agents Audit Database IBM WebSphere MQ File Transfer Edition - Product Introduction 26

27 WebSphere MQ File Transfer Edition Ant Task Extensions fte:copy fte:move fte:invoke fte:ping fte:uuid fte:awaitoutcome fte:cancel fte:ignoreoutcome Copy one or more files Move one or more files Launch any program or command and capture output results and return codes with optional retry capability Check connectivity Assign a unique id Wait for results Cancel a previously invoked copy or move request Continue processing IBM WebSphere MQ File Transfer Edition - Product Introduction 27

28 Include Common Tasks Metadata to be recorded In transfer History Pre and Post Transfer Program Launch Manage Steps & Conditional Processing 28 Introduction 08/13/09 IBM WebSphere MQ File Transfer Edition - Product Introduction 28

29 Variables To be Overridden Command to Invoke Ant Job Ant -f CDDOTXP1.xml DSNODE AGENT1 DPNODE AGENT2 -Djobname JOBXXXX -DDSN1 GIS.DMVOUT.I5201TX.N DDSN2 /mailbox/outboundxxxxx/gis.n24392.out IBM WebSphere MQ File Transfer Edition - Product Introduction 29

30 z/os Batch JCL invocation example //IBMUSER JOB. //STEP2 EXEC PGM=BPXBATCH,REGION=0M, PARM='PGM /bin/sh/usr/ant' //STDOUT DD SYSOUT=* //STDERR DD SYSOUT=* //STDIN DD * -f CDDOTXP1.xml DSNODE FIG_FTE_SERVER DPNODE PNODE -Djobname JOBXXXX -DDSN1 GIS.DMVOUT.I5201TX.N DDSN2 /mailbox/outboundxxxxx/gis.n24392.out //* IBM WebSphere MQ File Transfer Edition - Product Introduction 30

31 Benefits IBM Software Group One central control instance for processes Processes may cross OS and Platform borders Integration of distributed platforms and mainframe systems in the automated process Today, data is processed on many different platforms and system One central place to store all automation scripts, files, jobs,... Central processing and control, including easy archiving of the logs Ant is a standard for complex Build tasks, now it can be the standard for Distributed MFT and batch automation IBM WebSphere MQ File Transfer Edition - Product Introduction 31

32 Eclipse GUI integrated into WebSphere MQ Explorer IBM WebSphere MQ File Transfer Edition - Product Introduction 32

33 Creating File Transfers List of sources and targets is built automatically Specify directory and file name Choose advanced options Choose mode (Binary or Text with automatic conversion) Add each individual transfer to a group of transfers Choose priority of transfer Deploy file transfer to network IBM WebSphere MQ File Transfer Edition - Product Introduction 33

34 Scheduling & Triggering File Transfers Choose when to start the scheduled transfer Choose advanced options Choose when to repeat the scheduled transfer and how often Choose the trigger for the transfer Deploy file transfer to network IBM WebSphere MQ File Transfer Edition - Product Introduction 34

35 Logging & Monitoring File Transfers IBM WebSphere MQ File Transfer Edition - Product Introduction 35

36 Why IBM? IBM Software Group Over 15 years of proven experience Connect virtually anything Over 15 years leadership in Messaging technology innovation Broad coverage of platforms, technologies, languages Draw skills from a larger pool use who you have today Over 9,300 certified developers for IBM Messaging alone Most widely deployed Messaging Backbone Over 10,000 customers using IBM Messaging Backbone Over 90% of the Fortune 50 and 9 of the Fortune 10 Over 80% of the Global 25 and 7 of the Global 10 Entrusted with Tens of billions of messages each day Relied upon as the mission-critical Backbone Continuously Investing and Innovating Results reported from actual IBM WebSphere MQ implementations * Government client sends 675 million messages per day* Banking client handles over 213 million messages per day on z/os alone* Financial Markets client handles US$1 trillion worth of traffic per day on one WebSphere MQ network* Banking client sends US$7-US$35 trillion worth of traffic per day on just one WebSphere MQ-based SWIFT gateway* Over 120 patents and filings within the messaging and ESB space New WebSphere MQ family products Regular enhancements, updates and new releases IBM WebSphere MQ File Transfer Edition - Product Introduction 36

37 IBM WebSphere MQ File Transfer Edition - Product Introduction 37

38 We appreciate your feedback. Please fill out the survey form in order to improve this educational event. IBM WebSphere MQ File Transfer Edition - Product Introduction 38