Unity Data Mover Release Definition Release B C April 2014

Transcription

1 Release Definition Release B C April 2014

2 The product or products described in this book are licensed products of Teradata Corporation or its affiliates. Teradata, Active Data Warehousing, Active Enterprise Intelligence, Applications-Within, Aprimo Marketing Studio, Aster, BYNET, Claraview, DecisionCast, Gridscale, MyCommerce, QueryGrid, SQL-MapReduce, Teradata Decision Experts, "Teradata Labs" logo, Teradata ServiceConnect, Teradata Source Experts, WebAnalyst, and Xkoto are trademarks or registered trademarks of Teradata Corporation or its affiliates in the United States and other countries. Adaptec and SCSISelect are trademarks or registered trademarks of Adaptec, Inc. AMD Opteron and Opteron are trademarks of Advanced Micro Devices, Inc. Apache, Apache Hadoop, Hadoop, and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. Apple, Mac, and OS X all are registered trademarks of Apple Inc. Axeda is a registered trademark of Axeda Corporation. Axeda Agents, Axeda Applications, Axeda Policy Manager, Axeda Enterprise, Axeda Access, Axeda Software Management, Axeda Service, Axeda ServiceLink, and Firewall-Friendly are trademarks and Maximum Results and Maximum Support are servicemarks of Axeda Corporation. Data Domain, EMC, PowerPath, SRDF, and Symmetrix are registered trademarks of EMC Corporation. GoldenGate is a trademark of Oracle. Hewlett-Packard and HP are registered trademarks of Hewlett-Packard Company. Hortonworks, the Hortonworks logo and other Hortonworks trademarks are trademarks of Hortonworks Inc. in the United States and other countries. Intel, Pentium, and XEON are registered trademarks of Intel Corporation. IBM, CICS, RACF, Tivoli, and z/os are registered trademarks of International Business Machines Corporation. Linux is a registered trademark of Linus Torvalds. LSI is a registered trademark of LSI Corporation. Microsoft, Active Directory, Windows, Windows NT, and Windows Server are registered trademarks of Microsoft Corporation in the United States and other countries. NetVault is a trademark or registered trademark of Dell Inc. in the United States and/or other countries. Novell and SUSE are registered trademarks of Novell, Inc., in the United States and other countries. Oracle, Java, and Solaris are registered trademarks of Oracle and/or its affiliates. QLogic and SANbox are trademarks or registered trademarks of QLogic Corporation. Quantum and the Quantum logo are trademarks of Quantum Corporation, registered in the U.S.A. and other countries. Red Hat is a trademark of Red Hat, Inc., registered in the U.S. and other countries. Used under license. SAP is the trademark or registered trademark of SAP AG in Germany and in several other countries. SAS and SAS/C are trademarks or registered trademarks of SAS Institute Inc. SPARC is a registered trademark of SPARC International, Inc. Symantec, NetBackup, and VERITAS are trademarks or registered trademarks of Symantec Corporation or its affiliates in the United States and other countries. Unicode is a registered trademark of Unicode, Inc. in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Other product and company names mentioned herein may be the trademarks of their respective owners. The information contained in this document is provided on an "as-is" basis, without warranty of any kind, either express or implied, including the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. Some jurisdictions do not allow the exclusion of implied warranties, so the above exclusion may not apply to you. In no event will Teradata Corporation be liable for any indirect, direct, special, incidental, or consequential damages, including lost profits or lost savings, even if expressly advised of the possibility of such damages. The information contained in this document may contain references or cross-references to features, functions, products, or services that are not announced or available in your country. Such references do not imply that Teradata Corporation intends to announce such features, functions, products, or services in your country. Please consult your local Teradata Corporation representative for those features, functions, products, or services available in your country. Information contained in this document may contain technical inaccuracies or typographical errors. Information may be changed or updated without notice. Teradata Corporation may also make improvements or changes in the products or services described in this information at any time without notice. To maintain the quality of our products and services, we would like your comments on the accuracy, clarity, organization, and value of this document. Please teradata-books@lists.teradata.com Any comments or materials (collectively referred to as "Feedback") sent to Teradata Corporation will be deemed non-confidential. Teradata Corporation will have no obligation of any kind with respect to Feedback and will be free to use, reproduce, disclose, exhibit, display, transform, create derivative works of, and distribute the Feedback and derivative works thereof without limitation on a royalty-free basis. Further, Teradata Corporation will be free to use any ideas, concepts, know-how, or techniques contained in such Feedback for any purpose whatsoever, including developing, manufacturing, or marketing products or services incorporating Feedback. Copyright 2014 by Teradata. All Rights Reserved.

3 Table of Contents Preface...5 Purpose... 5 Supported Releases... 5 Additional Information...5 Product Documentation...5 Orange Books...6 Customer Support... 6 Customer Education... 6 Product Safety Information...6 Chapter 1: Release Information...9 New Features... 9 Release Notes...9 Improvements... 9 Resolved s...13 Unity Data Mover Release Definition, Release

4 Table of Contents 4 Unity Data Mover Release Definition, Release 14.11

5 Preface Purpose Unity Data Mover enables you to define and edit jobs that copy specified database objects, such as tables, users, views, macros, stored procedures, and statistics, from one Teradata Database system to another. Tables can be copied between Teradata systems and an Aster or Hadoop system. Unity Data Mover Release Definition provides an overview of Unity Data Mover and its requirements. It also includes release notes, helpful hints, and information on how to contact customer support. Supported Releases Unless otherwise specified in this document, "this release" refers to Data Mover Teradata Database (for Data Movement between) 13.00, 13.10, 14.0, and systems Teradata Viewpoint Teradata Tools and Utilities Aster 4.6.1, 5.10, and 6.0 (for Teradata Database 12.0, 13.0, 13.10, and 14.0) Aster 6.0 (for Teradata Database 14.10) Hadoop (for Teradata Database 13.0, 13.10, 14.0, and 14.10) Teradata Connector for Hadoop (version 1.1, latest incremental release) Additional Information Product Documentation Access the Teradata Information Products web site to view or download additional documentation related to this product. 1. Go to 2. Under the Downloadable Publications subcategory, click General Search. 3. Search on the title or publication ID. Unity Data Mover Release Definition, Release

6 Preface Product Safety Information Title Unity Data Mover User Guide Describes how to use the Unity Data Mover portlets and command-line interface. Unity Data Mover Configuration and Upgrade Guide Describes how to configure Unity Data Mover software. Publication ID B B Orange Books Orange Books enhance product-specific documentation by describing the best practices for completing a task or solving a problem using the tools and features of one or more products. Orange Books are available to: Teradata customers with a service contract Teradata Professional Services associates Teradata Customer Service associates 1. Go to 2. Follow the instructions to log on or to register. 3. From the menu bar, click Knowledge Search. 4. In the search box, type Orange Book and click Search. Note: If you do not select a language option, it defaults to English. Customer Support Customer support is available around-the-clock, seven days a week through the Global Technical Support Center (GSC). To learn more, go to Customer Education Teradata Customer Education delivers training for your global workforce, including scheduled public courses, customized on-site training, and web-based training. For information about the classes, schedules, and the Teradata Certification Program, go to Product Safety Information This document may contain information addressing product safety practices related to data or property damage, identified by the word Notice. A notice indicates a situation which, if not avoided, could result in damage to property, such as equipment or data, but not related to personal injury. 6 Unity Data Mover Release Definition, Release 14.11

7 Preface Product Safety Information Example Notice: Improper use of the Reconfiguration utility can result in data loss. Unity Data Mover Release Definition, Release

8 Preface Product Safety Information 8 Unity Data Mover Release Definition, Release 14.11

9 New Features The following are new features that enhance the operation of Data Mover: Enhanced Viewpoint portal user interface. Move data to and from Teradata Hadoop and Teradata Aster. Create diagnostic bundle with job and system information for troubleshooting. Edit jobs through the command line. Change priority of jobs in queue. Change jobs on global level using credential pools for source and target systems. Dynamic default performance settings of streams and sessions. Send events for one job to multiple events tables in the command-line interface. Automatically determine when it is beneficial to use NOPI staging tables. Release Notes The following conditions affect Unity Data Mover 14.11, and will be added to future releases of the Unity Data Mover User Guide: Feature Automatic Failover Best Practices More information about failover support when multiple Teradata Data Mover servers are configured in a master and slave configuration. More information about best practices for users and permissions. Improvements Improvements are changes to existing features that enhance the operation of Data Mover, including related documentation. DM-4465 The number of rows and bytes are now reported in separate columns of the TMSMEvent table Unity Data Mover Release Definition, Release

10 Improvements DM-7758 DM-9921 DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM The Unity Data Mover portlets now support double characters in data source names and when renaming database objects Users can now specify additional command line parameters to ARC, and this enhancement was specifically implemented so that VBMASK=15 output can be requested when trying to debug ARC failures Users can now edit a job through the command line without losing the job history Users can now use TPTAPI in Unity Data Mover jobs where the source and target systems are the same Users can now change job priority in the command line interface after submittal to the queue, but before it starts executing To improve supportability, the Daemon now displays the command line IP address and host information when the command line is connected For disaster recovery reasons, Unity Data Mover can now be configured through the command line interface to send events for one job to multiple TMSMEvent tables Job start times have been reduced by batching SELECT, INSERT, and SQL statements when many tables need to be moved For disaster recovery reasons, Unity Data Mover can now be configured through the Unity Data Mover portlet to send events for one job to multiple TMSMEvent tables In order to improve performance, Unity Data Mover now dynamically sets the number of streams for source and target systems Unity Data Mover now evaluates return codes <= 4 from arcmain jobs correctly Allow use of TPTAPI Load Operator for Multi-Set Tables Unity Data Mover now uses CachingConnectionFactory instead of PooledConnectionFactory for all Message Producers in each project All Unity Data Mover install logic has been moved into the RPM file for improved usability with PUT Security authentication logic has been modified in Unity Data Mover commands by not sending the user profile information through messages The Unity Data Mover backdown procedure has been simplified and somewhat automated Failed attempts to start a job now visible through list_jobs or portlet When the Unity Data Mover agent goes down, tasks that were started by the agent before it went offline now fail, and tasks from ActiveMQ that had been sent to the downed agent but were not consumed before the agent went down are now cleaned up 10 Unity Data Mover Release Definition, Release 14.11

11 Improvements DM DM Unity Data Mover Intel Romley managed servers have been certification tested with Infiniband network connectivity In order to improve performance, Unity Data Mover now dynamically sets the number of sessions for source and target systems DM The Unity Data Mover portlet now uses the Viewpoint framework 2.0 architecture DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM Handling pools of credentials for both source and target systems has been added for simplified administration so that job changes can be made at a global level Unity Data Mover now has a support bundle containing diagnostic information when problems occur Streamline Automatic Failover Setup and Configuration BYNET and Unity managed server configuration when using Infiniband network connectivity has been documented and certification tested When creating a job, Unity Data Mover now automatically determines if it is beneficial to use NOPI staging tables Unity Data Mover now supports data movement between Hadoop and Teradata systems Unity Data Mover now supports data movement between Aster and Teradata systems Automatically setting default queryband variable/value pairs for all database activities that Unity Data Mover drives has been added Unity Data Mover now automatically attaches a diagnostic bundle when a Critical or Degraded TVI alert is fired Unity Data Mover now records which user invoked jobs from the portlet The Unity Data Mover ARC control socket now uses strong and secure ciphers Users can now change job priority in the Unity Data Mover portlet after submittal to the queue, but before it starts executing The <dmedit> XML tag is now used for both the edit and parameterized job variables commands Unity Data Mover now only allows the command line admin user or the portlet admin user to change job permissions and ownership The Unity Data Mover sync service now updates the slave repository after executing the daemon restore command Unity Data Mover now purges the dmauditlog table when the sync service has processed all of the rows Parameters have been added to list_jobs command to support queries by date and time Unity Data Mover Release Definition, Release

12 Improvements DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM The Unity Data Mover repository purge feature is now turned on by default Users can now purge job definitions from the Unity Data Mover repository in addition to job history Unity Data Mover now reports the number of sessions that the ARC and TPTAPI protocols used during job execution When the row count validation step fails and the override lock access option is in use, a warning has been added to the error information stating that inserts may have occurred on the source table while the job was running The Unity Data Mover command line package is now supported on the Windows Server 2012 operating system The Unity Data Mover documentation was enhanced to provide guidelines and best practices when using user rights and roles When moving data to 14.0 and older Teradata databases, use SET staging tables when moving SET tables An Aster and Hadoop system compatibility matrix was added to the Unity Data Mover user guide Improvements were made to the dmsupport.sh script including the collection of.core and.snap files in multiple directories With the implementation of the Viewpoint framework 2.0, the viewpoint service code to stop Viewpoint in the Unity Data Mover portlet pre-install script was removed The Unity Data Mover command line package is now supported on the Windows 8.1 operating system Naming threads at the main entry point, logging all exceptions with a logger error, and catching/sending exceptions as errors to stdout, have all been implemented in the Unity Data Mover command line for improved debugging purposes Add Monitor Permission reference so that the ARC protocol can gather AMP information to Unity Data Mover User Guide Implemented for debugging network issues, an optional property called wireformat.maxinactivityduration in the.properties files so that the default value of the wireformat.maxinactivityduration in the ActiveMQ URL can be configured The list_job_steps command now shows the number of sessions and streams, and whether they were user defined or dynamically generated Passwords are now encrypted in the Unity Data Mover portlet when saving to Favorite systems preferences During Unity Data Mover upgrades a process now exists to modify the number of repository AMPS 12 Unity Data Mover Release Definition, Release 14.11

13 Resolved s DM For security reasons, the ifconfg and netstat -an commands have been removed from the script that generates a support bundle Resolved s Resolved issues are corrections to Data Mover 14.11, including related documentation. DM-3955 DM-8031 DM-8831 DM DM DM DM DM DM DM DM DM DM DM Unity Data Mover no longer reports a TARGET INSERT-SELECT END event after an ALERT occurs If multiple Unity Data Mover portlets are open, default preferences are now maintained within each portlet instance The host name no longer changes to host IP in the source/target drop down menu when creating a new job in the Unity Data Mover portlet Table object settings that have default values only, and whose parent is selected in the Unity Data Mover portlet, are no longer included in the object settings The error message was corrected when a parent database can't be found in the portlet while executing queries to traverse the job tree Special characters (e.g., double quotes) in job names no longer affect the functionality of the Unity Data Mover portlet When editing a job in the Unity Data Mover portlet created in the command line interface now shows the table(s) as checked The relocate database feature option is now working in the Unity Data Mover portlet Long system names are now completely displayed on the Unity Data Mover portlet source and target drop downs in Internet Explorer 8 An edit Job issue in the Unity Data Mover portlet has been resolved when changing a job from copying tables to copying the entire database Copy stats conditions were updated in the Unity Data Mover portlet The Unity Data Mover portlet user no longer loses an old object selection when he/she is the owner of the job and edits the source credentials When using the TPTAPI protocol, the fully qualified table name no longer has a duplicate database name in status output. When using the TPTAPI protocol, the fully qualified table name no longer has a duplicate database name in status output. Tables of different databases no longer show under one database when 'show selected only' is checked in the Unity Data Mover portlet Unity Data Mover Release Definition, Release

14 Resolved s DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM Selected and unselected tables are now shown in the Unity Data Mover portlet when 'show selected only' is checked and the database is expanded The Unity Data Mover portlet now shows object of a selected job when editing jobs after marking the 'Show selected only' checkbox The Unity Data Mover portlet now shows information on the Job Summary tab correctly when more than one object is selected The Unity Data Mover portlet now shows objects for a Teradata system that has non-numeric characters in its version The job name no longer gets saved if the job was not created successfully Validation logic now exists on the Unity Data Mover portlet's Advance Settings screen to ensure that the data streams value is always greater than or equal to the Max_agents_per_task value A restriction now exists to prevent copying any DBC object, but Unity Data Mover doesn t prevent users and child databases directly under DBC from getting copied to the target system The Unity Data Mover portlet now shows a message when a new job is placed into the queue because the maximum number of jobs executed has been reached Command line package installation issues on Solaris Sparc 10 have been resolved When changing the password in the target user pool and using start command to run an existing job, it no longer fails When a staging database is used, the TPTAPI Cleanup command now uses the staging database name in the cleanup process' drop table statement The dependency on JAVA Runtime Environment (JRE) 5 has been removed from the installation architecture now that JRE 6 is used Unity Data Mover now returns an error if the user provides an unknown parameter on the command line Unity Data Mover now copies temporal tables properly when using the TPTAPI protocol When executing copy stats, Teradata systems no longer throw 3760 errors for incomplete statements when a very long COLLECT STATISTICS statement is broken into multiple statements of a SQL task For stored procedure errors, the Unity Data Mover sync service is now working properly after a daemon backup and restore process completes Japanese and Simplified Chinese localization issues in the Unity Data Mover portlets have been resolved Users can now add 2 tables with the same name to a job as long as they exist in 2 different databases The list_jobs output no longer truncates large job names 14 Unity Data Mover Release Definition, Release 14.11

15 Resolved s DM DM DM DM DM DM DM DM DM DM DM DM DM DM The Unity Data Mover documentation was modified to alert users that spaces are not allowed in user names The daemon restore process has been fixed to ensure the stored procedure jar file is in a usable state Unity Data Mover now send a TVI alert message when an error occurs in inserting into a TMSMEvent table When enabling the "use user ID pool" in the Unity Data Mover portlet, a "Marshal to XML failed" error no longer occurs The command line's DMLC functionality is automatically set to false to avoid consuming a message that was not meant for that instance The Unity Data Mover command line's DMLC is now shutdown so that the connection to ActiveMQ is closed cleanly In setting the cachingconsumers property to false, the camproducerconnectionfactory no longer leaks message consumers The cleanup task now has double quotes around the database name and password so that it can process special characters An exception is no longer thrown when deleting the event table for a database that doesn't exist The Unity Data Mover agent's max concurrent job property was renamed max concurrent tasks to eliminate confusion around the number of tasks the agent can execute The Unity Data Mover command line no longer throws error when viewing status of a job in which its name has spaces Several bugs were fixed in the parameterized start command The Unity Data Mover cleanup command now properly disables online_archive Text in the Unity Data Mover user guide related to the "blocked.job.retry.enabled" property was corrected DM The default port for the sync service was changed from to DM DM DM DM The default event table is now switched for a job when its value is changed in the configuration.xml file A file locking issue forcing an exist in the slave sync service to exit caused by 2 threads on the slave side, one writing to the.sql file and one reading from the.sql file, has been resolved Validation logic for source and target system names has been added to the Unity Data Mover portlet when creating new jobs The ActiveMQ issue "com.teradata.commons.messaging.messagingexception: Create session failed Caused by: javax.jms.jmsexception: Wire format negotiation timeout: peer did not send his wire format" has been resolved Unity Data Mover Release Definition, Release

16 Resolved s DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM DM ActiveMQ temp storage filled errors found at several customer sites has been resolved The cleanup task now has double quotes around the database name and password so that it can process special characters The Unity Data Mover installation and upgrade process no longer throws a "com.teradata.jdbc.jdbc_4.util.jdbcexception: [Teradata Database] [TeraJDBC ] [Error 3760] [SQLState 42000] String not terminated before end of text." error A queued job now starts when the maxconcurrentjobs in daemon.properties value is reset to larger value Unity Data Mover ARC job no longer fail with a NullpointerException if it receives an unexpected message from the ARC de-serializer and serializer The metadata creation thread pool is now getting shutdown properly to avoid future OutOfMemory issues When encrypted passwords are provided in both the start and edit commands, the parameterized command no longer has a password decryption exception When installing the Data Mover failover package on a Viewpoint managed server, the /var/opt/teradata/datamover/logs directory is created if it does not exist TPTAPI cleanup tasks now sent to all connected Unity Data Mover agents The Unity Data Mover TPTAPI cleanup tasks are now executed in the proper order The dmcluster startmonitor command no longer initiates a Unity Data Mover failover The create_event_table command no longer fails when no password is specified in the XML The Unity Data Mover portlet now reports total bytes properly when multiple agents and/or streams are used in a job The delete_event_table command now has a clear error message when the user doesn t have drop table permissions When a job definition doesn't use a user id pool, credentials need to be specified when source_tdpid or target_tdpid is changed during the edit or start commands User documentation now states that the there is a 1 to 1 relationship between Unity Data Mover managed servers and Viewpoint servers for security reasons The Cleanup command now release ARC locks properly when a user pool is used 16 Unity Data Mover Release Definition, Release 14.11

17 Resolved s DM DM DM DM DM DM DM DM Copy statistics logic has been fixed for data movement to and from Teradata 14.0 and later release systems The complete rpm and tar.gz file names are now provided in the dminstallcomponents, dmupgradecomponents, and dminstallupgradeagent scripts ARC jobs no longer hang due to an ExecutorControllerNotFoundException for an individual task within a job A URL request is now presented when switching tabs from Objects to Selection Summary in the Unity Data Mover portlet The dmsync command no longer disconnects from the slave with OverlappingFileLockException when more than 1 slave exists When using the TPTAPI protocol, Unity Data Mover no longer fails to move a table with a derived period column The error message when the key columns are specified without specifying the WHERE clause has been corrected Incorrect "dir" parameter references were fixed in the Unity Data Mover User Guide Unity Data Mover Release Definition, Release

18 Resolved s 18 Unity Data Mover Release Definition, Release 14.11